[EN] Harbor-to-Harbor Migration from Block Storage to Object Storage (AWS S3)

tl;dr — I migrated our staging Harbor registry from local disk storage to a new S3-backed Harbor instance using Harbor's built-in replication. The data move itself was the easy part. The surprises came from an old AWS SDK baked into Harbor's registry component, IAM policy gotchas, and the things replication doesn't copy.

Background

We run a self-hosted Harbor registry for our staging environment. It has served us well for years, but it stores every image layer on the instance's local disk. Disk-backed registries have a predictable life cycle: they fill up, you resize, they fill up again. At some point you stop resizing and start rethinking.

The obvious fix is to move the storage backend to object storage. Harbor's registry component (Docker Distribution under the hood) supports S3 natively, so the plan was simple: spin up a fresh Harbor with S3 as the storage backend, replicate everything from the old instance, then switch the domain over. No rsync, no downtime, no touching the old production setup.

For authentication to S3, I wanted to avoid static credentials entirely. The new instance runs on EC2 with an IAM role attached, so the registry should pick up credentials from the instance metadata service (IMDS) automatically.

The Problem

The fresh install went fine — until the registry container started crash-looping:

panic: invalid region provided: ap-southeast-3

Harbor's registry binary bundles an old version of aws-sdk-go with a hardcoded region list. Our region (Jakarta) launched in late 2021, which is apparently too new for it. The known workaround is setting regionendpoint explicitly in harbor.yml, which skips the region validation.

That fixed the panic but not the pulls and pushes:

s3aws: NoCredentialProviders: no valid providers in chain

This one took longer to figure out. IMDS was reachable from inside the container — I verified it manually with IMDSv2 token requests. The catch: the bundled SDK is so old it does not speak IMDSv2 at all, and our instance enforced HttpTokens=required. The SDK falls back to plain IMDSv1 GET requests, which the instance rejects with a 401. No tokens, no credentials, no S3.

💡 If you're debugging this: a hop limit of 2 is not enough to fix it. Hop limit controls how far IMDS responses can travel; it does nothing about the token requirement itself.

Then came the IAM layer. Pushes failed with AccessDenied on s3:ListBucketMultipartUploads — an action our role policy didn't include. Image pushes use S3 multipart uploads, and the registry needs to list in-progress uploads to support resumable, chunked blob pushes. One more gotcha: it's a bucket-level action. Put it on the arn:aws:s3:::bucket/* resource and AWS silently ignores it — it must be on the bucket ARN itself.

Finally, replication moved all 575 artifacts flawlessly but left behind everything else: robot accounts (their secrets are stored hashed and can never be exported), tag retention policies, project members, and GC schedules. And after the domain cutover, every Kubernetes pull secret in the cluster still held the old robot password — cue a namespace-wide wall of ErrImagePull.

The Solution

The final working storage configuration in harbor.yml:

storage_service:
  s3:
    region: ap-southeast-3
    bucket: my-harbor-bucket
    secure: true
    v4auth: true
    regionendpoint: https://s3.ap-southeast-3.amazonaws.com

No accesskey/secretkey — the credential chain falls through to the instance role. To unblock the old SDK, I relaxed the instance to allow IMDSv1:

aws ec2 modify-instance-metadata-options \
  --instance-id i-xxxxxxxxxxxx \
  --http-tokens optional \
  --http-endpoint enabled

⚠️ This is a security trade-off. IMDSv1 is more exposed to SSRF-style credential theft. For a single-purpose registry box on a private network I accepted it; review against your own security standards before copying this.

The IAM policy needs both halves in the right place:

{
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:ListBucket", "s3:GetBucketLocation", "s3:ListBucketMultipartUploads"],
      "Resource": "arn:aws:s3:::my-harbor-bucket"
    },
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:GetObject", "s3:DeleteObject",
                 "s3:AbortMultipartUpload", "s3:ListMultipartUploadParts"],
      "Resource": "arn:aws:s3:::my-harbor-bucket/*"
    }
  ]
}

For the data move, I went with pull-based replication configured on the new instance: register the old Harbor as a registry endpoint, create a replication rule with a ** filter, and trigger it manually. Pull-based means the newer replication engine does the work, the old production instance only needs a read account, and nothing on it gets reconfigured. 575/575 artifacts landed in S3 on the first run.

Everything replication skips, I recreated through the Harbor API: retention policies (export the rules from the old instance via GET /retentions/{id}, replay them with POST /retentions), a weekly GC schedule so retention actually frees S3 space, and the robot accounts. Robot secrets can't be copied — but if you recreate the robot, you can PATCH /api/v2.0/robots/{id} with a custom secret, so CI consumers keep working without a config change.

After the cutover, the ErrImagePull storm was a one-liner per namespace:

kubectl create secret docker-registry my-registry-secret \
  --docker-server=registry.example.com \
  --docker-username='robot$ci' \
  --docker-password="$NEW_SECRET" \
  -n my-namespace --dry-run=client -o yaml | kubectl apply -f -

One debugging note that saved me here: Harbor's token service returns HTTP 200 even for wrong credentials — it just issues a token with an empty access list, and the registry rejects it later. If your pulls 401 but the token endpoint looks fine, decode the JWT and check the access claim before trusting the status code.

The old instance is still around as a fallback, one final delta replication away from retirement. Total artifact loss during the migration: zero.