# [EN] Harbor-to-Harbor Migration from Block Storage to Object Storage (AWS S3)
💡
AI helped me write this article to make it more understandable and structured.
**tl;dr** — I migrated our staging Harbor registry from local disk storage to a new S3-backed Harbor instance using Harbor's built-in replication. The data move itself was the easy part. The surprises came from an old AWS SDK baked into Harbor's registry component, IAM policy gotchas, and the things replication *doesn't* copy. ## Background We run a self-hosted Harbor registry for our staging environment. It has served us well for years, but it stores every image layer on the instance's local disk. Disk-backed registries have a predictable life cycle: they fill up, you resize, they fill up again. At some point you stop resizing and start rethinking. The obvious fix is to move the storage backend to object storage. Harbor's registry component (Docker Distribution under the hood) supports S3 natively, so the plan was simple: spin up a fresh Harbor with S3 as the storage backend, replicate everything from the old instance, then switch the domain over. No `rsync`, no downtime, no touching the old production setup. For authentication to S3, I wanted to avoid static credentials entirely. The new instance runs on EC2 with an IAM role attached, so the registry should pick up credentials from the instance metadata service (IMDS) automatically. ## The Problem The fresh install went fine — until the registry container started crash-looping: ```plaintext panic: invalid region provided: ap-southeast-3 ``` Harbor's registry binary bundles an old version of `aws-sdk-go` with a *hardcoded* region list. Our region (Jakarta) launched in late 2021, which is apparently too new for it. The known workaround is setting `regionendpoint` explicitly in `harbor.yml`, which skips the region validation. That fixed the panic but not the pulls and pushes: ```plaintext s3aws: NoCredentialProviders: no valid providers in chain ``` This one took longer to figure out. IMDS was reachable from inside the container — I verified it manually with IMDSv2 token requests. The catch: the bundled SDK is so old it **does not speak IMDSv2 at all**, and our instance enforced `HttpTokens=required`. The SDK falls back to plain IMDSv1 GET requests, which the instance rejects with a 401. No tokens, no credentials, no S3. 💡 If you're debugging this: a hop limit of 2 is *not* enough to fix it. Hop limit controls how far IMDS responses can travel; it does nothing about the token requirement itself. Then came the IAM layer. Pushes failed with `AccessDenied` on `s3:ListBucketMultipartUploads` — an action our role policy didn't include. Image pushes use S3 multipart uploads, and the registry needs to list in-progress uploads to support resumable, chunked blob pushes. One more gotcha: it's a **bucket-level** action. Put it on the `arn:aws:s3:::bucket/*` resource and AWS silently ignores it — it must be on the bucket ARN itself. Finally, replication moved all 575 artifacts flawlessly but left behind everything else: robot accounts (their secrets are stored hashed and can never be exported), tag retention policies, project members, and GC schedules. And after the domain cutover, every Kubernetes pull secret in the cluster still held the old robot password — cue a namespace-wide wall of `ErrImagePull`. ## The Solution The final working storage configuration in `harbor.yml`: ```yaml storage_service: s3: region: ap-southeast-3 bucket: my-harbor-bucket secure: true v4auth: true regionendpoint: https://s3.ap-southeast-3.amazonaws.com ``` No `accesskey`/`secretkey` — the credential chain falls through to the instance role. To unblock the old SDK, I relaxed the instance to allow IMDSv1: ```bash aws ec2 modify-instance-metadata-options \ --instance-id i-xxxxxxxxxxxx \ --http-tokens optional \ --http-endpoint enabled ``` ⚠️ This is a security trade-off. IMDSv1 is more exposed to SSRF-style credential theft. For a single-purpose registry box on a private network I accepted it; review against your own security standards before copying this. The IAM policy needs both halves in the right place: ```json { "Statement": [ { "Effect": "Allow", "Action": ["s3:ListBucket", "s3:GetBucketLocation", "s3:ListBucketMultipartUploads"], "Resource": "arn:aws:s3:::my-harbor-bucket" }, { "Effect": "Allow", "Action": ["s3:PutObject", "s3:GetObject", "s3:DeleteObject", "s3:AbortMultipartUpload", "s3:ListMultipartUploadParts"], "Resource": "arn:aws:s3:::my-harbor-bucket/*" } ] } ``` For the data move, I went with **pull-based replication configured on the new instance**: register the old Harbor as a registry endpoint, create a replication rule with a `**` filter, and trigger it manually. Pull-based means the newer replication engine does the work, the old production instance only needs a read account, and nothing on it gets reconfigured. 575/575 artifacts landed in S3 on the first run. Everything replication skips, I recreated through the Harbor API: retention policies (export the rules from the old instance via `GET /retentions/{id}`, replay them with `POST /retentions`), a weekly GC schedule so retention actually frees S3 space, and the robot accounts. Robot secrets can't be copied — but if you recreate the robot, you can `PATCH /api/v2.0/robots/{id}` with a custom secret, so CI consumers keep working without a config change. After the cutover, the `ErrImagePull` storm was a one-liner per namespace: ```bash kubectl create secret docker-registry my-registry-secret \ --docker-server=registry.example.com \ --docker-username='robot$ci' \ --docker-password="$NEW_SECRET" \ -n my-namespace --dry-run=client -o yaml | kubectl apply -f - ``` One debugging note that saved me here: Harbor's token service returns `HTTP 200` even for *wrong* credentials — it just issues a token with an empty `access` list, and the registry rejects it later. If your pulls 401 but the token endpoint looks fine, decode the JWT and check the `access` claim before trusting the status code. The old instance is still around as a fallback, one final delta replication away from retirement. Total artifact loss during the migration: zero.