The Problem

The developers are actively build and test their projects. There are dozens of projects that can be built in a day. This sometimes causes the disk of the CI/CD system (e.g., Jenkins, self-hosted runner) to be full, so we always delete unused images and build cache after each build. When a project needs to be rebuilt, it has to pull the base images from Docker Hub. But there is a rate limit from dockerhub, 100 pulls/6h unauthenticated, 200 authenticated per IPv4 address. If we use more than this, the build pipeline will fail.

The Solution

Because I use AWS, I can utilize AWS ECR pull-through cache to pull images from dockerhub, store them in ECR, then I can pull unlimited images from ECR. No dependency on Docker Hub uptime/rate limits. If Hub goes down, you can still pull (assuming the image you need is already cached on ECR). However, ECR storage cost for cached images. Mitigate it with lifecycle policies to evict old/unused tags.

Create a Docker Hub personal access token

This will be used by ECR to login to the Dockerhub. Token is used instead of your Docker hub password.

1. Log into Docker Hub

2. Navigate to Account Settings > Settings > Personal access tokens

3. Click the "Generate new token" button

4. Fill the token description, set the expiration date, and the permissions. Usually, I only give read-only permission.

5. Write down the generated PAT for the next step in AWS configuration.

Setting up the ECR pull-through rule

1. First, you need to store your Docker Hub credentials in AWS Secrets Manager with a specific naming convention. Use this AWS CLI command:

   aws secretsmanager create-secret \
       --name "ecr-pullthroughcache/docker-hub" \
       --description "Docker Hub credentials for ECR pull through cache" \
       --secret-string '{
           "username": "<your-docker-username>",
           "accessToken": "<your-docker-access-token>"
       }' \
       --region <region>
   

   Important: The secret name must use the ecr-pullthroughcache/ prefix and be in the same account and region as your pull-through cache rule.

2. Proceed with creating the pull-through rule. You can use this AWS CLI command:

   aws ecr create-pull-through-cache-rule \
       --ecr-repository-prefix docker-hub \
       --upstream-registry-url registry-1.docker.io \
       --credential-arn <secretsmanager-arn-you-got-from-the-previous-command> \
       --region <region>
   

3. Or, if you prefer to use the console, navigate to ECR > Private Registry > Pull through cache > Add Rule.

   

4. Set the Upstream registry to Docker Hub.

5. In the Authentication section, select the Use an existing AWS secret and select the secret created in step 1. Note that you can also skip step 1 and create the secrets from here if you use the console.

6. In the Step 3: Specify namespaces, configure the prefix configuration for both the cache and upstream namespace. I usually select use a specific prefix for the namespace, and no prefix in the Upstream namespace.

7. On Step 4: Review and create, review your configuration and choose Create

Pull the image through the cache

# Login to the ECR private registry
aws ecr get-login-password --region <region> | docker login --username AWS --password-stdin <your-acc-id>.dkr.ecr.<region>.amazonaws.com

# Pull an image from docker hub:
docker pull aws_account_id.dkr.ecr.region.amazonaws.com/docker-hub/library/image_name:tag

Notice the library in the repository URL is specific to Docker Hub's official images and reflects how Docker Hub organizes its repositories internally.

# Original Docker Hub command:
docker pull nginx:latest
docker pull alpine:3.23.4

# ECR pull-through cache equivalent:
docker pull aws_account_id.dkr.ecr.region.amazonaws.com/docker-hub/library/nginx:latest

When you pull these images, ECR creates repositories with these exact names:

• docker-hub/library/nginx

• docker-hub/library/alpine

Setting up the Repository Policy

I personally use a repository policy template to automatically apply a policy when ECR create Dockerhub pull-through repository. Such as, to only keep the last 6 images.

1. Navigate to ECR > Private Registry > Features & Settings > Repository creation templates > Configure > Create Template.

   

2. On Step 1: Define template > Template details > Applied for select the Pull through cache. Because I use a prefix in the previous step, I select the A specific prefix and set the prefix there.

3. On Step 2: Add repository creation configuration > Repository lifecycle policy, you can select predefined templates in the Lifecycle policy examples.

4. On Step 4: Review and create, review your configuration and choose Create

Now, if you pull a Dockerhub image through ECR, a lifecycle policy will be automatically applied.