Pulling images from AWS ECR to Kubernetes automatically

byGink▻ Wed, 22 Sep 2021

To use images from ECR inside a non-AWS cluster, we have to provide credentials with username and password just like any other private registry. There's a matter, the password ECR providing us will be expired soon. It's available in 12 hours only.

That means everytime restart or rolling out new containers that need pulling from ECR, we must finish authentication again. Of course it can be done with a script from a CICD process. But it still needs an extra step and pretty tedious. And we're not always working with K8s cluster via CICD server.

So all we need is a cronjob to renew the ECR password in docker registry secret before it get expired.

Step by step:

• Create an image which will run and update the secret right in the cluster. I've made one on docker hub: ginkcode/ecr-kube. It's simply an alpine linux, pre-installed awscli and kubectl, also contains a script to run on starting up as below:

#!/bin/sh
if [ -z "$SA_USER" ]; then
  export SA_USER='default'
fi

if [ -z "$AWS_DEFAULT_REGION" ]; then
  export AWS_DEFAULT_REGION="ap-southeast-1"
fi

if [ -z "$AWS_ACCESS_KEY_ID" ] || [ -z "$AWS_SECRET_ACCESS_KEY" ] || [ -z "$ECR_REGISTRY_URL" ]; then
  echo 'Must pass 3 env variables:'
  echo "(*) AWS_ACCESS_KEY_ID"
  echo "(*) AWS_SECRET_ACCESS_KEY"
  echo "(*) ECR_REGISTRY_URL"
  echo "AWS_DEFAULT_REGION (ap-southeast-1)"
  echo "SA_USER (default)"
  exit 1
fi

export ECR_TOKEN=$(aws ecr get-login-password) && \
kubectl delete secret --ignore-not-found=true ecr-registry-secret && \
kubectl create secret docker-registry ecr-registry-secret \
  --docker-server=${ECR_REGISTRY_URL} \
  --docker-username=AWS \
  --docker-password=${ECR_TOKEN} && \
kubectl patch serviceaccount ${SA_USER} \
  -p '{"imagePullSecrets": [{"name": "ecr-registry-secret"}]}' && \
echo 'Successfully updated ECR token. 🎁'

• Because the container will work inside the cluster, it'll be run as default service account (which belong to the namespace of this cronjob). So we must grant default service account a few permissions to delete/create the secret as in yaml config below.

• It needs permissions to patch its own attribute at imagePullSecrets also. A full yaml for this cronjob as following:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: ecr-renew
rules:
- apiGroups: [""]
  resources: ["secrets", "serviceaccounts"]
  verbs:
  - create
  - update
  - patch
  - delete
  - get
  - list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ecr-renew-binding
subjects:
- kind: ServiceAccount
  name: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: ecr-renew
---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: ecr-renew-job
spec:
  schedule: "5 */6 * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: ecr-renew
            image: ginkcode/ecr-kube:latest
            imagePullPolicy: IfNotPresent
            envFrom:
            - secretRef:
                name: aws-credential
          restartPolicy: OnFailure

So I've configured to pass the credentials of AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, ECR_REGISTRY_URL to the cronjob container via another secret named aws-credential. You can also pass them from env or by any other secret you want, just provide enough info to make the renew script works.

In my way, I just do something like this:

cat > ecr.env << EOL
AWS_DEFAULT_REGION=ap-southeast-1
AWS_ACCESS_KEY_ID=xxx
AWS_SECRET_ACCESS_KEY=xxx
ECR_REGISTRY_URL=xxx
EOL

kubectl create secret generic aws-credential --from-file=./ecr.env

# Can create the secret by passing env directly as arguments also:
# kubectl create secret generic aws-credential --from-literal=key1=val1 --from-literal=key2=val2

Now, running up this cronjob and it will automatically renew your ECR docker-registry secret per 6 hours. The password from ECR will last long for 12 hours so the cluster is always ready to pull any images from connected ECR registry.