Kaniko can't auth to AWS using IRSA

[ReeSilva]

Actual behavior
When running kaniko within a Gitlab Job in a k8s pod gitlab runner, even with the right service account properly annotated, kanico is not being able to authenticate in AWS ECR.

Expected behavior
When kaniko ran from a Gitlab Runner pod, it should still be able to authenticate to ECR using IRSA.

To Reproduce
Steps to reproduce the behavior:

1. Deploy a Gitlab Runner on an AWS EKS Cluster with the following config:

imagePullPolicy: Always
gitlabUrl: https://gitlab.com/
unregisterRunners: true
concurrent: 20
checkInterval: 30
logLevel: warn
rbac:
  create: true
  serviceAccountName: gitlab-runner
  clusterWideAccess: false
  serviceAccountAnnotations:
    eks.amazonaws.com/role-arn: [REDACTED]

metrics:
  enabled: true

runners:
  tags: [REDACTED]
  secret: [REDACTED]
  privileged: true
  outputLimit: 10240
  config: |
    [[runners]]
      environment = ["FF_GITLAB_REGISTRY_HELPER_IMAGE=1", "AWS_DEFAULT_REGION=eu-west-1"]
      [runners.kubernetes]
        image = "alpine:latest"
        cpu_request = "400m"
        memory_request = "1024Mi"
        service_cpu_request = "200m"
        service_memory_request = "256Mi"
        request_concurrency = 10
        pull_policy = "if-not-present"
        service_account = "gitlab-runner"
        service_account_overwrite_allowed = "^gitlab-runner$"
        [runners.kubernetes.node_selector]
          [REDACTED]
        [runners.cache]
          Type = "s3"
          Path = "cache"
          Shared = true
          [runners.cache.s3]
            BucketName = "bucket-cache"
            BucketLocation = "eu-west-1"

## Configure resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
##
resources:
  limits:
    memory: 512Mi
    cpu: 400m
  requests:
    memory: 256Mi
    cpu: 200m

2. Configure a job on Gitlab CI with the following configs:

.kaniko_publish_to_ecr:
  needs: ["set_image_tags"]
  dependencies:
    - set_image_tags
  artifacts:
    paths:
      - target/
      - pom.xml
      - build.properties
  variables:
    AWS_EC2_METADATA_DISABLED: "true"
    AWS_SDK_LOAD_CONFIG: "true"
    KUBERNETES_SERVICE_ACCOUNT: "gitlab-runner"
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  rules:
    - if: ($CI_COMMIT_MESSAGE =~ /^chore\(release\):/ || $CI_COMMIT_MESSAGE =~ /\[JENKINS\]/ ) && $CI_COMMIT_REF_NAME == "master"
      when: never
    - if: $CI_COMMIT_BRANCH && ($CI_COMMIT_REF_NAME == "master" || $CI_COMMIT_REF_NAME == "main")
      when: on_success
  tags:
    - [REDACTED]
  before_script:
    - mkdir -p /kaniko/.docker
    - echo "{\"credsStore\":\"ecr-login\"}" > /kaniko/.docker/config.json
  script:
    - cat /kaniko/.docker/config.json
    # Warm cache
    - /kaniko/warmer
        -v debug -i ${IMAGE_URL}
        --cache-dir /kaniko/gitlab-runner/cache
    # Build image
    - /kaniko/executor
        -v debug
        --context "${CI_PROJECT_DIR}"
        --dockerfile "${CI_PROJECT_DIR}/Dockerfile"
        --destination "${IMAGE_URL}:${CI_COMMIT_SHORT_SHA}"
        --destination "${IMAGE_URL}:${CI_PIPELINE_ID}"
        ${ADDITIONAL_DESTINATIONS}

3. Trigger a pipeline for this job. In the warmer, it will throw this error: Error while trying to warm image: "<account_id>.dkr.ecr.<region>.amazonaws.com/<project_name>" Failed to verify image name: "<account_id>.dkr.ecr.<region>.amazonaws.com/<project_name>": could not parse reference: "<account_id>.dkr.ecr.<region>.amazonaws.com/<project_name>" Failed warming cache: Failed to warm any of the given images. If remove warmer, executor throws this error: error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: getting tag for destination: repository can only contain the characters abcdefghijklmnopqrstuvwxyz0123456789_-./: <project_name>"

Additional Information

• Dockerfile

FROM <account_id>.dkr.ecr.<region>.amazonaws.com/<base_image>

ENV TOMCAT_WEBAPP_HOME /usr/local/tomcat/webapps
ARG CONTEXT=ROOT
ARG PROJECT_NAME=<project_name>

COPY ./target/$PROJECT_NAME.war $TOMCAT_WEBAPP_HOME/$CONTEXT.war
COPY ./target/$PROJECT_NAME $TOMCAT_WEBAPP_HOME/$CONTEXT

• Build Context
  The target folder is created in a previous Gitlab CI step, and they exists, are passed through artifacts.
• Kaniko Image (fully qualified with digest): gcr.io/kaniko-project/executor:debug:sha256:964426c9205d644e2964869d1d311a05dc9f301594300d3732ea26b5733e94fc

I'm trying to push the image to an ECR repository, with authentication through IRSA. The pod for the gitlab executor has the right web identity token properly mounted to the container.

Funny thing is that if I run the pod with kubectl run using the same svc account, works like a charm, but it fails when running from Gitlab CI.

There is a short version of the pod description: https://gitlab.com/-/snippets/2546250

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile

[bobbywatson3]

We are also seeing failures when trying to use kaniko with IRSA. Builds work with KIAM and with AWS access keys, but not with IRSA.

[mifonpe]

Same for us, any updates?

[cgill27]

I'm seeing the same issue, but it only started occurring for me when upgrading the Gitlab runner version to current version 16.4.1.
I can roll back to version 15.8.3 for example and no IRSA issue for kaniko pushing to ECR.

[balonik]

I am facing similar issue, but in my case Kaniko is not assuming IRSA at all, It is using the EC2 Instance IAM role. I have this issue with GitLab Runner created Pod, but for troubleshooting purposes I have created my own Pod and the issue is the same. In my case it is not happening just on GitLab Runner job pods.

My Pod spec:

apiVersion: v1
kind: ConfigMap
metadata:
  name: kaniko
  namespace: gitlab-runner
data:
  Dockerfile: |
    FROM debian:latest
    RUN rm -rf /aws /usr/local/aws-cli
    RUN apt-get update && apt-get install -y --no-install-recommends \
      less \
      ca-certificates \
      curl \
      unzip \
      && curl -sS "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" \
      && unzip awscliv2.zip \
      && ./aws/install
    RUN /usr/local/bin/aws --version
    RUN AWS_PAGER="" /usr/local/bin/aws sts get-caller-identity
---
apiVersion: v1
kind: Pod
metadata:
  labels:
    run: kaniko
  name: kaniko
  namespace: gitlab-runner
spec:
  serviceAccount: gitlab-runner
  serviceAccountName: gitlab-runner
  containers:
    - name: kaniko
      image: gcr.io/kaniko-project/executor:latest
      args:
        - "--dockerfile=/workspace/Dockerfile"
        - "--no-push"
        - "--cache=false"
      env:
      - name: AWS_SDK_LOAD_CONFIG
        value: "true"
      - name: AWS_EC2_METADATA_DISABLED
        value: "true"
      volumeMounts:
        - name: dockerfile
          mountPath: /workspace
          readOnly: true
  volumes:
    - name: dockerfile
      configMap:
        name: kaniko

Running this Pod results in:

INFO[0035] Running: [/bin/sh -c AWS_PAGER="" /usr/local/bin/aws sts get-caller-identity] 
{
    "UserId": "[REDUCTED]",
    "Account": "[REDUCTED]",
    "Arn": "arn:aws:sts::[REDUCTED]:assumed-role/dev-karpenter-eks-node-group/i-0b8337aa48bfed98a"
}

instead of expected:

{
    "UserId": "[REDUCTED]",
    "Account": "[REDUCTED]",
    "Arn": "arn:aws:sts::[REDUCTED]:assumed-role/dev-gitlab-runner/botocore-session-1613826698"
}

Am I missing something from my spec for Kaniko to assume IRSA role?

Kaniko version: 1.22.0

[balonik]

I figured it out and ended up adding ARG and ENV into my Dockerfile

ARG AWS_ROLE_ARN
ARG AWS_WEB_IDENTITY_TOKEN_FILE
ENV AWS_ROLE_ARN=$AWS_ROLE_ARN
ENV AWS_WEB_IDENTITY_TOKEN_FILE=$AWS_WEB_IDENTITY_TOKEN_FILE

and adding --build-arg into container's args

- '--build-arg="AWS_ROLE_ARN=$AWS_ROLE_ARN"'
- '--build-arg="AWS_WEB_IDENTITY_TOKEN_FILE=$AWS_WEB_IDENTITY_TOKEN_FILE"'

[xavbourdeau]

It is working for me. I'm runnning kaniko from a EKS pod with IRSA enabled. Here is the policy that is attached to the irsa role, using image gcr.io/kaniko-project/executor:v1.23.2-debug

Note, "ecr:GetAuthorizationToken" has to be set to "*" resource, rest of action can be set to a specific repo

{
    {
        "Effect": "Allow",
        "Action": "ecr:GetAuthorizationToken",
        "Resource": "*"
    },
    {
      "Action": [
        "ecr:BatchCheckLayerAvailability",
        "ecr:BatchGetImage",
        "ecr:CompleteLayerUpload",
        "ecr:DescribeImages",
        "ecr:DescribeImageScanFindings",
        "ecr:DescribeRepositories",
        "ecr:GetDownloadUrlForLayer",
        "ecr:GetLifecyclePolicy",
        "ecr:GetLifecyclePolicyPreview",
        "ecr:GetRepositoryPolicy",
        "ecr:InitiateLayerUpload",
        "ecr:ListImages",
        "ecr:ListTagsForResource",
        "ecr:PutImage",
        "ecr:UploadLayerPart"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:ecr:REGION:AWS_ACCOUNT_ID:repository/REPO_NAME"
    }
}