GCP Workload identity federation on Gitlab passing authentication between jobs

GCP Workload Identity Federation on Gitlab Passing Authentication between Jobs

Gitlab (late 2022) is relatively new to workload identity federation, and there are not many good templates or guides out there. The official guides explain how to set up the federation pool and authenticate with it, but not really how to use this in an enterprise pipeline. A basic question that pops up all the time is how to pass authentication downstream to other jobs within your pipeline.

For example – lets say you are doing infrastructure as code and have a relatively simple workflow that looks like this:

authenticate → validate → plan → apply

A naive approach would be to authenticate and assume validate/plan/apply also knows that context. It does not…

If you were using credentials as a secret it would be available to all the 4 jobs, and a common thing to do would be to set GOOGLE_APPLICATION_CREDENTIALS to that JSON, and then all jobs would be authenticated by default.

But when we are using workload identity federation we no longer have that secret — in fact — we are creating that secret when we authenticate! If you are used to Github actions, this is just how they do it and doing it on Gitlab should be simple… right?

Not really.

At the time of writing there are no stackoverflows, blog posts or guides that shows how to do this in a good way. The official Gitlab example is quite simplistic and assumes you should authenticate in each step.

So how do we solve this?

The concept is this: When we authenticate using workload identity federation with gcloud we actually get a JSON (like a service account) and a token back that we can use downstream in our pipeline. What we need to do is pass the JSON and token downstream as artifacts.

1. Authenticate — using gcloud Docker image, write files to shared folder
2. validate — using Terraform Docker image, use credentials from share
3. plan — using Terraform Docker image, use credentials from share
4. apply — using Terraform Docker image., use credentials&plan from share

To do this I am defining an _auth/ shared folder to pass between jobs (Gitlab term — artifacts).

Authentication

I have created an authentication job that uses gcloud to authenticate.

This job needs 3 parameters (environment variables)

SERVICE_ACCOUNT_EMAIL — is the email address of a service account (like name@gcp-project.iam.gserviceaccount.com)

WORKLOAD_IDENTITY_PROVIDER — on this form: projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/POOL_NAME/providers/PROVIDER_ID

GOOGLE_CLOUD_PROJECT — the GCP projectid to set as default.

.gcp-auth:
  image: "google/cloud-sdk:slim"
  artifacts:
    paths:
      - _auth/
  script:
    - |
      if [ -z "${SERVICE_ACCOUNT_EMAIL}" ]; then
        echo "MISSING SERVICE_ACCOUNT_EMAIL variable. (expected: full email of service account)"
        exit 1
      fi      
      if [ -z "${WORKLOAD_IDENTITY_PROVIDER}" ]; then
        echo "MISSING WORKLOAD_IDENTITY_PROVIDER variable (expected: projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/POOL_NAME/providers/PROVIDER_ID)"
        exit 1
      fi      
      if [ -z "${GOOGLE_CLOUD_PROJECT}" ]; then
        echo "MISSING GOOGLE_CLOUD_PROJECT variable.  (expected: gcp project id)"
        exit 1
      fi      
      mkdir -p _auth 
      echo "$CI_JOB_JWT_V2" > $CI_PROJECT_DIR/_auth/.ci_job_jwt_file
      echo "$GOOGLE_CLOUD_PROJECT" > $CI_PROJECT_DIR/_auth/.GOOGLE_CLOUD_PROJECT
      gcloud iam workload-identity-pools create-cred-config 
      $WORKLOAD_IDENTITY_PROVIDER 
      --service-account=$SERVICE_ACCOUNT_EMAIL 
      --service-account-token-lifetime-seconds=600 
      --output-file=$CI_PROJECT_DIR/_auth/.gcp_temp_cred.json 
      --credential-source-file=$CI_PROJECT_DIR/_auth/.ci_job_jwt_file
      gcloud config set project $GOOGLE_CLOUD_PROJECT
    - !reference [.gcp_ensure_auth_script, before]
    - "gcloud auth login --cred-file=$GOOGLE_APPLICATION_CREDENTIALS"

as you can see, it sets up _auth/ folder as an artifact to pass between jobs, and dumps project id, token and JWT into this folder.

Terraform

To simplify the Terraform steps I have created a base job for Terraform that is aware of gcp auth state, that I just need to extend in my main pipeline. This gives me the flexibility of doing whatever in my Terraform scripts. I could for example, create a one-step plan & apply instead of splitting up in 3 parts like I am doing here.

.terraform:
  extends: .gcp-auth
  image:
    name: hashicorp/terraform:1.3.5
    entrypoint:
      - '/usr/bin/env'
      - 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
  artifacts:
    paths:
      - planfile
      - _auth/
  before_script:
    - !reference [.gcp_ensure_auth_script, before]
    - *tf_before_script

Example pipeline

This example should be working more or less out of the box if you have set up the workload identity federation. Replace variables with something that fits your use case!

include:
  - local: '/.gitlab-gcp-auth.yml'

stages:
  - auth
  - validate
  - plan
  - apply

auth:
  extends: .gcp-auth
  stage: auth
  variables:
     SERVICE_ACCOUNT_EMAIL: "terraform@foo.iam.gserviceaccount.com"
     WORKLOAD_IDENTITY_PROVIDER: "projects/123123/locations/global/workloadIdentityPools/gitlab-pool/providers/gitlab-provider"
     GOOGLE_CLOUD_PROJECT: "example"

validate:
  extends: .terraform
  stage: validate
  variables:
     TF_BACKEND_CONFIG: "bucket=yoursecretstatebucket"
  script:
    - terraform validate
  only:
    refs:
      - main

plan:
  extends: .terraform
  stage: plan
  variables:
     TF_BACKEND_CONFIG: "bucket=your-terraform-state"
  script:
    - terraform plan -out ../planfile -input=false -var-file=env/main.tfvars
  dependencies:
    - validate
  only:
    refs:
      - main

apply:
  extends: .terraform
  stage: apply
  variables:
     TF_BACKEND_CONFIG: "bucket=your-terraform-state"
  script:
    - terraform apply -input=false ../planfile -var-file=env/main.tfvars
  dependencies:
    - plan
    - auth
  when: manual
  only:
    refs:
      - main

.gitlab-gcp-auth.yml

I have included the full file here for convenience. Feel free to cut and paste and use as you like.

.gcp_ensure_auth_script:
  before:
    - |
      if [ ! -d "$CI_PROJECT_DIR/_auth" ]; then
         echo "Missing auth folder, please make sure your job defines _auth as an artifact!"
         exit 1
      fi
    - export GOOGLE_APPLICATION_CREDENTIALS=$CI_PROJECT_DIR/_auth/.gcp_temp_cred.json
    - export CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE=$CI_PROJECT_DIR/_auth/.gcp_temp_cred.json
    - export GOOGLE_GHA_CREDS_PATH=$CI_PROJECT_DIR/_auth/.gcp_temp_cred.json
    - export GOOGLE_CLOUD_PROJECT=$(cat $CI_PROJECT_DIR/_auth/.GOOGLE_CLOUD_PROJECT)
    - export CLOUDSDK_PROJECT=$(cat $CI_PROJECT_DIR/_auth/.GOOGLE_CLOUD_PROJECT)
    - export CLOUDSDK_CORE_PROJECT=$(cat $CI_PROJECT_DIR/_auth/.GOOGLE_CLOUD_PROJECT)
    - export GCP_PROJECT=$(cat $CI_PROJECT_DIR/_auth/.GOOGLE_CLOUD_PROJECT)
    - export GCLOUD_PROJECT=$(cat $CI_PROJECT_DIR/_auth/.GOOGLE_CLOUD_PROJECT)

.gcp-auth:
  image: "google/cloud-sdk:slim"
  artifacts:
    paths:
      - _auth/
  script:
    - |
      if [ -z "${SERVICE_ACCOUNT_EMAIL}" ]; then
        echo "MISSING SERVICE_ACCOUNT_EMAIL variable. (expected: full email of service account)"
        exit 1
      fi      
      if [ -z "${WORKLOAD_IDENTITY_PROVIDER}" ]; then
        echo "MISSING WORKLOAD_IDENTITY_PROVIDER variable (expected: projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/POOL_NAME/providers/PROVIDER_ID)"
        exit 1
      fi      
      if [ -z "${GOOGLE_CLOUD_PROJECT}" ]; then
        echo "MISSING GOOGLE_CLOUD_PROJECT variable.  (expected: gcp project id)"
        exit 1
      fi      
      mkdir -p _auth 
      echo "$CI_JOB_JWT_V2" > $CI_PROJECT_DIR/_auth/.ci_job_jwt_file
      echo "$GOOGLE_CLOUD_PROJECT" > $CI_PROJECT_DIR/_auth/.GOOGLE_CLOUD_PROJECT
      gcloud iam workload-identity-pools create-cred-config 
      $WORKLOAD_IDENTITY_PROVIDER 
      --service-account=$SERVICE_ACCOUNT_EMAIL 
      --service-account-token-lifetime-seconds=600 
      --output-file=$CI_PROJECT_DIR/_auth/.gcp_temp_cred.json 
      --credential-source-file=$CI_PROJECT_DIR/_auth/.ci_job_jwt_file
      gcloud config set project $GOOGLE_CLOUD_PROJECT
    - !reference [.gcp_ensure_auth_script, before]
    - "gcloud auth login --cred-file=$GOOGLE_APPLICATION_CREDENTIALS"


.gcp-ensure-auth:
  artifacts:
    paths:
      - _auth/
  before_script:
    - !reference [ .gcp_ensure_auth_script, before ]

.tf_before_script: &tf_before_script
  - |
    if [ -z "${TF_BACKEND_CONFIG}" ]; then
       echo "Missing TF_BACKEND_CONFIG, expected bucket=value"
       exit 1
    fi
  - rm -rf .terraform
  - terraform --version
  - terraform init -backend-config="$TF_BACKEND_CONFIG"


.terraform:
  extends: .gcp-auth
  image:
    name: hashicorp/terraform:1.3.5
    entrypoint:
      - '/usr/bin/env'
      - 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
  artifacts:
    paths:
      - planfile
      - _auth/
  before_script:
    - !reference [.gcp_ensure_auth_script, before]
    - *tf_before_script

Note that this example only is aware of one environment, so you have to adjust for a real use case. You would also want to set a lifetime of your _auth artifacts.

Please clap if this is of use to you, or give a comment if there are better ways to do this!

Links i checked:

I did quite a bit of googling before writing my own, so all of these links inspired me in some way.

• 403 trying to run terraform from Gitlab without json file
• Configure OpenID Connect with GCP Workload Identity Federation | GitLab
• Using GitLab CI with GCP Workload Identity Federation.
• Gitlab and Workload Identity Federation on Google Cloud
• Integrate Gitlab with Google Cloud workload identity federation

---

GCP Workload identity federation on Gitlab passing authentication between jobs was originally published in Compendium on Medium, where people are continuing the conversation by highlighting and responding to this story.