ci: GCP Workload Identity Federation pool/provider missing — Push API Image fails #94

Activity

1. don-petry

   added

   bugBug reportsBug reports

   compliance-auditAutomated compliance audit findingAutomated compliance audit finding

   on Apr 8, 2026
2. don-petry commented on Apr 8, 2026

   

   don-petry

   on Apr 8, 2026

   ContributorAuthor

   More actions

   Confirmed: GCP-side, not a secret issue

   Verified all three GCP secrets are still set in the repo:

   $ gh secret list --repo petry-projects/broodly | grep GCP_
   GCP_PROJECT_ID                  2026-03-31T20:28:17Z
   GCP_SERVICE_ACCOUNT             2026-03-31T20:28:18Z
   GCP_WORKLOAD_IDENTITY_PROVIDER  2026-03-31T20:28:17Z
   

   The failure (The target service indicated by the "audience" parameters is invalid. This might either be because the pool or provider is disabled or deleted or because it doesn't exist.) is purely GCP-side: the Workload Identity Pool github-actions and/or its provider github no longer exists in project broodly-491920.

   Recovery runbook

   The setup is documented in infra/terraform/GCP_SETUP.md. Run these in order from any machine with gcloud Owner-level auth on broodly-491920:

   Step 1 — Verify the current state

   # Does the pool exist?
   gcloud iam workload-identity-pools describe github-actions \
     --project=broodly-491920 --location=global
   
   # Does the provider exist?
   gcloud iam workload-identity-pools providers describe github \
     --project=broodly-491920 --location=global \
     --workload-identity-pool=github-actions

   If either errors with NOT_FOUND, that's your culprit.

   Step 2 — Recreate whatever's missing

   If the pool is gone:

   gcloud iam workload-identity-pools create "github-actions" \
     --project="broodly-491920" \
     --location="global" \
     --display-name="GitHub Actions"

   If the provider is gone (or after recreating the pool):

   gcloud iam workload-identity-pools providers create-oidc "github" \
     --project="broodly-491920" \
     --location="global" \
     --workload-identity-pool="github-actions" \
     --display-name="GitHub" \
     --attribute-mapping="google.subject=assertion.sub,attribute.repository=assertion.repository" \
     --issuer-uri="https://token.actions.githubusercontent.com"

   Step 3 — Re-grant the IAM binding

   (If the service account binding was lost when the pool was deleted, re-add it)

   gcloud iam service-accounts add-iam-policy-binding \
     "broodly-ci@broodly-491920.iam.gserviceaccount.com" \
     --project="broodly-491920" \
     --role="roles/iam.workloadIdentityUser" \
     --member="principalSet://iam.googleapis.com/projects/922040293304/locations/global/workloadIdentityPools/github-actions/attribute.repository/petry-projects/broodly"

   Step 4 — Validate

   Push an empty commit (or re-run the failed workflow):

   git -C ~/repos/broodly commit --allow-empty -m "chore: re-trigger CI after WIF reconciliation"
   git -C ~/repos/broodly push

   The next Push API Image run should succeed.

   Why this is GCP-side, not a code fix

   There's no Terraform module in this repo for the WIF resources (grep -rln workload_identity_provider infra/terraform/ --include=*.tf finds nothing), so the pool/provider live exclusively as imperative gcloud state. That's tracked in the doc but not version-controlled. Strong recommendation: file a separate issue to import the existing WIF resources into Terraform (or recreate them as IaC) so this never happens again — a deleted pool would then be detected on the next terraform plan.

   What I can autonomously do

   Nothing further on this issue without GCP Owner credentials. Marking as needs-user.
3. don-petry commented on Apr 9, 2026

   

   don-petry

   on Apr 9, 2026

   ContributorAuthor

   More actions

   Resolved! This check is now passing as of 2026-04-09. Closing automatically.
4. don-petry

   closed this as completedon Apr 9, 2026