extproc: GCP ADC (external_account) auth fails with "context canceled" after the first config reload

[kisalay-hub]

Description:

With type: GCPCredentials and only projectName + region set (the Application Default Credentials path from #1979), the extproc builds the GCP handler once per configuration load and keeps creds.TokenSource from google.FindDefaultCredentials(ctx, …) (internal/backendauth/gcp.go:52-56). That token source uses the ctx it was created with for every later token exchange.

The handler is created inside LoadConfig, which the bundle watcher calls with a per-tick context that it cancels as soon as the load returns (internal/filterapi/watcher.go: perTickCtx, cancel := context.WithTimeout(ctx, tick) … cw.loadConfig(perTickCtx) … cancel()). Only the initial load at startup uses the root context.

Result: after the first config reload, every request to a GCP backend fails with HTTP 500 and

failed to get GCP access token: oauth2/google: unable to generate access token: … Post "https://sts.googleapis.com/v1/token": context canceled

until the proxy pod is restarted.

Who is affected: ADC credentials of type external_account — Workload Identity Federation from EKS/AKS/self-hosted Kubernetes with GOOGLE_APPLICATION_CREDENTIALS pointing at a credential configuration file (Google's documented method). externalaccount.NewTokenSource(ctx, …) keeps the context for its HTTP calls.

Not affected: GKE Workload Identity — the metadata-server token source (computeSource.Token()) takes no context, which is why this was not seen there. AWS (aws.go) — credentialsProvider.Retrieve(ctx) uses the request context.

Expected: authentication keeps working across configuration reloads; the token source must outlive the load that created it.

Repro steps:

1. Run the gateway on a non-GKE cluster with Workload Identity Federation: GOOGLE_APPLICATION_CREDENTIALS=/etc/gcp/config.json (an external_account credential configuration whose credential_source.file is a projected service-account token), and a BackendSecurityPolicy with type: GCPCredentials, gcpCredentials: {projectName, region} — no credentialsFile, no workloadIdentityFederationConfig.
2. Send a chat completion to a Vertex backend → 200.
3. Change any AIServiceBackend / BackendSecurityPolicy / model override so the filter config bundle is rewritten (extproc logs loading a new bundled config).
4. Send the same request → 500 with the log above. Restart the proxy pod → 200 again until the next reload.

Unit reproduction: build the handler with a context, cancel it, call Do against an httptest STS. Fails on main (7d7c07f) with the same error; passes with the fix. Included in the PR.

Environment:

Envoy AI Gateway v1.1.0 (controller and envoyproxy/ai-gateway-extproc:v1.1.0), Envoy Gateway v1.8.1, EKS ap-south-1, Vertex AI asia-south1. The code is unchanged on main.

Logs:

extproc:

time=2026-09-22T17:34:07.475Z level=ERROR msg="error processing request message" error="cannot process request headers: failed to do auth request: failed to get GCP access token: oauth2/google: unable to generate access token: Post \"https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/<sa>:generateAccessToken\": oauth2/google: invalid response from Secure Token Server: Post \"https://sts.googleapis.com/v1/token\": context canceled"

Envoy:

[warning][ext_proc] Received gRPC error on stream: 2, message error processing request message: cannot process request headers: failed to do auth request: failed to get GCP access token: …

Access log: response_code=500 response_flags=- duration=48ms (the exchange fails immediately; it is a cancelled context, not a timeout).

Proposed fix: detach the token source from the load context (context.WithoutCancel(ctx) in newGCPHandler) — PR to follow. Longer term, cloud.google.com/go/auth (already in go.mod) exposes TokenProvider.Token(ctx) per call, which would let Do pass the request context instead.