`copilot-api.<subdomain>.ghe.com` — AWF's own derived Copilot target for GHEC data-residency tenants — is absent from the auth-prefix decision and always receives `Bearer`

[jaroslawgajewski]

The gap, in containers/api-proxy/providers/copilot-auth.js (v0.27.44)

deriveCopilotApiTarget() derives copilot-api.<subdomain>.ghe.com for a GHEC data-residency tenant:

const serverHost = classifyGithubServerHost(env);            // tenant.ghe.com -> { kind: 'ghec', subdomain: 'tenant' }
if (serverHost.kind === 'ghec') return `copilot-api.${serverHost.subdomain}.ghe.com`;

But that derived host is not represented anywhere in the prefix decision. GITHUB_TOKEN_PREFIX_COPILOT_TARGETS contains only the two github.com catalog hosts:

const GITHUB_TOKEN_PREFIX_COPILOT_TARGETS = new Set([
  'api.enterprise.githubcopilot.com',
  'api.business.githubcopilot.com',
]);

so for the GHEC-DR host the decision falls through to Bearer on every path:

AWF_PLATFORM_TYPE	trace	prefix
ghec (what the in-code comment says gh-aw injects on *.ghe.com runners)	not in the target set → env.AWF_PLATFORM_TYPE !== 'ghes' → return false	Bearer
unset (what we actually observe in compiled output — see below)	not in the target set → falls to isGhesInstance() → not api.enterprise… → classifyGithubServerHost().kind === 'ghec' !== 'ghes' → false	Bearer
ghes	isGhesInstance() line 225 → true	token

The prefix machinery models exactly two worlds — github.com catalog endpoints and GHES — and the GHEC-DR endpoint that AWF itself derives is in neither. The only way to get the token prefix onto it is to misdeclare the platform as ghes.

Secondary observation: the comment above requiresGithubTokenPrefix says gh-aw performs "automatic AWF_PLATFORM_TYPE=ghec injection on *.ghe.com runners". In gh-aw v0.82.1 and v0.85.4 compiled lock files we see zero occurrences of AWF_PLATFORM_TYPE, so either that injection happens elsewhere, no longer happens, or the comment is stale. Worth reconciling, since the ordering rationale in that comment depends on it.

Symptom this produces

Copilot inference returns a bare 400 Bad Request on *.ghe.com across five AWF versions — two independent failure layers, only one fixable by endpoint/credential configuration.

Environment

GitHub host	tenant.ghe.com — GitHub Enterprise Cloud with data residency, not GitHub Enterprise Server (classifyGithubServerHost() → {kind: 'ghec', subdomain: 'tenant'})
gh-aw compiler	v0.82.1 (production) and v0.85.4 (test)
AWF versions tested	v0.27.4, v0.27.13, v0.27.16, v0.27.31, v0.27.44
Engine	Copilot (copilot CLI 1.0.78 at v0.85.4)
Model	claude-sonnet-4.6
Copilot licensing	Business, via api.business.githubcopilot.com
Workaround in place	AWF pinned fleet-wide to v0.25.55, which works

Symptom

On every AWF v0.27.x we have tested, the Copilot harness exhausts all retries and the agent job fails. Each attempt ends:

400 400 400 Bad Request
Changes    +0 -0
Duration   19s

Notably, the harness's own error classifiers do not recognise it:

failureClass=partial_execution
isCAPIError400=false      isHTTP400ResponseError=false
isAuthError=false         isAuthenticationFailedError=false
isModelNotSupportedError=false     tokenCount=0
stdout=0B stderr=124B

Historically this surfaced as 400 bad request: Authorization header is badly formatted. On v0.27.44 that string no longer appears anywhere in the logs — only the bare 400 Bad Request above. The failure mode has changed shape while remaining fatal.

Endpoint and credential are ruled out by a complete 2×2

Four controlled runs on the same AWF v0.27.44 build, varying only the Copilot endpoint and the credential.
Everything else — firewall allowlist, JSON config, sandbox topology, model — held constant.

endpoint	GH_AW_ENGINE_API_HOSTS	credential	/models	completion
api.business.githubcopilot.com	4 github.com hosts	github.com Business PAT	400	400
copilot-api.tenant.ghe.com (forced)	4 github.com hosts	tenant-issued	200 (21 models)	400
copilot-api.tenant.ghe.com (via engine.api-target)	narrowed to that host	github.com Business PAT	401	400
copilot-api.tenant.ghe.com (via engine.api-target)	narrowed to that host	tenant-issued	200 (20 models)	400

The last row is the configuration your own data-residency guide
prescribes — engine.api-target set in frontmatter, no COPILOT_API_TARGET override, a tenant-issued Copilot
credential. /models authenticates cleanly and completions still fail. So the completion 400 is independent
of both the endpoint and the credential.

Two side findings from that matrix, both actionable independently of the main bug:

• A github.com Business-licensed PAT gets 401 from the tenant Copilot endpoint. That is a hard requirement worth
  stating explicitly in the data-residency guide: the credential must be issued from the tenant, not just the
  endpoint changed. The guide currently says only "The workflow uses existing engine credentials".
• Row 1 versus row 3 shows the two misconfigurations mask each other: wrong endpoint yields 400, and correcting
  only the endpoint yields 401. Anyone debugging by changing one variable at a time will conclude both are wrong
  and neither is fixable.

Two independent layers

We isolated the endpoint/credential variable by rewriting only COPILOT_API_TARGET and the Copilot credential in the two Execute GitHub Copilot CLI env blocks, on the same AWF v0.27.44 build. Everything else — the firewall allowlist, the JSON config, the sandbox topology — was identical.

	api.business.githubcopilot.com + github.com-Business token	copilot-api.tenant.ghe.com + GHE-issued token
/models via the AWF api-proxy	HTTP 400 — awf-reflect: models fetch returned 400 for http://api-proxy:10002/models	HTTP 200, 21 models, 16.8 KB
AWF run-summary "Available models"	— (empty)	claude-haiku-4.5, claude-opus-4.6, claude-opus-4.7, … (20)
token-tracker-audit.jsonl	no successful entry	{"provider":"copilot","path":"/models","status":200}
Copilot CLI outcome	400 400 400 Bad Request ×4	400 400 400 Bad Request ×4 — unchanged

Layer 1 — the model-listing 400 is a genuine endpoint/credential mismatch and is fixed by pairing the GHE-tenant Copilot endpoint with a GHE-issued credential. This layer was previously invisible, because the CLI's exit code is the same either way.

Layer 2 — inference still fails. tokenCount=0, and token-tracker-audit.jsonl records no completion request at all (only /models). But the squid access log shows 11 successful CONNECT … 200 TCP_TUNNEL:HIER_DIRECT tunnels to copilot-api.tenant.ghe.com:443, so completion traffic does leave the sandbox and reach upstream, and is rejected inside TLS where the proxy cannot see the response body.

Ruled out

• Model availability — claude-sonnet-4.6 is present in awf-reflect.json in the passing-/models configuration.
• Firewall/domain blocking — *.tenant.ghe.com is in GH_AW_ALLOWED_DOMAINS; all 11 CONNECTs return 200 TCP_TUNNEL, 0 denied.
• The token-prefix work — copilot-auth.js is semantically unchanged v0.27.31 → v0.27.44, and #5872 (Business token prefix, v0.27.26+) was already present in v0.27.31, which we tested live and which failed identically.
• X-GitHub-Api-Version on /models (#6695, v0.27.43) — present in the v0.27.44 build that still fails.
• AWF generally — the Gemini/Vertex engine runs green on native v0.27.44 in the same repo, same branch, same PR. This is specific to the Copilot path.
• Sandbox topology — the failing run's resolved config shows enableHostAccess=false, localhostDetected=false, enableApiProxy=true, enableTokenSteering=true, i.e. a clean native configuration with no downgrade leftovers.