Daily AIC guardrail's per-run listArtifacts fan-out drains the shared GITHUB_TOKEN core rate limit #42761

Summary

The daily AI-Credits (AIC) guardrail that runs in the activation job of every agentic-workflow run inspects prior workflow runs by calling GET /repos/{owner}/{repo}/actions/runs/{run_id}/artifacts per prior run (on cache-miss). Under high run frequency + cold cache, this produces an O(N×M) fan-out where N concurrent activations each inspect M prior runs, and it authenticates as github-actions[bot] against the repo's shared ~15,000/hr GITHUB_TOKEN core budget.

In a real incident this accounted for 7,771 of 7,807 (99.5%) of the /actions/runs/:run_id/artifacts calls in one hour (~9.6 list calls per run across ~807 distinct run IDs), consuming ~50% of the shared core budget. Because that budget is shared across all workflows in the repo, unrelated critical steps (e.g. Build-Provenance attestation persistence) were starved and failed with HTTP 403.

This is not the attestation step and not normal Actions artifact upload/download — uploads go through the separate Twirp / ACTIONS_RESULTS_URL path. It is purely the guardrail's usage-inspection reads.

Call path

• Compiler emits a "Check daily workflow token guardrail" step (id daily-effective-workflow-guardrail) in the activation job, gated only by if: ${{ env.GH_AW_MAX_DAILY_AI_CREDITS != '' }}, with default ${{ vars.GH_AW_DEFAULT_MAX_DAILY_AI_CREDITS || '5000' }} → on by default for essentially every run.
• check_daily_aic_workflow_guardrail.cjs main() lists the last 24h of completed runs (actions.listWorkflowRuns, per_page=100), then for each candidate run not in the local cache calls getRunAIC(...) → artifactClient.listArtifacts({ latest: true, findBy: { workflowRunId } }).
• artifact_client.cjs listArtifacts() issues GET /repos/{owner}/{repo}/actions/runs/{run_id}/artifacts?per_page=100&page=1 with header User-Agent: gh-aw-artifact-client and Authorization: Bearer <GITHUB_TOKEN>.
• The step uses github-token: ${{ secrets.GITHUB_TOKEN }} / GH_AW_GITHUB_TOKEN (i.e. github-actions[bot]), drawing on the shared repo core budget.

Why existing guards are insufficient

There are already three partial mitigations, so the cache is the intended defense — it just cold-misses badly under fan-out:

1. Per-workflow usage cache (loadAICUsageCache + actions/cache/restore with a prefix restore-keys): a cache hit skips listArtifacts + download. But actions/cache is branch-scoped, so pull_request runs on unique per-PR branches almost always miss.
2. Artifact-based cross-branch fallback (restore_aic_usage_cache_fallback.cjs): only searches MAX_RUNS_TO_SEARCH = 5 recent runs for the aic-usage-cache artifact — too narrow to reliably hit.
3. Rate-limit-aware cap (computeMaxInspectableRuns, RATE_LIMIT_RESERVE = 100, ESTIMATED_API_OPERATIONS_PER_RUN = 2): computed once up front from this job's rate-limit snapshot only. It has no cross-run/global coordination, so N concurrent activations each independently decide they have budget and collectively blow it. It is also never re-checked during the per-run inspection loop.

Proposed mitigations

1. Make the aggregate usage cache authoritative (O(1) read). The conclusion job already writes a rolling per-run usage entry (write_daily_aic_usage_cache.cjs) and uploads an aic-usage-cache artifact. Have the activation read that single aggregate blob once and trust it, replacing the per-run listArtifacts+download fan-out. Short-circuit inspection entirely when the aggregate is fresh (covers the 24h window). This turns O(M) per run into O(1).
2. Widen / harden the cross-branch fallback. Bump MAX_RUNS_TO_SEARCH (currently 5) and/or key the aic-usage-cache artifact by a stable, non-branch-scoped name so pull_request runs reliably hit it instead of falling through to per-run inspection.
3. In-loop rate-limit backoff. github_rate_limit_logger.cjs already captures x-ratelimit-* response headers; use remaining to back off during the per-run loop, not just at the start.
4. Cross-run coordination / jitter. Under bursts of concurrent activations, add jitter/throttling so simultaneous jobs don't all hit the API in the same second, and/or gate on a shared freshness signal so only one run refreshes the aggregate.
5. Consider an isolated token for the guardrail. Use a dedicated/elevated token (or app-installation token) so guardrail reads don't consume the same GITHUB_TOKEN core budget that attestation and other critical steps depend on — directly preventing cross-workflow starvation regardless of cache performance.
6. Surface a repo-wide guardrail cost metric. The guardrail already logs consumed per run (before/after rate-limit delta, added in Measure and log GitHub API rate limit around daily AIC guardrail #38256). Aggregate and surface this so regressions are caught by monitoring rather than by a 403.

Acceptance criteria

• On a warm aggregate cache, the activation guardrail performs at most one artifact download (the aggregate) and zero per-run listArtifacts calls.
• The cross-branch artifact fallback reliably resolves on pull_request branches (widen MAX_RUNS_TO_SEARCH and/or stable artifact key).
• The guardrail re-checks rate-limit headroom inside the inspection loop and backs off before exhausting the reserve.
• Concurrent activations cannot collectively exceed the reserve (cross-run coordination or shared-freshness short-circuit).
• Decision recorded on whether to isolate the guardrail's token from the shared GITHUB_TOKEN core budget.
• A repo-wide aggregate of guardrail API consumption is emitted for monitoring.

Notes / affected files

• actions/setup/js/check_daily_aic_workflow_guardrail.cjs
• actions/setup/js/artifact_client.cjs
• actions/setup/js/restore_aic_usage_cache_fallback.cjs
• actions/setup/js/write_daily_aic_usage_cache.cjs
• actions/setup/js/github_rate_limit_logger.cjs
• pkg/workflow/compiler_activation_job_builder.go (emits the guardrail + cache-restore steps)
• pkg/workflow/notify_comment.go (buildDailyAICUsageCacheSteps)

Related prior work

• Add actions/cache-based AIC usage cache to skip artifact downloads in daily guardrail #38856 — introduced the actions/cache-based AIC usage cache (core mitigation to extend)
• optimize: stop paginating listWorkflowRuns once 24h cutoff is reached #38779 — stop paginating listWorkflowRuns once the 24h cutoff is reached (bounds N)
• Measure and log GitHub API rate limit around daily AIC guardrail #38256 — measure/log GitHub API rate limit around the guardrail (basis for monitoring)
• fix: enforce 48h data retention on AIC usage cache entries #39084 / write usage cache entry even for AIC = 0 #39066 — 48h cache retention + writing AIC=0 entries (reduce re-download churn)
• fix: scope daily AIC guardrail to repo+workflow, not per user #37634 — scoped guardrail to repo+workflow (removed actor: filter), which widened the run set each check considers

Note: verified against main — the guardrail code path is unchanged between recent commits, so this is not fixed by bumping the pinned actions/setup ref; it needs the mitigations above.