The fleet and every session share one GitHub API quota (don-petry, 5,000/hour) and the fleet alone uses about that much: it ran out on 2026-10-10 and failed a workflow #2197

Problem

The automation fleet and every interactive session authenticate to the GitHub REST API as the same user, don-petry, so they share one quota of 5,000 requests an hour. The fleet alone uses about that much. Anything extra exhausts it, and then every workflow that calls the API with that token fails until the hourly reset.

Evidence (2026-10-10)

• 12:33:02Z: quota exhausted. gh returned API rate limit exceeded for user ID 36422719; gh api rate_limit showed core: 0/5000, reset at 12:34:10Z.

• A workflow failed because of it. Run 38051651146 (Persona Runtime, repository_dispatch) ended at 12:33:03Z with ##[error]posting token authenticates as '{ "message": "API rate limit exceeded for user ID 36422719 ….

• Baseline burn rate, measured after the reset with almost no session traffic (about 3 calls a minute from the session measuring it):

  Time (UTC)	Remaining
  12:34:22	4,997
  12:42:15	4,097
  12:42:28	4,067
  12:42:40	4,045

  That is roughly 95 to 115 requests a minute, or 5,700 to 6,900 an hour, against a limit of 5,000.

• What pushed it over this time: an interactive session was polling canary-rollout runs and downloading their logs every 10 to 15 seconds for about an hour. The window ran out one minute before its reset.

• A second-order failure: the session's own guard read the quota error as "the next tag changed" and cancelled a release promotion that was in flight.

Why it matters

• At the measured rate the quota can run out on its own late in an hour, with no session involved. Every dev-lead, pr-review and persona run that starts in that gap fails or misbehaves, and the failures look unrelated to their cause.
• Scripts that treat any API error as a meaningful answer (a missing tag, an empty list, "not found") make wrong decisions during the gap.

Acceptance criteria

1. Measure who spends the quota. Break the hourly usage down by workflow, and list the top consumers (for example: listing all PR comments on every pass, repeated gh run view --log, polling loops).
2. Bring the fleet's steady-state usage under a stated budget, for example 60% of the limit, by cutting or caching the top consumers, moving read-only calls to the workflow's own GITHUB_TOKEN where that works, or moving the fleet to a GitHub App installation token with its own quota.
3. Fail loudly and distinctly on quota exhaustion. A shared helper recognises the rate-limit response and reports it as such; no script treats it as an empty or negative answer. Add a bats test with a gh stub that returns the 403 rate-limit body.
4. Preflight. Long-running or write-heavy runs check the remaining quota at the start and defer, with a clear message, when it is below a floor.
5. Visibility. The health report shows remaining quota and the day's lowest point.

Notes

• Sessions can read tag and branch state over git (git ls-remote), which does not spend the REST quota.