Repository navigation
[Phase 1] ADR: agent rate-limit & circuit-breaker taxonomy (decision record) #637
Copy link
Copy link
Closed
Labels
Description
Story
As a platform engineer / org-lead,
I want a decision record that fixes the per-agent-type control taxonomy, the token-budget breaker semantics, the telemetry source, and the public/private enforcement boundary,
so that the follow-on config, gate, and wiring stories encode a verified mechanism instead of guessing APIs or baking in unconfirmed numbers.
Acceptance Criteria
- A new ADR at docs/initiatives/agent-rate-limits-adr.md exists, mirroring the Pull-Request-Limits ADR structure (Status/Date/Epic/Story header, Context with measured baseline, numbered Decision, mechanism), and is documentation-only — it mutates no config, workflow, or repo setting.
- The ADR enumerates the in-scope agent types (dev-lead, compliance-audit, feature-ideation, initiative-driver) and, per type, the control dimensions to bound: max concurrent runs, max runtime, cooldown between runs, daily execution budget, and a consecutive-failure circuit breaker (open -> backoff -> half-open).
- The ADR specifies the token-budget circuit breaker requested in the discussion: pause new agentic dispatch when the rolling 5-hour Claude subscription budget reaches a configurable threshold (default 90%), with Claude-backed agents prioritized on recovery.
- The ADR names the telemetry source(s) exposing the 5-hour Claude budget and per-agent execution/timestamp state, or records that no native GitHub surface exists and selects a concrete alternative, citing first-hand doc/source verification per the org 'never guess an API' rule.
- The ADR records the cross-repo enforcement boundary: which controls land in the public standard/gate (this epic) vs. which require a companion change in the private repo where dev-lead's engine and token telemetry live.
- The ADR states the initiative-level success metrics and the operational cost cap the standard installs (per epic body), so Phases 2-5 inherit them.
Tasks / Subtasks
- Draft docs/initiatives/agent-rate-limits-adr.md mirroring the Pull-Request-Limits ADR structure; documentation only (AC: Addressing PR comments #1)
- Enumerate in-scope agent types and per-type control dimensions in a decision table (AC: Add multi-agent isolation strategy using git worktrees #2)
- Specify token-budget breaker semantics: 5-hour window, default 90% threshold, Claude priority on recovery (AC: Add coding standards & principles section to AGENTS.md #3)
- Verify and record the telemetry source for the 5-hour budget and per-agent state, citing sources; if none native, select a concrete alternative (AC: Add workflow, environment, and orchestration guidance #4)
- Document the public/private cross-repo enforcement boundary (AC: Add stacked PR strategy and Epic-level workflow guidance #5)
- Pin the initiative success metrics and cost cap into the ADR (AC: feat: add Structured Logging and CQRS standards #6)
Dev Notes
- Mirror the proven Pull-Request-Limits ADR shape and staging: epic Enable GitHub pull request limits to cut automation-PR noise #505 split into ADR ([Phase 1] ADR: pull-request-limits mechanism + policy decision #506) -> config single-source-of-truth ([Phase 2a] Encode the PR-limit config as a machine-readable single source of truth #507) -> source-side apply path ([Phase 3] Apply path: enforce the limit on target repo(s) + tests #508). This epic reuses that three-layer split, then adds two wiring stories. Use the target-repo ADR docs/initiatives/pull-request-limits-adr.md (petry-projects/.github) as the template — it opens with a Status/Date/Epic/Story header, a 'Context — why this decision is needed' section with a measured baseline, and a numbered 'Decision (summary)'.
- The Pull-Request-Limits ADR's core lesson: GitHub exposes NO native 'max open PRs' surface, so enforcement is source-side. Apply the same skepticism here — GitHub exposes no native per-workflow rate limiter and no 5-hour-budget surface; verify against docs before assuming one exists. Org rule: 'API surfaces and SHAs are looked up, never guessed' (see CLAUDE.md and AGENTS.md).
- The 5-hour Claude subscription budget is a rolling window shared across all Claude models (the private repo's scripts/engine.sh subscription-cap handling, issue Monitor and Optimize Token Usage #206). Today the engine only reacts AFTER a rate-limit (exit-2 walks the model fallback chain); nothing proactively tracks '% of the 5-hour budget consumed'. A proactive breaker needs a telemetry source that does not exist in the public repo yet — this ADR must resolve where it comes from.
- The three motivating bugs (dev-lead: issue-pickup runs cancelled by concurrency group → 111 labeled issues stuck with no PR #402 concurrency-group cancellation -> 111 stuck issues, fix(dev-lead): multi-label issue creation can cancel the dev-lead dispatch (race) #443 dispatch race, feature-ideation: discussion-triggered run fails at workflow setup (zero jobs) — discussion→dispatch bridge #571 feature-ideation zero-job dispatch) are symptoms of missing throughput controls; cite them as the baseline the standard must prevent, not as blockers to fix here.
- OWASP ASI08 (Denial of Service & Resource Exhaustion) from the OWASP Top 10 for Agentic Applications (2026) is the external framing; name it in the Context section.
Project Structure Notes
ADRs live under docs/initiatives/ in the target public repo (petry-projects/.github), alongside pull-request-limits-adr.md. Documentation-only change; no code or config touched.
References
- docs/initiatives/pull-request-limits-adr.md (petry-projects/.github — ADR structure and staged-epic template to mirror)
- standards/pr-limits.md (petry-projects/.github — human-readable standard companion pattern)
- AGENTS.md
- CLAUDE.md
- scripts/engine.sh (private repo — 5-hour subscription-cap handling, issue Monitor and Optimize Token Usage #206, cross-repo)
Likely target surface
docs/initiatives/agent-rate-limits-adr.md (new — the decision record)
Story prepared by the BMAD Scrum Master (Bob) for epic #636. Status: ready-for-dev.
Reactions are currently unavailable
Activity
don-petry commentedon Aug 20, 2026
Telemetry-source finding — resolves AC #4 (first-hand verified, 2026-08-20)
AC #4 asks this ADR to name the telemetry source exposing the Claude subscription budget, or record that none exists — citing first-hand verification, per the org "never guess an API" rule. A source does exist. It was probed live rather than inferred from documentation, and the probe returned
HTTP 200with real window state.The source
This is the server-side data that backs Claude Code's
/usagescreen. TheUser-Agentheader is load-bearing, not cosmetic — without aclaude-code/<version>value the caller lands in an aggressively rate-limited bucket and gets persistent 429s. Rate limiting is per access token, not per account; polling at ≥180 s intervals is safe, so an hourly monitor sits far inside the budget.Observed response shape (probe of 2026-08-20, per-surface buckets elided)
{ "five_hour": { "utilization": 3.0, "resets_at": "2026-08-21T03:29:59Z", "limit_dollars": null }, "seven_day": { "utilization": 85.0, "resets_at": "2026-08-25T15:59:59Z", "limit_dollars": null }, "seven_day_opus": null, "seven_day_sonnet": null, // per-model weekly buckets, null on this plan "extra_usage": { "is_enabled": false, "monthly_limit": 100, "used_credits": 171.0, "utilization": 100.0, "spend_limit_reached": true }, "limits": [ { "kind": "session", "group": "session", "percent": 3, "severity": "normal", "resets_at": "2026-08-21T03:29:59Z", "scope": null, "is_active": false }, { "kind": "weekly_all", "group": "weekly", "percent": 85, "severity": "warning", "resets_at": "2026-08-25T15:59:59Z", "scope": null, "is_active": false }, { "kind": "weekly_scoped", "group": "weekly", "percent": 100, "severity": "critical", "resets_at": "2026-08-25T15:59:59Z", "scope": { "model": { "display_name": "…" } }, "is_active": true } ] }Three things worth pinning into the decision record:
The
limits[]array is the better contract than the top-level keys. It carries a server-computedseverity(normal/warning/critical), anis_activeflag, and ascopeobject, so the gate can consume a graded signal instead of re-deriving one from a bare percentage. The top-levelfive_hour/seven_dayobjects are the same data flattened; preferlimits[]and treat the top-level keys as the fallback for older response shapes.Both windows this epic cares about come from one call.
kind: "session"is the rolling 5-hour window Phase 5 ([Phase 5] Token-budget circuit breaker: pause at the configured % of the 5-hour Claude window #641) breaks on;kind: "weekly_all"is the 7-day window. One adapter, one HTTP request, both breakers — which is the main argument for resolving the source here in the ADR rather than per-story.weekly_scopedmust not trip a fleet pause. The probe caught a per-model weekly limit already at 100% /critical/is_active: truewhile the account-wide weekly window was only at 85%. The correct response to a scoped exhaustion is the engine's existing model-fallback chain (swap models), not stopping the fleet. Onlyweekly_allandsessionare pause-worthy. Encoding this distinction in the ADR prevents Phase 5 and the follow-on story from independently getting it wrong.resets_atis authoritative — do not hardcode the reset windowThe probe returned
seven_day.resets_at = 2026-08-25T15:59:59Z, which is Tuesday 10:59:59 a.m. America/Chicago — matching the maintainer's understanding of the reset (Tuesday ~11 a.m. Central) without it being configured anywhere. Any threshold arithmetic should derive time-until-reset from this field, so the behaviour self-corrects if Anthropic ever shifts the window. No cron expression or timezone constant should encode the reset day.Risks this ADR must record
limits[]entry, the gate should log a warning and permit dispatch rather than defer. Deferring on telemetry failure converts an outage of an undocumented third-party endpoint into a full fleet stop — the exact failure mode Agentic Activity Circuit Breakers & Org-Wide Rate-Limiting Standard #636 exists to prevent. The account-level protection of last resort is unchanged: the engine still reacts to a real cap with the exit-2 fallback chain.claude setup-tokencredential. Both carry the samesk-ant-oat01-prefix and both are OAuth-issued, so acceptance is likely but unproven. Verify by running the monitor in dry-run in CI against the existing secret — not by minting a fresh setup-token locally, which risks invalidating the credential the fleet currently runs on.Cross-repo boundary note (AC #5)
Flipping a fleet-wide pause switch needs more privilege than the fleet's usual token carries. Writing an org-level Actions variable returns
403 — You must be an org admin or have the actions variables fine-grained permissionfor a classic token holdingrepo/workflow/read:org, even when the caller is an org admin; it needsadmin:org(classic) or the fine-grained org Variables: write permission. Repo-level variables are writable under plainreposcope. So the boundary is: the public repo owns the config and the gate; the private repo owns the poller and the credential; and an org-variable-write credential is a genuine new prerequisite, not an existing capability to be assumed.Recommendation
Select
GET /api/oauth/usageas the telemetry source for the whole epic, behind a single adapter that normalizeslimits[]into{ kind, percent, severity, resets_at, is_active }, with allow-with-warning degradation. That satisfies AC #4 for both the 5-hour breaker (#641) and the 7-day glide-path breaker filed as its follow-on, and it removes the "no source may exist" contingency that #641 AC #6 was written to absorb — #641 can now target a real source instead of a coarse proxy.don-petry commentedon Aug 21, 2026
Dev-Lead Implementation Plan
Issue: #637 — [Phase 1] ADR: agent rate-limit & circuit-breaker taxonomy (decision record)
Scope
Add a documentation-only decision record at
docs/initiatives/agent-rate-limits-adr.md, mirroring the Pull-Request-Limits ADR structure, that fixes the per-agent-type control taxonomy, the token-budget circuit-breaker semantics, the verified telemetry source, and the public/private enforcement boundary — mutating no config, workflow, or repo setting.Implementation Checklist
docs/initiatives/agent-rate-limits-adr.mdwith Status/Date/Epic/Story header + "scope: documentation only" banner (AC Addressing PR comments #1)/statusonly), select concrete alternative, cite sources (AC Add workflow, environment, and orchestration guidance #4)Tests to Write
.dev-lead/scripts/dev-lead-lint.sh), which validates shell/agent/CODEOWNERS surfaces this change does not touch.Test Command
bash .dev-lead/scripts/dev-lead-lint.shdon-petry commentedon Aug 21, 2026
Dev-Lead: Implementation Complete
Plan Execution
docs/initiatives/agent-rate-limits-adr.mdwith Status/Date/Epic/Story header + documentation-only scope banner (AC Addressing PR comments #1)Test Results
This is a pure-documentation change (a decision record) with no observable behavioral surface, so the TDD phase is skipped per the dev-lead process exception. Verification was by lint:
All internal doc links were verified to resolve to real files in-repo.
Telemetry verification (AC #4 — "never guess an API surface")
Four vendor surfaces were fetched first-hand (2026-08-21) and read for a rate-over-window / 5-hour-budget field:
concurrency— a single-lane serializer (caps simultaneity, not rate);cancel-in-progressis the dev-lead: issue-pickup runs cancelled by concurrency group → 111 labeled issues stuck with no PR #402 lever. No per-hour/daily execution budget, no run-count-over-time limit.anthropic-ratelimit-*headers + monthly spend cap. No 5-hour window, no subscription-budget-% surface./status, but no pollable programmatic budget endpoint.Conclusion: no native surface exposes the rolling 5-hour subscription budget. Selected alternative: derive it source-side in the private engine (
scripts/engine.sh, #206) as a private rolling-window ledger the proactive breaker reads; preferanthropic-ratelimit-tokens-remainingwhere an API key (not the subscription OAuth token) is used.Files Changed
docs/initiatives/agent-rate-limits-adr.md(new): the decision record. Documentation-only — no config, workflow, or repo setting touched.Notes
petry-projects/.github-private; this is recorded as a named follow-on dependency (§8), not a blocker for this ADR.fix(adr): correct AC #4 — the subscription budget IS queryable [skip …
fix(adr): add languages to fenced code blocks (MD040) [skip ci-relay]
docs: resolve rolling-vs-fixed window contradiction in rate-limit ADR
docs: correct ledger error direction; split credential type from quot…
feat: implement issue #637 — [Phase 1] ADR: agent rate-limit & circui…