Skip to content

[Phase 1] ADR: agent rate-limit & circuit-breaker taxonomy (decision record) #637

Closed

Description

@don-petry

Story

As a platform engineer / org-lead,
I want a decision record that fixes the per-agent-type control taxonomy, the token-budget breaker semantics, the telemetry source, and the public/private enforcement boundary,
so that the follow-on config, gate, and wiring stories encode a verified mechanism instead of guessing APIs or baking in unconfirmed numbers.

Acceptance Criteria

  1. A new ADR at docs/initiatives/agent-rate-limits-adr.md exists, mirroring the Pull-Request-Limits ADR structure (Status/Date/Epic/Story header, Context with measured baseline, numbered Decision, mechanism), and is documentation-only — it mutates no config, workflow, or repo setting.
  2. The ADR enumerates the in-scope agent types (dev-lead, compliance-audit, feature-ideation, initiative-driver) and, per type, the control dimensions to bound: max concurrent runs, max runtime, cooldown between runs, daily execution budget, and a consecutive-failure circuit breaker (open -> backoff -> half-open).
  3. The ADR specifies the token-budget circuit breaker requested in the discussion: pause new agentic dispatch when the rolling 5-hour Claude subscription budget reaches a configurable threshold (default 90%), with Claude-backed agents prioritized on recovery.
  4. The ADR names the telemetry source(s) exposing the 5-hour Claude budget and per-agent execution/timestamp state, or records that no native GitHub surface exists and selects a concrete alternative, citing first-hand doc/source verification per the org 'never guess an API' rule.
  5. The ADR records the cross-repo enforcement boundary: which controls land in the public standard/gate (this epic) vs. which require a companion change in the private repo where dev-lead's engine and token telemetry live.
  6. The ADR states the initiative-level success metrics and the operational cost cap the standard installs (per epic body), so Phases 2-5 inherit them.

Tasks / Subtasks

Dev Notes

Project Structure Notes

ADRs live under docs/initiatives/ in the target public repo (petry-projects/.github), alongside pull-request-limits-adr.md. Documentation-only change; no code or config touched.

References

  • docs/initiatives/pull-request-limits-adr.md (petry-projects/.github — ADR structure and staged-epic template to mirror)
  • standards/pr-limits.md (petry-projects/.github — human-readable standard companion pattern)
  • AGENTS.md
  • CLAUDE.md
  • scripts/engine.sh (private repo — 5-hour subscription-cap handling, issue Monitor and Optimize Token Usage #206, cross-repo)

Likely target surface

  • docs/initiatives/agent-rate-limits-adr.md (new — the decision record)

Story prepared by the BMAD Scrum Master (Bob) for epic #636. Status: ready-for-dev.

Activity

  1. don-petry commented on Aug 20, 2026

    @don-petry
    ContributorAuthor

    Telemetry-source finding — resolves AC #4 (first-hand verified, 2026-08-20)

    AC #4 asks this ADR to name the telemetry source exposing the Claude subscription budget, or record that none exists — citing first-hand verification, per the org "never guess an API" rule. A source does exist. It was probed live rather than inferred from documentation, and the probe returned HTTP 200 with real window state.

    The source

    GET https://api.anthropic.com/api/oauth/usage
      Authorization: Bearer <CLAUDE_CODE_OAUTH_TOKEN>
      anthropic-beta: oauth-2025-04-20
      User-Agent: claude-code/<version>
      Content-Type: application/json
    

    This is the server-side data that backs Claude Code's /usage screen. The User-Agent header is load-bearing, not cosmetic — without a claude-code/<version> value the caller lands in an aggressively rate-limited bucket and gets persistent 429s. Rate limiting is per access token, not per account; polling at ≥180 s intervals is safe, so an hourly monitor sits far inside the budget.

    Observed response shape (probe of 2026-08-20, per-surface buckets elided)

    {
      "five_hour": { "utilization": 3.0,  "resets_at": "2026-08-21T03:29:59Z", "limit_dollars": null },
      "seven_day": { "utilization": 85.0, "resets_at": "2026-08-25T15:59:59Z", "limit_dollars": null },
      "seven_day_opus": null, "seven_day_sonnet": null,   // per-model weekly buckets, null on this plan
      "extra_usage": { "is_enabled": false, "monthly_limit": 100, "used_credits": 171.0,
                       "utilization": 100.0, "spend_limit_reached": true },
      "limits": [
        { "kind": "session",       "group": "session", "percent": 3,   "severity": "normal",
          "resets_at": "2026-08-21T03:29:59Z", "scope": null, "is_active": false },
        { "kind": "weekly_all",    "group": "weekly",  "percent": 85,  "severity": "warning",
          "resets_at": "2026-08-25T15:59:59Z", "scope": null, "is_active": false },
        { "kind": "weekly_scoped", "group": "weekly",  "percent": 100, "severity": "critical",
          "resets_at": "2026-08-25T15:59:59Z", "scope": { "model": { "display_name": "…" } },
          "is_active": true }
      ]
    }

    Three things worth pinning into the decision record:

    1. The limits[] array is the better contract than the top-level keys. It carries a server-computed severity (normal / warning / critical), an is_active flag, and a scope object, so the gate can consume a graded signal instead of re-deriving one from a bare percentage. The top-level five_hour / seven_day objects are the same data flattened; prefer limits[] and treat the top-level keys as the fallback for older response shapes.

    2. Both windows this epic cares about come from one call. kind: "session" is the rolling 5-hour window Phase 5 ([Phase 5] Token-budget circuit breaker: pause at the configured % of the 5-hour Claude window #641) breaks on; kind: "weekly_all" is the 7-day window. One adapter, one HTTP request, both breakers — which is the main argument for resolving the source here in the ADR rather than per-story.

    3. weekly_scoped must not trip a fleet pause. The probe caught a per-model weekly limit already at 100% / critical / is_active: true while the account-wide weekly window was only at 85%. The correct response to a scoped exhaustion is the engine's existing model-fallback chain (swap models), not stopping the fleet. Only weekly_all and session are pause-worthy. Encoding this distinction in the ADR prevents Phase 5 and the follow-on story from independently getting it wrong.

    resets_at is authoritative — do not hardcode the reset window

    The probe returned seven_day.resets_at = 2026-08-25T15:59:59Z, which is Tuesday 10:59:59 a.m. America/Chicago — matching the maintainer's understanding of the reset (Tuesday ~11 a.m. Central) without it being configured anywhere. Any threshold arithmetic should derive time-until-reset from this field, so the behaviour self-corrects if Anthropic ever shifts the window. No cron expression or timezone constant should encode the reset day.

    Risks this ADR must record

    • The endpoint is undocumented and unsupported. It is not in the public API reference; upstream feature requests for a supported surface are open (anthropics/claude-code#44328, #32796). It can change shape or disappear without notice. This is precisely the risk the adapter boundary in [Phase 5] Token-budget circuit breaker: pause at the configured % of the 5-hour Claude window #641 AC Add multi-agent isolation strategy using git worktrees #2 already anticipates — keep it, and keep the fail-safe.
    • Recommended fail-safe: allow-with-warning. On a non-200, a parse failure, or a missing limits[] entry, the gate should log a warning and permit dispatch rather than defer. Deferring on telemetry failure converts an outage of an undocumented third-party endpoint into a full fleet stop — the exact failure mode Agentic Activity Circuit Breakers & Org-Wide Rate-Limiting Standard #636 exists to prevent. The account-level protection of last resort is unchanged: the engine still reacts to a real cap with the exit-2 fallback chain.
    • Credential type is not yet verified. The probe used the short-lived OAuth access token from the local credential store. CI authenticates with the long-lived claude setup-token credential. Both carry the same sk-ant-oat01- prefix and both are OAuth-issued, so acceptance is likely but unproven. Verify by running the monitor in dry-run in CI against the existing secret — not by minting a fresh setup-token locally, which risks invalidating the credential the fleet currently runs on.

    Cross-repo boundary note (AC #5)

    Flipping a fleet-wide pause switch needs more privilege than the fleet's usual token carries. Writing an org-level Actions variable returns 403 — You must be an org admin or have the actions variables fine-grained permission for a classic token holding repo / workflow / read:org, even when the caller is an org admin; it needs admin:org (classic) or the fine-grained org Variables: write permission. Repo-level variables are writable under plain repo scope. So the boundary is: the public repo owns the config and the gate; the private repo owns the poller and the credential; and an org-variable-write credential is a genuine new prerequisite, not an existing capability to be assumed.

    Recommendation

    Select GET /api/oauth/usage as the telemetry source for the whole epic, behind a single adapter that normalizes limits[] into { kind, percent, severity, resets_at, is_active }, with allow-with-warning degradation. That satisfies AC #4 for both the 5-hour breaker (#641) and the 7-day glide-path breaker filed as its follow-on, and it removes the "no source may exist" contingency that #641 AC #6 was written to absorb — #641 can now target a real source instead of a coarse proxy.

  2. don-petry commented on Aug 21, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead Implementation Plan

    Issue: #637 — [Phase 1] ADR: agent rate-limit & circuit-breaker taxonomy (decision record)

    Scope

    Add a documentation-only decision record at docs/initiatives/agent-rate-limits-adr.md, mirroring the Pull-Request-Limits ADR structure, that fixes the per-agent-type control taxonomy, the token-budget circuit-breaker semantics, the verified telemetry source, and the public/private enforcement boundary — mutating no config, workflow, or repo setting.

    Implementation Checklist

    Tests to Write

    • None — this is a pure-documentation change with no observable behavioral surface (Phase 2 exception per the dev-lead process). Verification is the repo lint (.dev-lead/scripts/dev-lead-lint.sh), which validates shell/agent/CODEOWNERS surfaces this change does not touch.

    Test Command

    bash .dev-lead/scripts/dev-lead-lint.sh

  3. don-petry commented on Aug 21, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead: Implementation Complete

    Plan Execution

    Test Results

    This is a pure-documentation change (a decision record) with no observable behavioral surface, so the TDD phase is skipped per the dev-lead process exception. Verification was by lint:

    === dev-lead-lint.sh ===
      [lint] shellcheck on 29 file(s)...
      [lint] validating .github/CODEOWNERS...
    OK: .github/CODEOWNERS
      [lint] all checks passed
    DEVLEAD_EXIT=0
    
    === markdownlint-cli2 (CI parity, .markdownlint-cli2.yaml) — docs/initiatives/agent-rate-limits-adr.md ===
    Summary: 0 issues in 0 files
    MDLINT_EXIT=0
    

    All internal doc links were verified to resolve to real files in-repo.

    Telemetry verification (AC #4 — "never guess an API surface")

    Four vendor surfaces were fetched first-hand (2026-08-21) and read for a rate-over-window / 5-hour-budget field:

    • GitHub Actions concurrency — a single-lane serializer (caps simultaneity, not rate); cancel-in-progress is the dev-lead: issue-pickup runs cancelled by concurrency group → 111 labeled issues stuck with no PR #402 lever. No per-hour/daily execution budget, no run-count-over-time limit.
    • Claude Messages API rate limits — per-minute token-bucket (RPM/ITPM/OTPM) via anthropic-ratelimit-* headers + monthly spend cap. No 5-hour window, no subscription-budget-% surface.
    • Claude Rate Limits API — returns configured ceilings, not live remaining budget.
    • Claude Pro/Max subscription — confirms shared usage limits + interactive /status, but no pollable programmatic budget endpoint.

    Conclusion: no native surface exposes the rolling 5-hour subscription budget. Selected alternative: derive it source-side in the private engine (scripts/engine.sh, #206) as a private rolling-window ledger the proactive breaker reads; prefer anthropic-ratelimit-tokens-remaining where an API key (not the subscription OAuth token) is used.

    Files Changed

    • docs/initiatives/agent-rate-limits-adr.md (new): the decision record. Documentation-only — no config, workflow, or repo setting touched.

    Notes

    • All §6 / §7 / §9 numeric values are explicitly marked proposals pending human sign-off, per the epic Agentic Activity Circuit Breakers & Org-Wide Rate-Limiting Standard #636 gate — mirroring the Pull-Request-Limits ADR's provisional-numbers posture.
    • Design is cross-repo by necessity: the token-budget breaker's telemetry can only be built in petry-projects/.github-private; this is recorded as a named follow-on dependency (§8), not a blocker for this ADR.
    • No follow-up issues required to close this story.
  4. added a commit that references this issue on Aug 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions