Repository navigation
Agentic Activity Circuit Breakers & Org-Wide Rate-Limiting Standard #636
Description
Summary
Define and enforce an org-wide standard for execution-rate limits, circuit breakers, and resource budgets for autonomous CI agents (dev-lead, compliance-audit, feature-ideation, initiative-driver), so a runaway or cascading failure is bounded systematically instead of patched per-incident. This installs the throughput controls that sit below the recently completed PR-limits queue-depth control (#505): PR-limits bounds how many PRs are open; this bounds how fast / how often agents act, plus a token-budget breaker that pauses agentic work before the shared Claude budget is exhausted.
Motivation
Three real production failures show the gap — all symptoms of agents operating without throughput controls, deadline awareness, or graceful degradation:
| Bug | Symptom | Root cause |
|---|---|---|
| #402 | 111 labeled issues stuck with no PR | dev-lead issue-pickup cancelled by concurrency group — no queue/retry |
| #443 | Multi-label issue creation cancels dev-lead dispatch | dispatch race — no serialization/backoff |
| #571 | feature-ideation discussion-triggered run fails at setup (zero jobs) | dispatch bridge with no error budget/retry |
External framing: OWASP ASI08 — Denial of Service & Resource Exhaustion (OWASP Top 10 for Agentic Applications, 2026) names agentic resource exhaustion a top-10 risk. GitHub's Workflow Execution Protections (2026 preview) add actor/event controls but no rate limiting. The discussion also raises token-budget depletion (undocumented 5-hour Claude subscription-budget exhaustion) as a primary throttling metric.
Approach
Mirror the proven Pull-Request-Limits pattern (#505), which split cleanly into an ADR (#506) → a machine-readable single-source-of-truth config (#507) → a source-side apply-path gate (#508). This epic reuses that three-layer split and adds two wiring stories:
- ADR / decision record — fix the per-agent control taxonomy + token-budget breaker + telemetry source + cross-repo boundary.
standards/agent-rate-limits.json+ companion doc — the single source of truth, inert until signed off.scripts/lib/agent-rate-limit.sh— a reusable, sourced admission + circuit-breaker gate, guard-only.- Wire the gate into the public-repo PR/agent-creating workflows (feature-ideation, initiative-driver).
- Token-budget breaker — pause dispatch at the configured % of the rolling 5-hour Claude budget.
Scope — controls per agent type
Max concurrent runs, max runtime, cooldown between runs, daily execution budget, consecutive-failure circuit breaker (open → backoff → half-open), and an org-wide token-budget breaker. In-scope agent types: dev-lead, compliance-audit, feature-ideation, initiative-driver.
Initiative success metrics
- Zero recurrences over 90 days post-rollout of the three failure signatures: concurrency-cancellation issue starvation (dev-lead: issue-pickup runs cancelled by concurrency group → 111 labeled issues stuck with no PR #402-class), dispatch races (fix(dev-lead): multi-label issue creation can cancel the dev-lead dispatch (race) #443-class), and zero-job dispatches (feature-ideation: discussion-triggered run fails at workflow setup (zero jobs) — discussion→dispatch bridge #571-class).
- Zero full-fleet blocks from 5-hour Claude-budget exhaustion once the token breaker is live — it pauses at the configured threshold (default 90%), leaving headroom for humans and priority Claude work.
- 100% of in-scope PR/agent-creating workflows read their limits from
standards/agent-rate-limits.json(no hardcoded throttles) and are gated by the shared library.
Cost bound
The initiative installs the org's operational cost ceiling: the per-agent daily_run_budget plus the cap on the rolling 5-hour Claude subscription budget (default pause at 90%) are the enforced spend bounds. The initiative's own delivery adds no new paid infrastructure (shell + JSON config + bats tests); dev-lead implementation runs are bounded by the existing per-tier engine timeouts and, once live, by these very rate limits.
Relationship to existing work
Complements — does not duplicate — PR-limits (#505): queue-depth vs. execution-rate. #402/#443/#571 are the motivating baseline this standard must prevent; they are cited as evidence, not fixed as prerequisites here.
Untracked prerequisites
- Human sign-off on the configured limit values (max concurrent, runtime, cooldown, daily budget, token threshold) before any gate enforces them — mirrors the pr-limits sign-off gate ([Phase 3 gate] Human sign-off: final PR-limit caps + exempt list + rollout scope #566). The Phase-2 config lands inert.
- Cross-repo enforcement companion in petry-projects/.github-private: dev-lead's reusable workflow, engine.sh tier routing, and the token-metrics telemetry live there, so wiring the gate into dev-lead and sourcing the 5-hour token budget requires a change outside this public-repo epic.
- A telemetry source for the rolling 5-hour Claude subscription budget — no native GitHub surface exposes it today; the Phase-1 ADR selects one (the Actions Data Stream 2026 roadmap item, or the private repo's token-metrics JSONL adapted to a windowed %).
Planned from idea discussion #586 by the BMAD Scrum Master initiative-planner. Inert until a maintainer adds initiative:auto.
Activity
don-petry commentedon Aug 20, 2026
Epic update — Phase 6 added, and the Phase-1 telemetry unknown is resolved.
100 − 2 × days_until_reset(Monday 98%, Sunday 96%, floor 86% just after reset). Complements [Phase 5] Token-budget circuit breaker: pause at the configured % of the 5-hour Claude window #641 the way this epic complements PR-limits: [Phase 5] Token-budget circuit breaker: pause at the configured % of the 5-hour Claude window #641 bounds a burst within a 5-hour window; [Phase 6] Weekly-budget glide-path breaker: pause at 100 − 2×days-until-reset of the 7-day Claude window #994 bounds the shape of the week, so the fleet cannot spend the whole weekly budget in the first two days and leave humans with nothing.AGENTS_PAUSEDswitch.The 5-hour-budget telemetry prerequisite in this body can be checked off.
GET https://api.anthropic.com/api/oauth/usage— the data behind Claude Code's/usagescreen — was probed first-hand and returns both the 5-hour and 7-day windows, with server-computed severity and exactresets_attimestamps, in one call. Full contract and risk notes are on #637. It is undocumented and unsupported, so the adapter boundary and fail-safe already specified in #641 stay exactly as written.Two prerequisites remain genuinely open, both on #1565: whether the long-lived
claude setup-tokencredential is accepted by that endpoint (verify via dry-run in CI, not by minting a new token), and whether an org-variable-write permission exists — writing an org Actions variable needsadmin:orgor the fine-grained org Variables:write permission, which the fleet's usualrepo/workflowscopes do not cover.don-petry commentedon Sep 1, 2026
Maintainer sign-off on the configured limit values — 2026-09-01
This clears the epic's untracked prerequisite #1 ("Human sign-off on the configured limit values before any gate enforces them"). Every value below is signed off as written in
standards/agent-rate-limits.json. Enumerated in full so the record is auditable rather than a blanket approval.Per-agent-type limits (ADR §6)
dev-leadcompliance-auditfeature-ideationinitiative-driverOrg-wide token budget (ADR §7)
session— 5-hour fixed window, pause-worthy, threshold 90%. Signed off.weekly_all— 7-day glide path (reserve_pct_per_day: 2,floor_pct: 86,ceiling_pct: 100). Values signed off, butenabledstaysfalse. Arming the weekly breaker is a separate, later decision — do not flip it as part of Phase 4.weekly_scoped— correctlypause_worthy: false; per-model exhaustion stays with the engine's model-fallback chain.claude_priority: true— signed off.resume_pctkey, per the fixed-window semantics pinned on [Phase 2] standards/agent-rate-limits.json config + standard doc (single source of truth) #638. Do not add one.Exemptions
Actors
dependabot[bot],OrganizationAdmin,@petry-projects/org-leads,dependabot-automerge-petry; labelsecurity. Signed off as written.What this authorizes
statusmay move fromprovisionalto signed-off as part of the Phase-4 wiring PR (#640), with this comment cited as the sign-off record — not by a direct push tomain. Enforcement is still canary-scoped per #640 AC4:initiative-driverfirst, every other agent type log-only until promoted.What this does NOT authorize
org_wide.limits.weekly_all(enabledstaysfalse).dev-lead,compliance-audit, orfeature-ideationbefore theinitiative-drivercanary has been observed.Values may be revised at any time before the gate goes enforcing; revisions supersede this comment and must be recorded the same way.
don-petry commentedon Sep 4, 2026
Initiative driver — stale in-flight slot(s) detected
The following open sub-issue(s) still carry the
dev-leadlabel but havehad no dev-lead activity for ≥ 72h, so they are occupying an in-flight
slot without progressing: #640.
No labels were changed. A human should decide whether to split-and-close, retry,
or park each one so the slot is freed for the rest of the epic.
don-petry commentedon Sep 7, 2026
Closing — all five phases delivered and verified on
main(2026-09-07)standards/agent-rate-limits.json+ standard docscripts/lib/agent-rate-limit.shgate libraryVerified against
origin/main, not inferred from issue statescripts/agent-rate-limit-gate.shis present..github/workflows/feature-ideation-reusable.yml— the reusable, not the thin caller stub — and into bothstandards/workflows/initiative-driver.yml(the canonical template) and.github/workflows/initiative-driver.yml, kept together so the next stub resync cannot revert it. This was the defect that made the original [Phase 4] Wire the gate into the PR/agent-dispatch workflows #640 unsafe to auto-implement; the rewrite fixed it and the implementation followed it.--mode log-onlyis the default; onlyinitiative-driverenforces (pure bash, no model spend, the direct fix(dev-lead): multi-label issue creation can cancel the dev-lead dispatch (race) #443 target). Every other agent type computes and logs a decision but never acts. The log-only → enforce promotion path is documented in the script header.canary-rollout.sh_run_jsonprimitive rather than a file-backed counter — the ephemeral-runner problem that would have made a persisted counter unable to ever trip.statusis nowsigned-off, citing the maintainer sign-off comment on this issue (2026-09-01).org_wide.token_budget.limits.weekly_all.enabledis stillfalse. Arming the weekly glide-path breaker remains a separate, later decision, exactly as the sign-off reserved. Noresume_pctkey was introduced.Review-thread note
PR #1059's two
codeant-aifindings were both false positives, refuted in-thread with executable evidence (the. as $rbinding makes$set | index($r.conclusion)resolve correctly;argate_concurrentexcludesGITHUB_RUN_IDandargate_fetch_runsrequestsdatabaseId, so the gate does not count itself). Worth recording because a real self-count would have deadlocked everyinitiative-driverdispatch atmax_concurrent_runs: 1.The PR then sat merge-ready for five days behind two stale cancelled
Dev-Lead Agentruns rendering as red checks — auto-review defers silently on red non-required checks (#1427) and nothing re-triggers on an unchanged SHA (#1551). Cleared with an empty commit; merged 2026-09-07T15:07.Nothing actionable remains. Closing.