Skip to content

[Phase 6] Weekly-budget glide-path breaker: pause at 100 − 2×days-until-reset of the 7-day Claude window #994

Description

@don-petry

Story

As a platform engineer / org-lead,
I want the agent fleet to pause automatically when the rolling 7-day Claude subscription budget crosses a threshold that tightens as the weekly reset approaches — reserving ~2% of the weekly budget per remaining day — so that the fleet cannot burn the whole week's budget in the first two days,
so that a human always has usable Claude headroom for the rest of the week, and the maintainer no longer has to notice budget pressure manually and withhold a secret to stop the bleeding.

Context — why this is separate from #641

Phase 5 (#641) breaks on the 5-hour window at a static threshold (default 90%). That protects against a short burst, but it is blind to the shape of the week: the fleet can sit under 90% of every 5-hour window and still exhaust the 7-day budget by Thursday, leaving four days with nothing for humans.

This story adds the orthogonal control — a time-varying threshold on the 7-day window:

pause when  weekly_all.percent  >=  100 - (reserve_pct_per_day * days_until_weekly_reset)

With reserve_pct_per_day = 2 and the reset derived from the API's resets_at (Tuesday ~11:00 America/Chicago, confirmed by live probe — see #637):

Day (relative to reset) Days left Pause at
Tuesday (reset day) 0 100%
Monday 1 98%
Sunday 2 96%
Saturday 3 94%
Friday 4 92%
Thursday 5 90%
Wednesday 6 88%
Tuesday (just reset) 7 86%

The floor of 86% immediately after a reset is intentional: it is the "don't spend the whole week on day one" guard. The ceiling of 100% on reset day is also intentional: with the budget minutes from refilling, there is nothing left to protect.

Motivating observation (2026-08-20): a live probe of the usage endpoint showed the account-wide weekly window at 85% with five days left before reset — one point under this story's Thursday threshold, with a model-scoped weekly limit already exhausted. The #641 5-hour breaker would not have fired at any point on that path.

Prerequisites

Acceptance Criteria

  1. The gate evaluates a 7-day glide-path threshold from an org_wide.token_budget.seven_day block in standards/agent-rate-limits.json — at minimum enabled, reserve_pct_per_day (default 2), floor_pct, and ceiling_pct. No threshold is hardcoded, and the schedule is expressible purely as config.
  2. days_until_reset is derived from the telemetry payload's resets_at, never from a hardcoded weekday, cron expression, or timezone constant. Partial days round up, so the day before reset evaluates as 1 day left (98%), not 0.
  3. The evaluation keys on the account-wide weekly window (limits[].kind == "weekly_all"). A weekly_scoped (per-model) limit at critical / is_active must not trip a fleet pause — the correct response to scoped exhaustion is the engine's existing model-fallback chain. This distinction is asserted by a test.
  4. On trip, the pause is surfaced through the existing human-clearable marker/label convention used by the rest of the library, and the reason is machine-readable (which window, observed percent, threshold, reset timestamp) rather than prose buried in a log line.
  5. Telemetry failure (non-200, malformed body, missing weekly_all entry) degrades to allow-with-warning per the ADR's documented default — an outage of an undocumented third-party endpoint must never stop the fleet.
  6. tests/test_agent_rate_limit_weekly_glide.bats exercises a mocked telemetry source: threshold at each day-offset 0–7, boundary conditions (exactly at threshold trips; one point under does not), partial-day rounding, scoped-limit-must-not-trip (AC Add coding standards & principles section to AGENTS.md #3), and telemetry-unavailable fail-safe (AC Add stacked PR strategy and Epic-level workflow guidance #5). No live network.
  7. Rollout is dry-run first — the gate logs the decision it would have made for a full week before it can defer anything — and activation respects the epic's Phase-2 human sign-off gate.

Tasks / Subtasks

Dev Notes

  • Pause semantics: the breaker gates new dispatch only. In-flight runs finish under the existing per-tier engine timeouts — killing work mid-flight wastes the tokens already spent and leaves PRs in a partial state.
  • Auto-clear is the subtle half. The pause must record that it set AGENTS_PAUSED, so the auto-resume after the weekly reset clears only its own pause and never a pause a human set deliberately. A companion marker (e.g. AGENTS_PAUSE_SOURCE=weekly-glide) is the cheapest way; the resume path checks it before clearing. Without this, an automated resume silently overrides a maintainer's decision — the inverse of the failure .github-private#1525 was filed to fix.
  • Hysteresis: once tripped, do not un-trip merely because usage percentage ticks back under the line (it will not — utilization is monotonic within a window). The only clean resume is the window reset, which is exactly what resets_at gives.
  • Non-goal: this story does not change model selection, the engine fallback chain, or the 5-hour breaker.
  • TDD (AGENTS.md): unit tests mock the telemetry source; no live network.

Project Structure Notes

Extends scripts/lib/agent-rate-limit.sh (Phase 3) and standards/agent-rate-limits.json (Phase 2), reusing Phase 5's telemetry adapter. New test under tests/. The poller and the org-variable write are cross-repo (private).

References

Likely target surface

  • standards/agent-rate-limits.json (extend — seven_day glide block)
  • scripts/lib/agent-rate-limit.sh (extend — weekly glide evaluation)
  • tests/test_agent_rate_limit_weekly_glide.bats (new)

Filed as the follow-on to #641 for epic #636. Inert until a maintainer arms the epic.


⚠️ Post-ADR / post-#641 clarification (added 2026-08-31 — read before implementing)

Phases 1–5 are all merged: the ADR (#637/PR #995), the config (#638/PR #1017), the gate library (#639/PR #1018), and the token-budget breaker (#641/PR #1020). This story's ACs are already consistent with the ADR — the notes below are about reusing what now exists rather than rebuilding it.

1. The adapter already exists and already serves this window — extend it

#641 landed with the adapter built to serve both session and weekly_all from a single call (ADR §4.1: "Both breakers share one call"), specifically so this story extends rather than refactors. scripts/lib/agent-rate-limit.sh on main now carries the arl_token_* family, including:

  • percent extraction preferring the limits[] array with the flattened five_hour/seven_day keys as fallback — weekly_all already maps to seven_day;
  • an is_active check and the pause_worthy scope guard (weekly_scoped is false in config);
  • fail-safe degradation to allow on malformed/missing/unreadable telemetry, plus a timeout on the telemetry command;
  • a stale-429 guard — a cached 429 envelope whose retry_until (or observed_at + retry_after) has passed is treated as expired degraded telemetry and allows, rather than deferring forever.

Add the glide-path threshold on top of this. Do not introduce a second telemetry path, a second fail-safe convention, or a second percent extractor.

2. Terminology

The story says "rolling 7-day". The weekly_all window is a fixed window with an authoritative resets_at (ADR §4.1) — it accumulates until the reset; old usage does not age out continuously. Please use "fixed" / "weekly window" in code comments and the standard doc. (AC #2's round-up days_until_reset derivation from resets_at is already correct and unaffected.)

3. What makes this breaker genuinely different — keep it explicit

Unlike the session breaker, this one can close before resets_at — but never because utilization fell. The glide-path threshold 100 - (reserve_pct_per_day * days_until_reset) rises as the reset approaches, so the breaker re-evaluates each poll and may close because the threshold moved past a static percent. State that in the code comment; it is the single most misread part of the design, and conflating it with the session breaker's "resets_at only" rule in either direction is the bug to avoid. There is still no percentage resume mark for either breaker.

4. reserve_pct_per_day is already in config

standards/agent-rate-limits.json carries org_wide.token_budget.limits.weekly_all.reserve_pct_per_day (default 2), inert pending sign-off. Read it; never hardcode. Its _note already records the rising-threshold semantics.

5. Actuation stays out of scope

AC-wise this story is the gate/threshold. Writing the org-level AGENTS_PAUSED variable, the poller, and the AGENTS_PAUSE_SOURCE marker are the private companion (.github-private#1565), which additionally needs admin:org or fine-grained org Variables: write — plain repo/workflow/read:org gets 403 even for an org admin. The auto-clear invariant in the Dev Notes (an automated resume must never clear a pause a human set) still belongs to that companion; keep this story's surface to the decision.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions