[Phase 2] Assess simplifying the sonnet->opus in-engine fallback given rate-limit equalization #1099

Activity

1. github-actions

   added

   initiativeEpic / initiative tracking issueEpic / initiative tracking issue

   on Jul 4, 2026
2. github-actions

   added a parent issue on Jul 4, 2026

   • Initiative: Integrate Claude Sonnet 5 into engine.sh model chains — capture introductory pricing, eval-gate quality, canary-promote #1095
3. github-actions

   marked this as blocked on Jul 4, 2026

   • [Phase 2] Wire claude-sonnet-5 into the Claude model chains as a candidate #1098
4. don-petry

   added

   dev-leadFor dev-lead agent pickupFor dev-lead agent pickup

   on Jul 8, 2026
5. don-petry commented on Jul 8, 2026

   

   don-petry

   on Jul 8, 2026

   Collaborator

   More actions

   Dev-Lead Implementation Plan

   Issue: #1099 — [Phase 2] Assess simplifying the sonnet->opus in-engine fallback given rate-limit equalization

   Scope

   Analysis-first story. Assess whether Sonnet 5's rate-limit equalization (Sonnet/Haiku now match Opus limits at every tier) makes the in-engine sonnet->opus RPM/TPM fallback hop redundant. Preliminary conclusion: not redundant — equalization changes per-model bucket size, not bucket independence, and layer-1 recovery depends on independence. Chains stay unchanged; rationale recorded durably.

   Analysis (AC #1, #2)

   • The sonnet->opus hop lives in CLAUDE_ACTION_MODEL_CHAIN=claude-sonnet-4-6,claude-opus-4-8 (engine.sh:154).
   • Per the two-layer doc (engine.sh:13-22): layer-1 in-Claude chain helps ONLY with per-model TPM/RPM buckets ("each Claude model has its own TPM/RPM bucket, so swapping models often recovers"); it does NOT help the shared daily subscription cap (feat(dev-lead): proactive provider headroom check before engine invocation #206). Layer-2 cross-provider fallback (exit code 2) handles exhaustion.
   • Equalization makes each model's RPM/TPM bucket the same size; it does NOT merge them into one shared bucket. A saturated sonnet-4-6 bucket is still relieved by hopping to opus-4-8's separate bucket regardless of bucket size. => hop is NOT redundant for RPM/TPM.
   • The daily subscription cap is shared across all Claude models (feat(dev-lead): proactive provider headroom check before engine invocation #206), so no per-model hop addresses it; equalization concerns per-model rate LIMITS only, so it has no bearing on the daily-cap fallback. Layer-2 is explicitly preserved.

   Implementation Checklist

   • Record the RPM/TPM-vs-daily-cap rationale + "hop retained" conclusion in the engine.sh two-layer fallback doc (lines 12-24), citing feat(dev-lead): proactive provider headroom check before engine invocation #206 (no chain edit).
   • Leave all CLAUDE_*_MODEL_CHAIN defaults unchanged (no weakening of cross-provider path or exit-2 contract).

   Tests to Write

   • Add a regression bats test in tests/dev-lead/unit/test_engine_chain.bats documenting [Phase 2] Assess simplifying the sonnet->opus in-engine fallback given rate-limit equalization #1099's conclusion: the ACTION chain retains the sonnet->opus hop and the daily-cap/exit-2 contract is untouched.

   Test Command

   bash .dev-lead/scripts/dev-lead-lint.sh (shellcheck) and bats tests/dev-lead/unit/test_engine_chain.bats
6. don-petry commented on Jul 8, 2026

   

   don-petry

   on Jul 8, 2026

   Collaborator

   More actions

   Dev-Lead: Implementation Complete

   Assessment outcome

   The in-engine sonnet→opus hop is NOT redundant — chains left unchanged; rationale recorded.

   Rate-limit equalization makes each Claude model's RPM/TPM bucket the same size across tiers, but it does not merge them into one shared bucket. Layer-1 recovery (engine.sh:13-22) works because "each Claude model has its own TPM/RPM bucket" — it depends on the buckets being separate, not on them differing in size. A saturated claude-sonnet-4-6 RPM/TPM bucket is still relieved by hopping to claude-opus-4-8's independent bucket regardless of equal sizing, so the hop (CLAUDE_ACTION_MODEL_CHAIN=claude-sonnet-4-6,claude-opus-4-8, engine.sh:154) stays useful and is retained.

   Per the Dev Notes, this "no code change to the chains, rationale recorded" is the expected, valid result — no chain edit was forced to look productive.

   Acceptance criteria

   • AC1 — Written analysis recorded in the engine.sh two-layer fallback doc (added after lines 12-24) + this comment; cites the layer split and the shared daily-cap caveat (feat(dev-lead): proactive provider headroom check before engine invocation #206). Concludes equalization does not make the RPM/TPM hop redundant.
   • AC2 — Daily-subscription-cap fallback explicitly preserved: equalization changes only per-model rate LIMITS, so it has no bearing on the shared daily cap (feat(dev-lead): proactive provider headroom check before engine invocation #206), which is a per-account pool no per-model hop can address. That case stays covered by layer-2 cross-provider fallback via the exit-2 path.
   • AC3 — Analysis concludes NOT redundant → chains unchanged; rationale recorded; test_engine_chain.bats extended with three [Phase 2] Assess simplifying the sonnet->opus in-engine fallback given rate-limit equalization #1099 regression guards.
   • AC4 — No chain trimming; cross-provider (claude→copilot→gemini) path and the exit-code-2 rate-limit contract untouched, and now guarded by an explicit exit-2 test.
   • AC5 — shellcheck clean (dev-lead-lint), dev-lead unit bats pass.

   Test Results

   tests/dev-lead/unit/test_engine_chain.bats: 29/29 ok (incl. 3 new #1099 guards)
     ok 15 issue-1099: action chain retains the sonnet->opus RPM/TPM hop (not trimmed)
     ok 16 issue-1099: sonnet RPM/TPM limit still recovers via the opus hop (layer-1 intact)
     ok 17 issue-1099: daily-cap/exhaustion still yields exit 2 (cross-provider contract preserved)
   
   Full dev-lead unit suite: 695 tests.
     - Only pre-existing failure: "debug: REVIEW_MCP_DEBUG set + healthy handshake ... ::notice::"
       — confirmed flaky on the clean baseline (git stash) and unrelated to this change;
       passed in the run that included these changes. No new failures introduced.
   
   dev-lead-lint.sh: all checks passed (shellcheck on 91 files + 4 agent profiles + CODEOWNERS).
   

   Files Changed

   • scripts/engine.sh: added a concise [Phase 2] Assess simplifying the sonnet->opus in-engine fallback given rate-limit equalization #1099 rationale note to the two-layer fallback doc explaining why equalization (equal bucket size, not shared bucket) does not collapse the sonnet→opus hop, and why the daily cap (feat(dev-lead): proactive provider headroom check before engine invocation #206) is layer-2 only. No chain defaults changed.
   • tests/dev-lead/unit/test_engine_chain.bats: added three regression guards documenting the [Phase 2] Assess simplifying the sonnet->opus in-engine fallback given rate-limit equalization #1099 conclusion — hop retention, layer-1 RPM/TPM recovery, and the preserved exit-2 cross-provider contract.

   Phase-2 (TDD) note

   Per the documentation/behavior-preserving exception: the assessment concluded no behavioral change, so there was no red→green cycle. The added tests are regression guards that assert the (unchanged) intended behavior and pass against current code.

   Notes

   No follow-ups. The daily-subscription-cap safety net is explicitly kept; only per-model RPM/TPM reasoning was re-evaluated.
7. don-petry

   added a commit that references this issue on Jul 8, 2026

   feat: implement issue #1099 — [Phase 2] Assess simplifying the sonnet…

   ...

   f882266
8. don-petry

   closed this as completedin #1123on Jul 8, 2026