Skip to content

Initiative: Integrate Claude Sonnet 5 into engine.sh model chains — capture introductory pricing, eval-gate quality, canary-promote #1095

Closed
5 of 5 issues completed

Description

@github-actions

Evaluate and integrate Claude Sonnet 5 (claude-sonnet-5-*, launched 2026-06-30) into engine.sh's Claude model chains as a candidate replacement for Sonnet 4.6, capturing the introductory $2/$10-per-MTok I/O rate (33% cheaper on input than Sonnet 4.6) before it reverts to $3/$15 on 2026-09-01. Sonnet 5 also scores 63.2% on SWE-bench Pro vs Sonnet 4.6's 58.1% and equalizes Sonnet/Haiku rate limits to Opus tiers, which changes the in-engine fallback calculus.

The integration path reuses infrastructure that already exists: effective-dated pricing rows (scripts/lib/model-pricing.tsv), per-tier CLAUDE_*_MODEL_CHAIN fallback chains (scripts/engine.sh), the held-out eval scorer (scripts/evals/run-eval.sh, gate.sh), and the dev-lead canary rings (standards/canary-rings.json, scripts/canary-rollout.sh). This is a configuration + evaluation change, not new architecture. It mirrors the in-flight Opus 4.8 upgrade (#835) but targets a different (Sonnet) tier.

Initiative success metric: Sonnet 5 scores >= Sonnet 4.6 on BOTH the triage and deep-review held-out eval sets (no quality regression), delivers a measured >=30% input-cost reduction on the affected tier while intro pricing holds (realized fleet-wide ONLY if the go/no-go promotes Sonnet 5 to the PRIMARY slot — as a fallback candidate it rarely runs, so the cost target is conditional on a 'go'), and passes the dev-lead canary gate through ring1 with no candidate-introduced regression — with a human go/no-go decision recorded before 2026-08-31.

Cost cap: The evaluation is bounded — the held-out sets are small (triage = 5 cases, deep-review = 5 cases). Cap the pre-canary evaluation at <= 4 full A/B eval cycles (~2 models x 2 sets x 5 cases per cycle, Haiku-tier judge for deep-review) and a total pre-promotion model spend of <= $25; canary exposure is bounded by the existing dev-lead ring dwell/sample gate knobs (next->ring0 sample <= 15 runs). No unbounded sweeps.

Scope: pricing data -> held-out eval gate -> wire as candidate -> fallback-simplification assessment -> canary + human go/no-go. Non-goals: making Sonnet 5 the fleet default without the human go/no-go; changing the Opus (deep) or Fable (audit) tiers; touching non-Claude engines. Source: Discussion #1039.

Untracked prerequisites

  • Confirm the exact dated model ID the claude CLI accepts for Sonnet 5 (e.g. claude-sonnet-5-YYYYMMDD) and that the Actions-runner API key/CLI can actually reach it — Discussion 💡 Claude Sonnet 5 Engine Chain Integration with Introductory Pricing Capture #1039 gives only the claude-sonnet-5-* glob, not a callable dated ID.
  • Confirm Anthropic's Sonnet 5 cache-read / cache-write multipliers match the standard 0.1x (cache read) and 1.25x (5-minute cache write) of base input used to derive every existing Claude row in model-pricing.tsv — the launch post quotes only base input/output ($2/$10 intro, $3/$15 standard).

Planned from idea discussion #1039 by the BMAD Scrum Master initiative-planner. Inert until a maintainer adds initiative:auto.

Activity

  1. added
    initiative:autoDriver may auto-release this epic's ready sub-issues to dev-lead
    on Jul 4, 2026
  2. don-petry commented on Jul 14, 2026

    @don-petry
    Collaborator

    ✅ Complete — closing

    All 5 stories across the 3 phases are closed as completed:

    Introductory pricing captured (#1096), quality eval-gated (#1097), and the change rode a PR through the dev-lead pipeline. From here it soaks on the canary rings as normal fleet ops; stable/default is already live on main. Closing as completed. — Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    initiativeEpic / initiative tracking issueinitiative:autoDriver may auto-release this epic's ready sub-issues to dev-lead

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions