You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Phase 1] Run and record the Haiku 5.5 vs Haiku 4.5 held-out A/B and liveness evidence #2107
As a maintainer deciding the triage swap,
I want recorded non-regression and liveness evidence for claude-haiku-5-5 against claude-haiku-4-5-20251001,
so that the swap PR is grounded in measured evidence rather than release notes.
Acceptance Criteria
model-ab.yml is dispatched with candidate claude-haiku-5-5, incumbent claude-haiku-4-5-20251001 and sets triage pr-review, once per candidate effort arm (low, medium) within the epic cost cap (runs clamped 1..3, at most 2 arms).
For each arm, a comment on this issue records the run URL, the model-ab.sh exit code (0 accept / 1 regression / 2 infra), per-set scores and the false-approve / false-escalate counts.
The liveness results (headroom probe, effort acceptance, real-engine triage parse) are recorded. Any FAIL is reported, not waived.
The chosen triage effort is recorded using the fixed rule: among arms with model-ab.sh exit 0 AND false-approves <= the incumbent's, pick the cheapest. If no arm qualifies, the comment records that and the swap story stays blocked.
model-ab.yml is workflow_dispatch only: every run spends tokens, so dispatch is a deliberate maintainer action (workflow header). That is why this story is hands-off.
Infra exit 2 is re-run and never counted as a regression (model-ab.sh semantics).
The pricing row must already be merged so the A/B's token records are priced.
Project Structure Notes
Evidence only. No code changes.
References
.github/workflows/model-ab.yml
scripts/evals/model-ab.sh
docs/evals/model-ab-regression.md
docs/initiatives/sonnet-5-default-go-no-go.md
Story prepared by the BMAD Scrum Master (Bob) for epic #2101. Status: ready-for-dev.
Activity
github-actions commentedon Oct 9, 2026
Superseded by the re-planned initiative epic #2113 (re-planned from idea discussion #2095).