Skip to content

[Phase 1] Run and record the Haiku 5.5 vs Haiku 4.5 held-out A/B and liveness evidence #2107

Closed

Description

@github-actions

Story

As a maintainer deciding the triage swap,
I want recorded non-regression and liveness evidence for claude-haiku-5-5 against claude-haiku-4-5-20251001,
so that the swap PR is grounded in measured evidence rather than release notes.

Acceptance Criteria

  1. model-ab.yml is dispatched with candidate claude-haiku-5-5, incumbent claude-haiku-4-5-20251001 and sets triage pr-review, once per candidate effort arm (low, medium) within the epic cost cap (runs clamped 1..3, at most 2 arms).
  2. For each arm, a comment on this issue records the run URL, the model-ab.sh exit code (0 accept / 1 regression / 2 infra), per-set scores and the false-approve / false-escalate counts.
  3. The liveness results (headroom probe, effort acceptance, real-engine triage parse) are recorded. Any FAIL is reported, not waived.
  4. The chosen triage effort is recorded using the fixed rule: among arms with model-ab.sh exit 0 AND false-approves <= the incumbent's, pick the cheapest. If no arm qualifies, the comment records that and the swap story stays blocked.

Tasks / Subtasks

Dev Notes

  • model-ab.yml is workflow_dispatch only: every run spends tokens, so dispatch is a deliberate maintainer action (workflow header). That is why this story is hands-off.
  • Infra exit 2 is re-run and never counted as a regression (model-ab.sh semantics).
  • The pricing row must already be merged so the A/B's token records are priced.

Project Structure Notes

Evidence only. No code changes.

References

  • .github/workflows/model-ab.yml
  • scripts/evals/model-ab.sh
  • docs/evals/model-ab-regression.md
  • docs/initiatives/sonnet-5-default-go-no-go.md

Story prepared by the BMAD Scrum Master (Bob) for epic #2101. Status: ready-for-dev.

Activity

  1. added
    initiativeEpic / initiative tracking issue
    initiative:holdInitiative driver must not release this issue (human hold)
    dev-lead:hands-offExclude from dev-lead agent automation
    on Oct 9, 2026
  2. github-actions commented on Oct 9, 2026

    @github-actions
    ContributorAuthor

    Superseded by the re-planned initiative epic #2113 (re-planned from idea discussion #2095).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    dev-lead:hands-offExclude from dev-lead agent automationinitiativeEpic / initiative tracking issueinitiative:holdInitiative driver must not release this issue (human hold)

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions