[Phase 1] Non-regression + liveness validation of Opus 5.5 against Opus 4.8 on the frozen held-out sets

Story

As a review-engine maintainer,
I want run the held-out model A/B, the cost estimate, an engine-liveness check, and a reasoning-effort probe for claude-opus-5-5,
so that we have objective evidence Opus 5.5 is a non-regression and is invokable end-to-end before any engine.sh swap ships to callers.

Acceptance Criteria

scripts/evals/model-ab.sh claude-opus-5-5 claude-opus-4-8 is run on the default held-out sets (triage and deep-review) and exits 0 (candidate ≥ incumbent on every SCORED set); infra/un-scored arms are re-run, not recorded as regressions.
scripts/evals/eval-cost-estimate.sh is run with the Story-1 pricing row and reports the per-run deep-tier cost delta (expected ≥ 20% input/output reduction, ≥ 50% cache-read reduction).
A live claude --print --model claude-opus-5-5 invocation succeeds end-to-end in CI (engine liveness), and its --output-format json result parses through the _claude_chain_invoke extraction path including any thinking content blocks.
The default reasoning-effort behaviour of claude --print --model claude-opus-5-5 is documented (medium vs high); if it defaults below the Opus-4.8 posture, the finding is recorded so Story 3 can add an explicit --effort high for the deep tier.
A short validation note (A/B scores, cost delta, liveness result, effort finding) is captured on this issue as the go/no-go input for Phase 2.

Tasks / Subtasks

Run model-ab.sh candidate=claude-opus-5-5 incumbent=claude-opus-4-8 on the default (triage, deep-review) held-out sets; capture per-set scores and exit code. (AC: test issue from agent #1)
Run eval-cost-estimate.sh with the new pricing row and record the deep-tier cost delta vs Opus 4.8. (AC: Go-live improvements for PR review agent #2)
Add/confirm an engine-liveness invocation of claude --print --model claude-opus-5-5 and verify JSON+thinking-block parsing through _claude_chain_invoke. (AC: Add @claude delegation, auto-merge, and rebase handling #3)
Probe and document the CLI default reasoning-effort for opus-5-5; note whether --effort high is required for the deep tier. (AC: Optimize review: small-PR and incremental fast paths #4)
Summarize A/B, cost, liveness, and effort findings as the Phase-2 go/no-go note on the issue. (AC: feat: add Copilot engine support via REVIEW_ENGINE toggle #5)

Dev Notes

model-ab.sh applies a >= NON-REGRESSION bar (not gate.sh's strict >); it pins the generator per arm via a single-id CLAUDE_TRIAGE_MODEL_CHAIN and holds the judge FIXED (AB_JUDGE_MODEL) so only the generator model varies — do not change the judge model between arms (scripts/evals/model-ab.sh header).
EVAL SAFEGUARD (overfitting/reward-hacking + artifact immutability): the held-out sets are FROZEN and the incumbent baseline is the pinned claude-opus-4-8; this is a model-replacement A/B with the skill markdown held at its incumbent file (SKILL_PROMPT_FILE default) — do NOT tune prompts, edit the eval corpus, or move the baseline to make the candidate pass. An un-scored/throttled arm is re-run, never scored as a regression.
COST CAP: run each A/B arm at most 3 times to smooth throttling; do not expand the held-out corpus. This story spends eval tokens only — no production traffic.
_claude_chain_invoke shells claude --print --model X --output-format json and extracts .result; Opus 5.5 always returns thinking blocks, so AC-3 confirms the existing stateless parse path handles them (the idea flags this as low risk but explicitly asks to verify).
The effort finding (AC-4) is the input contract for Story 3's optional --effort high task — record it precisely (observed default + whether an override is needed).
This is a validation-only story: it must NOT modify engine.sh model assignments; its output gates Phase 2.

Project Structure Notes

Exercises scripts/evals/{model-ab.sh,eval-cost-estimate.sh,run-eval.sh} and validate-engines/liveness paths; any new check should live alongside the existing eval scripts and their bats tests, not in engine.sh.

References

scripts/evals/model-ab.sh#non-regression-bar
scripts/evals/eval-cost-estimate.sh
scripts/engine.sh#_claude_chain_invoke
scripts/validate-engines.sh#validate_engines

Likely target surface

scripts/evals/model-ab.sh
scripts/evals/eval-cost-estimate.sh
scripts/validate-engines.sh

Story prepared by the BMAD Scrum Master (Bob) for epic #1895. Status: ready-for-dev.

Dev-Lead Implementation Plan

Issue: #1897 — [Phase 1] Non-regression + liveness validation of Opus 5.5 against Opus 4.8 on the frozen held-out sets

Scope

This is a validation-only story (per Dev Notes: it must NOT modify engine.sh model assignments, prompts, or the frozen held-out corpus). The durable, honest artifact is a regression test that locks the two findings that are deterministically verifiable offline — the AC-2 cost delta (derived from the frozen model-pricing.tsv row) and the AC-3 thinking-block parse path — plus a validation note recording the go/no-go evidence. The live-model ACs (A/B run, live liveness invocation, effort probe) require a CI run with real claude-opus-5-5 access and are recorded honestly rather than fabricated (AGENTS.md: "verify by execution, never by run status"; never let "cannot check" become "nothing to check").

What is verifiable offline (locked by regression test)

AC-2 (cost delta, deep tier — deterministic from pricing data): claude-opus-5-5 vs claude-opus-4-8 in scripts/lib/model-pricing.tsv: input 5.00 -> 4.00 (20% ↓), output 25.00 -> 20.00 (20% ↓), cache-read 0.50 -> 0.20 (60% ↓). Meets the AC-2 thresholds (≥20% input/output, ≥50% cache-read). A new test asserts these via scripts/lib/model-pricing.sh so a future pricing edit that erodes the reduction fails CI.
AC-3 (thinking-block parse — the low-risk verify the Dev Notes ask for): _claude_chain_invoke extracts .result via extract_engine_text claude (jq -j '.result // empty'). A new test feeds a claude --print --output-format json envelope that also carries thinking content blocks and asserts the stateless parse returns the final .result verbatim, unaffected by the thinking blocks.

What requires a live CI run (recorded, not fabricated)

AC-1 model-ab.sh claude-opus-5-5 claude-opus-4-8 on the frozen triage + deep-review held-out sets — needs live model access.
AC-3 liveness a live claude --print --model claude-opus-5-5 --output-format json end-to-end invocation.
AC-4 the CLI default reasoning-effort probe (medium vs high) for opus-5-5.

Implementation Checklist

tests/test_opus_5_5_validation.bats — new regression test locking AC-2 cost delta thresholds and AC-3 thinking-block parse path (offline, network-free).
.github/workflows/lint.yml — register the new bats file in the bats job list (per AGENTS.md repo-specific-test-list rule).

Tests to Write

AC-2: opus-5-5 vs opus-4-8 per-token deep-tier cost shows ≥20% input, ≥20% output, ≥50% cache-read reduction (from the price table, not hardcoded).
AC-2 counterfactual: rates are table-derived (removing/blanking the row breaks the assertion).
AC-3: extract_engine_text claude returns .result verbatim from a JSON envelope containing thinking content blocks.

Test Command

bash scripts/run-bats.sh tests/test_opus_5_5_validation.bats (full suite via the bats job in lint.yml); repo lint via bash .dev-lead/scripts/dev-lead-lint.sh.

Dev-Lead: Progress + provisional validation note (in progress — not yet durable)

Provisional. Nothing is pushed yet; the automation posts the authoritative
completion record (with PR number + head SHA) after the push. The findings
below split into deterministic offline evidence (derived from committed
data + pure functions — stable regardless of the push) and live-model ACs
that require a CI run and are therefore recorded as pending, never fabricated
(AGENTS.md: verify by execution; never let "cannot check" become "nothing to check").

Approach

This is a validation-only story (must not touch engine.sh assignments, prompts,
or the frozen held-out corpus). The durable artifact locks the two ACs that are
deterministic offline; the live A/B, liveness, and effort probes are left to CI
with real claude-opus-5-5 access.

AC-2 — deep-tier cost delta (opus-5-5 vs opus-4-8), from the frozen `model-pricing.tsv` row (#1896)

Derived via scripts/lib/model-pricing.sh (price_for) on/after the 2026-09-22 effective date:

rate	opus-4-8 (`claude-opus-4-*`)	opus-5-5 (`claude-opus-5-5*`)	reduction	AC-2 bar	meets
input (USD/MTok)	5.00	4.00	20%	≥20%	yes
output (USD/MTok)	25.00	20.00	20%	≥20%	yes
cache-read (USD/MTok)	0.50	0.20	60%	≥50%	yes

A representative deep call is strictly cheaper on opus-5-5. This is data-derived,
not hardcoded (blanking the table yields no price rather than an invented rate).

AC-3 — thinking-block parse path (low-risk verify requested by the Dev Notes)

_claude_chain_invoke extracts the answer via extract_engine_text claude
(jq -j '.result // empty'). Confirmed offline: a --output-format json envelope
that also carries thinking content blocks parses to exactly .result — the
thinking text never leaks — and the usage block still parses for token accounting.
The stateless parse path is unaffected by thinking blocks.

Recorded as requiring a live CI run (not attempted from this workspace, not fabricated)

AC-1 scripts/evals/model-ab.sh claude-opus-5-5 claude-opus-4-8 on the frozen triage + deep-review held-out sets (candidate ≥ incumbent; un-scored/throttled arms re-run, cost cap ≤3 runs/arm).
AC-3 liveness a live claude --print --model claude-opus-5-5 --output-format json end-to-end invocation.
AC-4 the CLI default reasoning-effort probe (medium vs high); if it defaults below the Opus-4.8 posture, that is the input contract for Story 3's optional --effort high.

Local check output (provisional — not the pushed run)

tests/test_opus_5_5_validation.bats — 5 checks green (AC-2 margins, per-run delta,
  table-derived guard; AC-3 thinking-block extraction + usage parse).
related cluster (model_pricing, token_metrics, eval_cost_estimate, model_ab): green.
counterfactuals: eroding the opus-5-5 row fails the AC-2 margin; a non-.result
  envelope yields empty (thinking never leaks) — both assertions are meaningful.
repo lint (.dev-lead/scripts/dev-lead-lint.sh): all checks passed.

Files touched (this workspace, not yet pushed)

tests/test_opus_5_5_validation.bats: new offline regression test locking AC-2 + AC-3.
.github/workflows/lint.yml: register the new bats file in the bats job list.

Provisional go/no-go for Phase 2

Offline evidence is positive (cost non-regression + reduction thresholds met;
parse path handles thinking blocks). The live A/B, liveness, and effort findings
must be produced by a CI run before Phase 2 flips to a final go.

Dev-Lead: Implementation Complete — PR #1912

The implementation for issue #1897 is durable: commits are pushed and a pull request is open. This record is backed by verifiable artifacts you can confirm in one click.

Pull request: feat: implement issue #1897 — [Phase 1] Non-regression + liveness validation of Opus 5.5 against Opus 4.8 on the frozen held-out sets #1912
Head commit: 610009b2841c9d032fcc8d12faf5ced0d97d132c
Net change against main: 2 file(s), +133/-0 lines
Run: https://github.com/petry-projects/.github-private/actions/runs/35816493380

Review happens on the PR. Acceptance criteria and test results are evidenced by the PR's diff and CI checks — not asserted here. If a later stage fails, this claim is retracted in place.

feat: implement issue #1897 — [Phase 1] Non-regression + liveness val…

Navigation Menu

Repository navigation

[Phase 1] Non-regression + liveness validation of Opus 5.5 against Opus 4.8 on the frozen held-out sets #1897

Story

Acceptance Criteria

Tasks / Subtasks

Dev Notes

Project Structure Notes

References

Likely target surface

Dev-Lead Implementation Plan

Scope

What is verifiable offline (locked by regression test)

What requires a live CI run (recorded, not fabricated)

Implementation Checklist

Tests to Write

Test Command

Dev-Lead: Progress + provisional validation note (in progress — not yet durable)

Approach

AC-2 — deep-tier cost delta (opus-5-5 vs opus-4-8), from the frozen `model-pricing.tsv` row (#1896)

AC-3 — thinking-block parse path (low-risk verify requested by the Dev Notes)

Recorded as requiring a live CI run (not attempted from this workspace, not fabricated)

Local check output (provisional — not the pushed run)

Files touched (this workspace, not yet pushed)

Provisional go/no-go for Phase 2

Dev-Lead: Implementation Complete — PR #1912

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Phase 1] Non-regression + liveness validation of Opus 5.5 against Opus 4.8 on the frozen held-out sets #1897

Description

Story

Acceptance Criteria

Tasks / Subtasks

Dev Notes

Project Structure Notes

References

Likely target surface

Activity

don-petry commented on Sep 23, 2026

Dev-Lead Implementation Plan

Scope

What is verifiable offline (locked by regression test)

What requires a live CI run (recorded, not fabricated)

Implementation Checklist

Tests to Write

Test Command

don-petry commented on Sep 23, 2026

Dev-Lead: Progress + provisional validation note (in progress — not yet durable)

Approach

AC-2 — deep-tier cost delta (opus-5-5 vs opus-4-8), from the frozen model-pricing.tsv row (#1896)

AC-3 — thinking-block parse path (low-risk verify requested by the Dev Notes)

Recorded as requiring a live CI run (not attempted from this workspace, not fabricated)

Local check output (provisional — not the pushed run)

Files touched (this workspace, not yet pushed)

Provisional go/no-go for Phase 2

don-petry commented on Sep 23, 2026

Dev-Lead: Implementation Complete — PR #1912

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

AC-2 — deep-tier cost delta (opus-5-5 vs opus-4-8), from the frozen `model-pricing.tsv` row (#1896)