Opus 4.8 engine upgrade: validate the deep-review tier, finish the reference rollout, and keep cost data accurate

[github-actions]

Upgrade the self-hosted PR-review engine's deep tier to Claude Opus 4.8 (shipped 2026-05-28) for its honesty/quality gains (4x less likely to miss flawed code) and its model-internal Dynamic Workflows on large diffs. The deep tier (ENGINE_DEEP_MODEL) and its primary fallback chain are already pinned to claude-opus-4-8 in scripts/engine.sh, so this initiative is about (a) validating that the live upgrade is non-regressive against the frozen held-out eval set before we declare it done, (b) finishing the half-done reference rollout (stale claude-opus-4-7 entries still linger in fallback chains, scripts/aw.sh, and docs/dev-lead/spec.md), and (c) keeping model-pricing.tsv correct for the new model id.

Scope decision — NO Express/Fast tier. The idea proposed a Fast Mode "express" tier at the new $10/$50 per-MTok rate. The repo maintainer rejected it (discussion #793 comment): "do not want to enable the fast tier as we do not have any need for an Express PR review capability. The bottleneck in PR reviews is not the speed of the model." This initiative therefore touches only standard Opus 4.8 routing and pricing; it adds no express tier and no Fast Mode pricing.

Initiative success metric (how we know it worked): the deep-review held-out eval suite (evals/deep-review/holdout/cases.jsonl, scored by scripts/evals/run-eval.sh in llm-judge mode) run under claude-opus-4-8 yields an aggregate score >= the recorded claude-opus-4-7 baseline on the identical frozen set (strict non-regression). The baseline score is recorded as an immutable artifact before the candidate run. A regression is a STOP signal: the deep tier must be reverted to claude-opus-4-7 rather than the rollout being finished.

Cost cap / budget bound. Validation is bounded to the 5-case deep-review holdout set, run via run_agentic (Opus deep tier) for at most 2 models (4.7 baseline + 4.8 candidate) x at most 2 repeats = <= 20 Opus deep invocations total, with TOKEN_LOG_FILE capture so the dollar cost is reported per AGENTS.md cost reporting. Steady-state review cost is unchanged: claude-opus-4-8 is priced by the same claude-opus-4-* row ($5 / $0.50 / $6.25 / $25 per MTok, effective 2025-11-01) that already priced 4.7 — the upgrade is cost-neutral per review. The rejected Fast Mode rate ($10/$50) is explicitly NOT introduced.

Untracked prerequisites

• Claude Opus 4.8 (claude-opus-4-8) must be reachable from the review/CI runtime via the claude CLI / Anthropic API with the available credentials — an external availability dependency, not a repo issue. If the model id is not yet served to this account, the eval-validation story cannot run.
• discussion 💡 Opus 4.8 Engine Upgrade: Dynamic Workflows + 4x Honesty for Review Quality #793 comment from the maintainer rejecting the Fast Mode / Express review tier is a recorded scope decision this plan honors (no express tier, no Fast Mode pricing).

---

Planned from idea discussion #793 by the BMAD Scrum Master initiative-planner. Inert until a maintainer adds initiative:auto.

[don-petry]

Status check (2026-07-13): headline shipped, epic deliverables still open — keeping OPEN

Audited main (@d83b3b1) directly. The deep-tier Opus 4.8 upgrade is LIVE — it shipped ad-hoc via the Fable-5 / Sonnet-5 chain work, not through this (still-inert) epic:

• scripts/engine.sh: ENGINE_DEEP_MODEL="claude-opus-4-8" and CLAUDE_DEEP_MODEL_CHAIN=claude-opus-4-8,claude-sonnet-4-6,claude-sonnet-5-0 ✅

But all three epic deliverables remain undone:

#836 — non-regression eval gate: No baseline/candidate JSON, no verdict, no comment, no PR on record. The success-metric gate (Opus 4.8 ≥ 4.7 on the frozen deep-review holdout) was never run/recorded. Since the deep tier is already live, this is now a retroactive validation (or an explicit maintainer waive decision).

#837 — finish the reference rollout: stragglers still on main:

• engine.sh:149 CLAUDE_AUDIT_MODEL_CHAIN still ends in ,claude-opus-4-7
• engine.sh:151 CLAUDE_SINGLE_MODEL_CHAIN still ends in ,claude-opus-4-7
• aw.sh:94 known_engines lists claude-opus-4-7 but not claude-opus-4-8 → a profile declaring engine: claude-opus-4-8 emits a spurious "unknown engine" warning (exactly AC#2)
• docs/dev-lead/spec.md:220 ("Opus 4.7 → issue implementation") and :605 (ENGINE_SINGLE_MODEL | claude-opus-4-7) are stale — and now doubly wrong, since the single tier is actually claude-fable-5

#838 — pricing lock test: claude-opus-4-8 already resolves to the correct standard rate ($5/$0.50/$6.25/$25) via the claude-opus-4-* wildcard, but no regression case pins it in tests/model_pricing.bats.

Verdict: NOT ready to close. Remaining work is small and mostly mechanical (#837 cleanup + #838 test), with #836 requiring a run-or-waive decision. Recommend activating this epic (initiative:auto) to let dev-lead sweep #837/#838, and deciding whether to run or waive the #836 retroactive eval.

---

Generated by Claude Code

[don-petry]

Activated (initiative:auto added, 2026-07-13). dev-lead will pick up #837 (purge the opus-4-7 stragglers in engine.sh audit/single chains, aw.sh known-engines, spec.md) and #838 (opus-4-8 pricing regression test). #836 (retroactive non-regression eval) is a maintainer run-vs-waive decision — the deep tier is already live on Opus 4.8, so this is confirmatory rather than gating. — Claude Code

[don-petry]

Initiative driver — stale in-flight slot(s) detected

The following open sub-issue(s) still carry the dev-lead label but have
had no dev-lead activity for ≥ 72h, so they are occupying an in-flight
slot without progressing: #837, #838.

No labels were changed. A human should decide whether to split-and-close, retry,
or park each one so the slot is freed for the rest of the epic.