[Phase 4] Bounded A/B ET measurement + held-out eval gate + human go/no-go before default-on #1107

Story

As a pipeline owner,
I want run a bounded on/off A/B over a fixed PR sample, measure the per-tier ET delta vs the frozen baseline, confirm the held-out deep-review eval does not regress, and record a human go/no-go before flipping the prefetch flag to default-on,
so that the initiative ships only if it demonstrably saves tokens without degrading review quality, and the production default flip is a deliberate, human-gated decision.

Acceptance Criteria

1. An A/B measurement over a fixed sample (<=~30 PRs per arm) reports the per-tier ET for flag-on vs flag-off (and vs the Story 1 frozen baseline), sourced from the existing token JSONL / token_report.sh aggregation — reported as an exact delta, not asserted to hit a preset number.
2. The held-out deep-review eval (scripts/evals/run-eval.sh deep-review over evals/deep-review/holdout/, 5 cases, pass_threshold 0.7) is scored for BOTH arms; the flag-on aggregate is NOT lower than the flag-off baseline (non-regression), and the held-out cases are unmodified (immutability guarded). Do NOT use scripts/evals/gate.sh — it is a STRICT-IMPROVEMENT comparator that rejects a tie, whereas this refactor targets equal quality.
3. The comparison is computed against the FROZEN Story 1 baseline artifact (read-only), so the measurement cannot be gamed by moving the baseline.
4. A written go/no-go recommendation (ET delta + eval result + freshness-safeguard status) is recorded for a human decision; the default-on flip is performed by a human, not auto-released.
5. The measurement + eval runs stay within the stated cost cap (fixed sample size, existing eval suite, no unbounded loops).

Tasks / Subtasks

• Run the bounded on/off A/B and aggregate per-tier ET (AC: test issue from agent #1, Add @claude delegation, auto-merge, and rebase handling #3, feat: add Copilot engine support via REVIEW_ENGINE toggle #5)
  • Reuse token_report.sh aggregation over the JSONL for both arms
  • Compare deep+audit ET against the frozen Story 1 baseline
• Score the held-out deep-review eval on both arms and assert non-regression (AC: Go-live improvements for PR review agent #2)
  • run-eval.sh deep-review on the flag-off baseline and the flag-on arm; compare aggregates (flag-on >= flag-off)
  • Do NOT use gate.sh (strict-improvement, rejects ties); do not edit held-out cases (holdout-guard.sh)
• Write the go/no-go recommendation doc (AC: Optimize review: small-PR and incremental fast paths #4)
  • Summarize ET delta, eval result, freshness-safeguard coverage
• Hand off the default-on flip to a human (AC: Optimize review: small-PR and incremental fast paths #4, feat: add Copilot engine support via REVIEW_ENGINE toggle #5)
  • The flip is a trivial one-line flag-default change performed by a human after approval

Dev Notes

• Reuse existing tooling: scripts/token_report.sh (ET aggregation, 📊 Token Cost Observatory — Weekly Report #464) for the A/B numbers and scripts/evals/run-eval.sh for the quality score on both arms. Do NOT build a new metric or eval set, and do NOT use scripts/evals/gate.sh — it demands a STRICT improvement (candidate score > incumbent; ties rejected) and would fail the equal-quality outcome this refactor is designed to produce. Compare run-eval aggregates directly for non-regression.
• Overfitting / reward-hacking guard: the quality check runs the HELD-OUT split (evals/deep-review/holdout/), not the dev split, and the cases are immutable (scripts/lib/holdout-guard.sh fails any proposer-authored change to guarded eval paths). The ET baseline from Story 1 is a read-only artifact, so 'savings' are measured against a frozen goalpost — you cannot lower the baseline to manufacture a win.
• Cost cap: cap the A/B at a fixed PR sample per arm (~30) and a single held-out eval pass; no self-tuning or loop-until-target. This mirrors the bounded dry-run discipline of the sibling eval-gated epics (e.g. [Phase 4] Held-out eval regression gate + 5-PR dry-run + human go/no-go before LIVE_MODE #1094, [Phase 1] Validate Opus 4.8 >= Opus 4.7 on the deep-review held-out eval suite (regression gate) #836).
• The default-on flip is a production behavior change to the highest-volume workflow — gate it on a recorded human go/no-go (hands_off), consistent with the org's LIVE_MODE / go-no-go pattern; the epic must never carry initiative:auto.
• Depends on all rewire + safeguard stories (3, 4, 5) being in place so the on-arm exercises the full optimized path.

Project Structure Notes

Go/no-go recommendation doc under docs/initiatives/. Measurement uses existing scripts (token_report.sh, evals/run-eval.sh); the reviewable unit is the measurement + recommendation. The default-flip is a trivial one-line human change performed only after approval.

References

• scripts/token_report.sh
• scripts/evals/run-eval.sh
• scripts/lib/holdout-guard.sh
• evals/deep-review/holdout/cases.jsonl
• AGENTS.md#initiative-planner-blocking-open-questions-gate

Likely target surface

• docs/initiatives/
• scripts/review-one-pr.sh

Story prepared by the BMAD Scrum Master (Bob) for epic #1101. Status: ready-for-dev.