[Phase 4] Held-out eval regression gate + 5-PR dry-run + human go/no-go before LIVE_MODE #1094

Story

As a review-pipeline maintainer,
I want prove the combined enhancements beat the frozen baseline on the held-out eval and a real-PR dry-run before flipping LIVE_MODE,
so that the pipeline only ships if it demonstrably finds more bugs without regressing false positives or cost, and a human owns the go decision.

Acceptance Criteria

1. With stories 2–5 integrated, scripts/evals/run-eval.sh deep-review against evals/deep-review/holdout shows a strict improvement in aggregate score over the Phase-1 frozen baseline.
2. The false-positive rate (kind:"finding_verification" records aggregated by scripts/token_report.sh) shows no regression, and median ET/escalated-review stays within the 1.5× cost cap, both versus the frozen baseline.
3. A DRY_RUN pass runs the enhanced pipeline over at least 5 real merged PRs from this repo, and the results (findings + FP-rate + ET) are captured for author review.
4. A human reviews the dry-run output and records an explicit go/no-go decision; LIVE_MODE is only enabled (vars.LIVE_MODE / the DRY_RUN gate in pr-review.yml) after a 'go'.
5. If any metric regresses, the gate fails closed: LIVE_MODE stays off and the offending enhancement is reverted or knob-disabled.

Tasks / Subtasks

• Integrate stories 2–5 behind their opt-in knobs and run the deep-review held-out eval; assert strict improvement over the frozen baseline. (AC: test issue from agent #1)
• Aggregate FP-rate + median ET via token_report.sh and assert no regression / within cost cap. (AC: Go-live improvements for PR review agent #2)
• Run a DRY_RUN pass over ≥5 real merged PRs (DRY_RUN=true path in pr-review.yml / review-batch.sh) and capture findings, FP-rate, ET for review. (AC: Add @claude delegation, auto-merge, and rebase handling #3)
• Present the dry-run results and record the human go/no-go; only on 'go' set vars.LIVE_MODE=true (or advance the review-batch knobs). (AC: Optimize review: small-PR and incremental fast paths #4)
• Wire the fail-closed behavior: on any regression, keep LIVE_MODE off and disable/revert the offending knob. (AC: feat: add Copilot engine support via REVIEW_ENGINE toggle #5)

Dev Notes

• The LIVE_MODE / DRY_RUN gate already exists: pr-review.yml sets DRY_RUN via github.event_name == 'repository_dispatch' && 'false' || inputs.dry_run || (vars.LIVE_MODE == 'true' && 'false' || 'true'). The default is dry-run unless LIVE_MODE=true — so 'enable in LIVE_MODE' means flipping the repo variable AFTER the human go. review-one-pr.sh already gates all posting on DRY_RUN.
• Reuse the existing scorer end-to-end: scripts/evals/run-eval.sh deep-review (llm-judge, pass_threshold 0.7 from evals/deep-review/scorer.json) for the quality gate; scripts/token_report.sh for ET + FP aggregation (finding_verification records from token-metrics.sh). Compare against the Phase-1 frozen baseline artifact — do not recompute or move the baseline here.
• This is the convergence gate; it must fail closed. Mirror the strict-improvement discipline of holdout-guard.yml / scripts/lib/holdout-guard.sh (a candidate merges only if it does not regress the held-out score). The go/no-go record echoes the human-decision pattern of epic [Phase 3] Record the human go/no-go decision and fleet-wide recommendation #845.
• The human go/no-go and the LIVE_MODE flip are a human responsibility (hands_off) — never auto-flip LIVE_MODE from an agent. Batch review runs via scripts/review-batch.sh over a prs.txt list; the ≥5 merged PRs feed that path with DRY_RUN=true.
• Testing: no new pipeline behavior beyond wiring the gate; assert the eval/report commands run and that a regression keeps LIVE_MODE off (bats/e2e where practical); shellcheck.

Project Structure Notes

Uses the existing eval + token-report + LIVE_MODE machinery; the only durable artifact is the go/no-go decision record (mirror epic #845's decision capture) and any knob-default flip. Human-owned promotion step.

References

• .github/workflows/pr-review.yml
• scripts/review-one-pr.sh
• scripts/review-batch.sh
• scripts/evals/run-eval.sh
• evals/deep-review/scorer.json
• evals/deep-review/holdout/cases.jsonl
• scripts/token_report.sh
• scripts/lib/token-metrics.sh
• scripts/lib/holdout-guard.sh
• AGENTS.md#cost-reporting

Likely target surface

• .github/workflows/pr-review.yml
• scripts/review-batch.sh
• docs/initiatives/

Story prepared by the BMAD Scrum Master (Bob) for epic #1088. Status: ready-for-dev.