[Phase 1.5] Shadow confusion-matrix report, with human-reviewed disagreements proposed as new triage held-out cases #2109

Story

As a maintainer deciding whether the shadow window passed,
I want a report comparing shadow and incumbent triage decisions, plus a reviewed path for disagreements into the held-out set,
so that the shadow metrics in the epic can be measured, and the 5-case held-out set grows from real traffic.

Acceptance Criteria

1. A script reads token-usage artifacts and prints, per candidate model, the shadow PR count, the escalate/approve confusion matrix against the incumbent, the agreement rate on escalate, and (where the PR was escalated) agreement with the deep-review result.
2. The report lists every disagreement with its PR URL so a human can review it.
3. Disagreements become held-out cases only through a separate, human-reviewed PR that appends to evals/triage/holdout/cases.jsonl. Labels come from the human reviewer, never from either model. Existing cases are never edited or removed.
4. The script has bats coverage on fixture JSONL and passes shellcheck.

Tasks / Subtasks

• Write the report script over token-usage JSONL (AC: test issue from agent #1, Go-live improvements for PR review agent #2, Optimize review: small-PR and incremental fast paths #4)
• Document the reviewed append-only path into the held-out set (AC: Add @claude delegation, auto-merge, and rebase handling #3)
  • Add the procedure to docs/evals/model-ab-regression.md

Dev Notes

• Eval safeguard (reward-hacking / immutability): held-out cases are append-only and labelled by a human. A case added from this window must not be used to re-decide the same comparison that produced it. It counts for later A/Bs only.
• Read the same JSONL format as token_report.sh. Reuse its parsing helpers if practical.
• The success thresholds (>=50 shadow PRs, >=90% escalate agreement, false-approve <= incumbent) are on the epic. The report prints the numbers and makes no promotion decision.

Project Structure Notes

A new script under scripts/ next to token_report.sh, plus a doc addition under docs/evals/.

References

• scripts/token_report.sh
• evals/triage/holdout/cases.jsonl
• docs/evals/model-ab-regression.md

Story prepared by the BMAD Scrum Master (Bob) for epic #2101. Status: ready-for-dev.