You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Phase 1.5] Shadow triage: run a candidate triage model alongside the incumbent on real PRs, logged but never acted on #2108
As a maintainer replacing the routing model in the review cascade,
I want the candidate's triage decision on real PRs logged next to the incumbent's,
so that false-approve risk is measured on real traffic, because a 5-case held-out set cannot show a rise in false-approves.
Acceptance Criteria
When a repository variable naming a shadow candidate model is set, scripts/review-one-pr.sh runs the candidate's triage on the same prompt after the incumbent's and writes the candidate's {escalate, risk} decision and model id to the token-usage record. The incumbent's result alone drives the cascade.
If the variable is unset (the default), behaviour and outputs are unchanged. A bats test asserts this.
Shadow runs stop by themselves once a configured end date (at most 14 days after enabling, per the epic cost cap) has passed, even if the variable is still set.
A shadow failure (non-JSON, refusal, error, rate limit) is logged as a shadow outcome and never changes the PR's review result or exit code.
Shadow token records carry the candidate model id, so token_report.sh prices them under the candidate's row.
Triage is invoked and validated in scripts/review-one-pr.sh (the cascade failed at tier 1 handling). The shadow call must sit outside that error path so it can never fail the review.
Pin the candidate for the shadow call only, by setting CLAUDE_TRIAGE_MODEL_CHAIN=<candidate> for that single run_triage invocation (model-ab.sh uses the same pattern).
Workflow env wiring goes in .github/workflows/pr-review.yml. Follow AGENTS.md for repository variables. Do not touch thin caller stubs.
This is pipeline Stage 6 (shadow) from discussion Enable Haiku 5.5 #2095. Keep it model-agnostic.
Project Structure Notes
Inert by default. The kill switch is unsetting the variable.
References
scripts/review-one-pr.sh
scripts/engine.sh
.github/workflows/pr-review.yml
scripts/token_report.sh
AGENTS.md
Likely target surface
scripts/review-one-pr.sh
.github/workflows/pr-review.yml
Story prepared by the BMAD Scrum Master (Bob) for epic #2101. Status: ready-for-dev.
Activity
github-actions commentedon Oct 9, 2026
Superseded by the re-planned initiative epic #2113 (re-planned from idea discussion #2095).