Category: quality · Source: control-tower gap-hunt 2026-07-19 (playbook audit + external research)
Problem
Self-preference bias in LLM judges is MODEL-level, not role-level — a perplexity/familiarity effect (judges favor text familiar to their own distribution), so separating writer/reviewer ROLES while using the same or same-family model does not fully mitigate it. Sources: https://arxiv.org/abs/2410.21819 ; correlated-error panel findings https://arxiv.org/pdf/2605.29800 ("Nine Judges, Two Effective Votes"), https://arxiv.org/html/2404.18796v1.
Current state (verified): work-lane writer = Opus; claude[bot] reviewer = claude-sonnet-4-6 (ci-workflows claude-review.yml:80 default, no consumer override). Different model — good — but same family; shared training priors remain. codex:codex-rescue (GPT-5) is already installed and unused for review.
Proposed fix (cheap, targeted)
- Add a cross-vendor (codex/GPT-5) verification pass to the work lane's existing high-blast-radius gate (skill frontmatter, cross-plugin contracts, hooks) — exactly where a different-vendor perspective pays; near-zero setup since the plugin is installed.
- For any future judge-panel design (digests, eval judges per the skill-safety-net umbrella): round-robin diverse judges ACROSS items rather than full panels per item — correlated errors make N same-family judges worth ~2 effective votes; rotation is the cost-efficient debias.
Explicitly NOT proposed: cross-vendor review on every PR — most pipeline PRs are small/prose; blast-radius-gated is the right spend.
Relations
#513 (the reviewer lane in the mini-SDLC — this specifies its model-diversity requirement), #476 (autopilot merge confidence), skill-safety-net umbrella (judge design).
Category: quality · Source: control-tower gap-hunt 2026-07-19 (playbook audit + external research)
Problem
Self-preference bias in LLM judges is MODEL-level, not role-level — a perplexity/familiarity effect (judges favor text familiar to their own distribution), so separating writer/reviewer ROLES while using the same or same-family model does not fully mitigate it. Sources: https://arxiv.org/abs/2410.21819 ; correlated-error panel findings https://arxiv.org/pdf/2605.29800 ("Nine Judges, Two Effective Votes"), https://arxiv.org/html/2404.18796v1.
Current state (verified): work-lane writer = Opus; claude[bot] reviewer =
claude-sonnet-4-6(ci-workflows claude-review.yml:80 default, no consumer override). Different model — good — but same family; shared training priors remain.codex:codex-rescue(GPT-5) is already installed and unused for review.Proposed fix (cheap, targeted)
Explicitly NOT proposed: cross-vendor review on every PR — most pipeline PRs are small/prose; blast-radius-gated is the right spend.
Relations
#513 (the reviewer lane in the mini-SDLC — this specifies its model-diversity requirement), #476 (autopilot merge confidence), skill-safety-net umbrella (judge design).