[Follow-up] Move model-ab retry to per-(model,set) arm granularity in the #1097 comparator #1954

Context

Split out from PR #1952 (issue #1950). CodeRabbit raised a valid functional-correctness concern about the model-ab-dispatch.sh retry loop, and the maintainer deliberately deferred the deep fix to keep #1952 scoped. This issue tracks that follow-up.

Problem

scripts/evals/model-ab.sh runs a candidate/incumbent pair per requested set and emits a single infra verdict with exit 2 when any set is unscored — even if another set/arm already scored. scripts/evals/model-ab-dispatch.sh then retries the whole command on exit 2, which can re-run arms that already produced a scored number. That violates the "a scored arm is never re-run" requirement.

The dispatcher-level mitigation already in #1952 (retry only when the exit-2 output is valid evidence JSON, i.e. a genuine transient infra verdict, not a deterministic hard error) narrows when the whole-command retry fires, but does not make retries per-arm — a valid-infra exit 2 that also carries an already-scored arm can still re-run that scored arm.

Proposed fix

Move retry handling into model-ab.sh so each unscored (model, set) arm retries independently, tracking a per-arm ledger so a scored arm is never re-run. Then remove the dispatcher's whole-command retry loop.

Why deferred from #1952

This requires refactoring the #1097 comparator's core arm loop (run_arm / the per-set scoring in scripts/evals/model-ab.sh) to expose per-arm exit status plus a "scored arms are never rerun" ledger, and rewiring the dispatcher around it. That is a heavy-lift change to a separately-tested component #1952 does not otherwise touch; rushing it risks regressing the held-out immutability and scoring guarantees the comparator already enforces.

References

• PR feat: implement issue #1950 — [Phase 2] Add a workflow_dispatch runner for model-ab.sh plus a liveness/effort probe, so the Opus 5.5 non-regression A/B actually executes #1952 review thread: scripts/evals/model-ab-dispatch.sh retry loop
• Comparator: scripts/evals/model-ab.sh ([Phase 1] Held-out eval regression gate: Sonnet 5 >= Sonnet 4.6 on triage + deep-review #1097, epic Initiative: Integrate Claude Sonnet 5 into engine.sh model chains — capture introductory pricing, eval-gate quality, canary-promote #1095)
• Dispatcher wrapper: scripts/evals/model-ab-dispatch.sh ([Phase 2] Add a workflow_dispatch runner for model-ab.sh plus a liveness/effort probe, so the Opus 5.5 non-regression A/B actually executes #1950)