[Phase 2] Canary go/no-go report: compare deep-tier cost, cache-read cost and latency by model from token-usage artifacts #1951

Story

As a release maintainer,
I want one command that turns pr-review token-usage artifacts into a pass/fail table for the epic's cost and latency metrics,
so that #1899's go/no-go is a mechanical read of evidence instead of hand-summing JSONL.

Why this blocks #1899

scripts/token_report.sh renders org-wide totals by workflow/tier/model. It does not compute per-invocation means, compare a candidate model with an incumbent, or apply #1899's bars (cost per invocation ≥ 20% lower, cache-read cost ≥ 50% lower, latency ≥ 20% better). There is also no guard against "PASS" on too few canary PRs.

Acceptance Criteria

1. New scripts/canary_report.sh with a pure render function that reads a directory of token-usage JSONL files. Inputs: --candidate and --incumbent (both required, no model default; see AC7), --workflow (default pr-review) and --tier (default deep). Output is a Markdown table per arm: invocation count, distinct PR count (from context), mean and p50 USD per invocation, mean cache-read USD per invocation, mean and p50 duration_ms, and the candidate-vs-incumbent delta for each metric.

2. USD is priced with price_for <model> <record-date> from scripts/lib/model-pricing.sh, using the same source as token_report.sh. Records whose model has no price are counted and reported, never silently dropped.

3. Each metric gets PASS / FAIL / INSUFFICIENT against its bar. INSUFFICIENT covers fewer than 5 distinct candidate PRs, fewer than 5 incumbent invocations, or no non-null duration_ms on either arm (latency). The overall verdict is PASS only if all three are PASS, and the exit codes are 0 = PASS, 1 = FAIL, 2 = INSUFFICIENT. Missing data must never read as PASS.

4. main() downloads token-usage-* artifacts for this repo over a --since/--until window. Reuse token_report.sh's collection helpers (source them or factor them into scripts/lib/) rather than duplicating them. It can also read a local directory (--dir), so it can score the model-ab-tokens-* artifacts from the model-ab runner story on identical inputs.

5. A workflow_dispatch entry (the existing token-report.yml or a small new job) runs it and writes the table to $GITHUB_STEP_SUMMARY.

6. tests/canary_report.bats exercises the pure function on fixture JSONL: pass, fail per metric, INSUFFICIENT (too few PRs, no durations) and unpriced-model accounting. Register it in lint.yml, and keep shellcheck clean.

7. Generic across models and windows (revision, 2026-10-08). The script must compare ANY two models, not one canary. No model ID and no calendar date may appear as a default or hard-coded value in scripts/canary_report.sh, .github/workflows/token-report.yml or the test logic:

   • --candidate and --incumbent are required. Omitting either exits with the operational-error code (3) and a usage message. Remove the claude-opus-5-5 / claude-opus-4-8 defaults (script lines ~350-351 and ~695) and their model-pin-ok markers.
   • The baseline window has no fixed default. Take --baseline-since / --baseline-until when given; otherwise derive it as --baseline-days (default 14) ending at --since. Remove the 2026-09-11 / 2026-09-25 constants (script lines ~356-357 and ~698) and the dated examples in the header comments, replacing them with placeholders.
   • The thresholds are flags as well as env vars, with flag taking precedence: --cost-bar, --cache-bar, --latency-bar, --min-prs, --min-invocations, --candidate-max-prs. Keep the current numbers as the defaults.

8. Workflow inputs. token-report.yml workflow_dispatch exposes candidate_model, incumbent_model, canary_since, baseline_days and the threshold inputs, and passes them through as arguments. run_canary fails fast with a clear message when a required model input is empty.

9. Header and docs describe the script as a model-vs-model comparison. Examples use <candidate-model> / <incumbent-model> placeholders, and the baseline-expiry note states the 30-day retention without naming a specific model or date.

10. Tests parametrized by model. tests/canary_report.bats takes the model IDs from variables set once at the top (use obviously synthetic IDs such as model-new and model-old), not literals repeated through the file. Add cases for: missing --candidate/--incumbent (exit 3), a baseline window derived from --baseline-days, flags overriding env vars, and two non-Anthropic-looking model names scoring correctly.

11. Latency when a baseline predates duration_ms: the report keeps scoring latency INSUFFICIENT (never PASS), and its output says which arm lacks durations so the maintainer knows to use --model-ab-dir.

Dev Notes

• Per-invocation cost on real PRs depends on PR size. Also print a size-normalised figure (USD per 1M input-equivalent tokens) as context, but apply the bar to per-invocation cost as the epic states.
• Baseline expiry: token-usage artifacts are kept 30 days (pr-review.yml upload step). The last Opus 4.8 deep-tier records in this repo date from before the canary cut at 2026-09-25 14:07Z and expire by about 2026-10-25. Document in the script header how to snapshot a baseline directory (--dir) before then.
• COST CAP: read-only over existing artifacts. It makes no model calls.

Likely target surface

• scripts/canary_report.sh (new), possibly scripts/lib/ for shared artifact collection
• tests/canary_report.bats (new)
• .github/workflows/token-report.yml, .github/workflows/lint.yml

Split out of #1899 (hands-off release story) so its measurable prerequisites can be delivered by dev-lead. Epic #1895.