💡 Model Catalog Sync: Auto-Detect New and Retiring Models via the Models API and Scaffold the Onboarding

[github-actions[bot]]

Summary

Add a scheduled, read-only model catalog sync. It polls Anthropic's GET /v1/models (and Gemini's model list) and diffs the result against the chains scripts/lib/engine-models.sh resolves for every task key (triage, deep, audit, action, single, duck). Three kinds of event come out of the diff: a new model in a line we use, a configured model that is deprecated or has a retires_at date, and a capability mismatch between a chain and the flags engine.sh sends to it. Each event opens one scaffolded onboarding issue with a pre-filled pricing row, a compatibility report, the model-ab command to run, and the ring-promotion checklist. Model enablement becomes a repeatable pipeline instead of a hand-written plan per release.

This is a direct answer to the maintainer's question on #2095: "Enabling new models is an ongoing concern. Thoughts on how to make this faster, easier and more predictable?"

Market Signal

• Release cadence is now weekly. Claude Opus 5.5 shipped Sep 22, Sonnet 5.5 Sep 28 and Haiku 5.5 Oct 7, 2026: three model swaps in 16 days (Anthropic news). GitHub put Haiku 5.5 into Copilot on Oct 7 (GitHub changelog).
• The catalog is now machine-readable. GET /v1/models returns lifecycle (active/deprecated/retired), deprecated_at, retires_at, line (haiku/sonnet/opus/fable/mythos), max_input_tokens, max_tokens and a capabilities object. That object covers effort.{low,medium,high,xhigh,max}, thinking.types.{adaptive,enabled,disabled}, structured_outputs and batch (Models API reference). The docs also say "do not infer a line from the id", and today we infer it from the id.
• New models break things beyond the model id:
  • Prompt-length pricing (new). Haiku 5.5 costs $0.10 / $0.50 per MTok up to 100K prompt tokens and $0.50 / $2.50 above that. The whole request is billed at the higher rate, cache reads included (pricing). model-pricing.tsv has no column for this, so the token reports will under-price any Haiku 5.5 triage call above 100K tokens by 5x.
  • Per-model cache-read multipliers. Opus 5.5 and Sonnet 5.5 bill cache reads at 0.05x base input, Fable 5.1 at 0.025x, and everything else at 0.1x.
  • Request parameters. Claude 4.7 and later return 400 on non-default temperature/top_p/top_k (deprecations).
  • Thinking defaults. Haiku 5.5 defaults to adaptive thinking at medium effort, and responses can open with a thinking block (secondary analysis; confirm against the migration guide). engine.sh leaves effort at the CLI default for triage/action (see the effort-pin comment near line 1730), so swapping Haiku in changes triage cost and latency without anyone choosing to.
• A retirement date is near for the triage default. The deprecations page lists claude-haiku-4-5-20251001, the first model of the default triage chain, as "not sooner than October 15, 2026". Anthropic gives at least 60 days' notice, so nothing breaks next week. But a retires_at will appear on the API before any human reads an email, and a sync job would catch it the same day. claude-sonnet-4-5-20250929 is already deprecated (retires Nov 30, 2026); the sync would confirm no chain or env override still names it.

User Signal

• Enable Haiku 5.5 #2095 (Haiku 5.5) and Enable Opus 5.5 #1893 (Opus 5.5): each enablement is a long hand-written plan, and the maintainer has asked how to make it predictable.
• [Phase 3] Refresh the Opus 4.8 fallback-chain hops to Opus 5.5 in engine.sh #1900: refresh the Opus 4.8 fallback hops after the Opus 5.5 swap. This is a "find every chain that names model X" task, done by hand.
• ai_models_fable_deprecation: tab-prefixed Fable chains miss the deprecation warning #1981: ai_models_fable_deprecation misses tab-prefixed chains. Deprecation detection is hand-coded per family (engine-models.sh:252) and string-matched, so the next deprecation needs another hand-written function.
• [Engine config · Phase 1] Versioned config/ai-engines.json as the single source of truth for providers, model catalog and per-task chains #1973 / [Engine config · Phase 2] Per-task provider preference: let each task (triage, deep, audit, action, duck) choose its own provider order #1974: config/ai-engines.json as the single source of truth, with per-task provider preference. The sync job is the natural reader and checker for that file.
• [Phase 2] Canary go/no-go report: compare deep-tier cost, cache-read cost and latency by model from token-usage artifacts #1951: a canary go/no-go report on deep-tier cost and latency per model. Its output feeds the promotion step this pipeline would scaffold.
• Gemini quota is unobserved: check_provider_headroom has no gemini branch, so degrading to Gemini would be unmetered #2030 / Gemini availability per task tier: show remaining capacity per model, key and day so a capped model is visible before the work fails #2041 / chore: cross-job Gemini quota and cooldown persistence via fleet-state (#2030 scope item 3) #2067: Gemini quota is unobserved. The same sync can record Gemini model availability per key.

Technical Opportunity

Nearly every part already exists. What's missing is the trigger and the glue:

Need	Existing asset
Resolve configured chains per task	scripts/lib/engine-models.sh (ai_model_for_family, AI_MODELS_* parsing)
Price data, date-versioned	scripts/lib/model-pricing.tsv + tests/model_pricing.bats
Keep concrete ids out of code	scripts/check-model-pins.sh (#1979)
Quality A/B on a candidate model	scripts/evals/model-ab.sh, model-ab.yml, evals/triage, evals/deep-review
Safe rollout	ADR-0002 channel tags and rings, pr_review_canary.sh
Pure logic plus tests	ADR-0004: the diff is a jq function over two JSON documents and can be tested in bats with fixtures

Proposed shape:

1. scripts/model-catalog-sync.sh (pure core). Input: the /v1/models JSON plus the resolved chain table. Output: a JSON list of events:
   • new_in_line
   • deprecated
   • retires_at_set
   • capability_mismatch, for example deep pins --effort high and the model has effort.high.supported=false, or a chain model has thinking.types.disabled=false while a tier needs thinking off
   • pricing_row_missing, for any configured or newly listed model with no matching model-pricing.tsv glob
2. Pricing schema extension. Add an optional prompt_tokens_over threshold column, so that a second row for the same glob covers the >100K tier. Price selection then also keys on the record's prompt size. This is append-only and backward compatible.
3. Scaffolder. Each event files one idempotent issue (marker plus model id) from a template. The template carries the pricing rows, the compatibility report, the exact model-ab dispatch command, and the ring checklist (v1-next canary, then the [Phase 2] Canary go/no-go report: compare deep-tier cost, cache-read cost and latency by model from token-usage artifacts #1951 report, then stable).
4. Wiring. Run daily inside the existing health or actions-fleet monitor, not as a new cron, because cron is best-effort (Improving reliability of scheduled GitHub Actions runs (cron is best-effort) #656). Start read-only.

Assessment

Dimension	Score	Rationale
Feasibility	high	One API call, jq diffing, existing price/eval/ring machinery; no new secrets (the engine already holds an Anthropic key)
Impact	high	Turns each model release from a multi-day plan into a filed checklist. Prevents silent cost mis-reporting (Haiku 5.5 tier) and silent 400s or effort changes on swap
Urgency	high	Three releases in 16 days; Haiku 5.5 enablement (#2095) is in flight now; Haiku 4.5 has a near "not sooner than" date

Adversarial Review

Strongest objection: Three model launches of toil don't justify a platform. A human reading the changelog is cheaper than another scheduled job, and the hard part of onboarding, judging whether the new model reviews as well, can't be automated.

Rebuttal: The job does not try to judge quality. That stays with model-ab and the canary ring, where it belongs. It automates the mechanical steps that keep failing in practice: finding every chain and override that names a model (#1900), detecting deprecation without per-family string matching (#1981), keeping a pricing row in step with each release, and catching capability mismatches before a production run 400s. The Haiku 5.5 price tier is a concrete case: once Haiku 5.5 is enabled, the cost observatory will be wrong unless the schema changes, whether or not anyone builds the sync. The read-only first phase costs one API call a day.

Other risks: the /v1/models capability schema could change, so pin the anthropic-version header and fail open with a warning. Gemini has no equivalent retires_at, so its side starts as presence-only.

Suggested Next Step

1. Spike scripts/model-catalog-sync.sh (pure diff plus bats fixtures built from a captured /v1/models response) and run it read-only in the daily health check for two weeks. It writes its event list to the step summary.
2. In parallel, open a small PR that extends model-pricing.tsv with the prompt-length tier and adds the two Haiku 5.5 rows. That unblocks Enable Haiku 5.5 #2095's Phase 1 on the right footing.
3. If the two-week report shows no false events, turn on issue filing and link the template from Enable Haiku 5.5 #2095 and Enable Opus 5.5 #1893 as the standard enablement path.