Skip to content

feat(sync_models): re-verify catalogue models by calling them - #2167

Merged
dsapandora merged 2 commits into
fix/deprecate-claude-opus-4-1from
fix/verify-existing-models
Sep 1, 2026
Merged

dsapandora merged 2 commits into
fix/deprecate-claude-opus-4-1from
fix/verify-existing-models

Conversation

@dsapandora

Copy link
Copy Markdown
Collaborator

Stacked on #2162 — base branch is fix/deprecate-claude-opus-4-1, not develop. Review that one first; this PR's diff is the last commit only. It cannot go in before #2162: without the deprecatedBy protection there, the sync would lift every mark this PR applies on the very next run, since the retired models are still in the listing.

Summary

  • --verify-existing calls every profile already in the catalogue, not just new ones, and deprecates the models the provider reports as gone — carrying over the replacement the provider names, when it names one.
  • A direct refusal deprecates regardless of modelSource, unlike absence from a listing.
  • The gemini live test makes one real call per profile instead of trusting models.list().

Type

fix

Testing

  • Tests added or updated
  • Tested locally
  • ./builder test passes

pytest tools/sync_models/test — 69 passed, 12 skipped (no provider keys locally). 10 new: the missing-model detector, the retired-model deprecation path and its migration note, and the guard that a busy or unauthorised provider never deprecates anything. --verify-existing exercised end to end without keys (skips cleanly). ruff check / format --check clean.

Not verified here: the new gemini live test and --verify-existing itself have not run against a real key — I don't have one. Both are skipped without ROCKETRIDE_GEMINI_KEY. The nine profiles the issue lists as 404 are therefore not marked in this PR; that data change is best made from an actual --verify-existing --apply run, which is a one-command follow-up once this merges.

Checklist

  • Commit messages follow conventional commits
  • No secrets or credentials included
  • Wiki updated (if applicable)
  • Breaking changes documented (if applicable) — none. --verify-existing is opt-in and off by default; nothing changes for existing invocations, including the scheduled workflow.

Notes for the reviewer

Why the authority rule is bypassed for this signal, when #2162 argued for keeping it. They answer different questions. A model missing from a source's listing means little across sources — an OpenRouter routing alias will not appear in a provider's native API, and that is expected, not a retirement. A provider refusing to run a model is not ambiguous in the same way: claude-opus-4-1 is modelSource: openrouter, and it is exactly as unusable through the Anthropic SDK as if it had been discovered natively. OpenRouter will keep listing it indefinitely because OpenRouter still serves it. So absence stays scoped by ownership; a refusal does not.

What counts as a refusal is deliberately narrow. Only a message naming the model as gone — 404, not_found_error, no longer available, does not exist and similar. A rate limit, a timeout or a permission error must never deprecate a profile, because a single bad run would otherwise wipe out a provider's whole catalogue. Tested in both directions.

The migration note keeps the provider's own suggestion. Google's 404 body says "Please update your code to use models/gemini-3.6-flash", and that replacement is extracted into the profile's migration field. A hand-written note already on the profile is never overwritten.

Cost. One request per profile, so --verify-existing is much slower than a normal run — 26 calls for llm_gemini, and they are billed. That is why it is a flag rather than default behaviour. Worth deciding separately whether the scheduled workflow should run it weekly, or on its own slower schedule; I have not wired it into CI here.

Linked Issue

Fixes #2154

@coderabbitai

coderabbitai Bot commented Sep 1, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review (2)
  • develop
  • release/.*

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Team

Run ID: 7d0ae419-80d7-4789-8641-57064c5740c3

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added the docs Documentation label Sep 1, 2026
@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor
🤖 Internal: Discord sync marker

Auto-managed by the Discord notification workflow. Stores the linked Discord message ID and forum thread ID. Do not edit or delete.

@github-actions github-actions Bot added the module:nodes Python pipeline nodes label Sep 1, 2026
@dsapandora

Copy link
Copy Markdown
Collaborator Author

Ran --verify-existing against a live Gemini key. It works, and it found exactly the five profiles that were still active — now marked in f75dbbf, with the replacement Google names in the 404 body:

models/gemini-2.5-pro            404 -> use models/gemini-3.1-pro-preview
models/gemini-2.5-flash          404 -> use models/gemini-3.6-flash
models/gemini-2.5-flash-lite     404 -> use models/gemini-3.5-flash-lite
models/gemini-2.5-pro-preview    404 not found for API version v1beta
models/gemini-2.5-pro-preview-05-06  404 not found for API version v1beta

All five are still returned by models.list() with generateContent among their supported methods, which is why the listing-based test could never have caught them.

Two things the run turned up that were not in the issue:

Two migration notes pointed at models in this same batch. gemini-2.0-flash and gemini-2.0-flash-lite were already deprecated and directing people to 2.5-flash and 2.5-flash-lite, both of which answer 404 today. Repointed. Every migration target in the file now resolves to a profile that actually runs.

The run must not be applied before #2160 lands. In dry run it wanted to write modelOutputTokens: 65536 -> 4096 on fifteen gemini profiles, plus 65535 -> 4096, 32768 -> 4096 and so on. Gemini's native API returns a context window but no completion limit, so the resolver falls through to the 4096 default and overwrites the good values — the "~70 profiles silently truncating at 4096" defect, live. #2160 adds the guard that stops a fallback default from overwriting a value already in the catalogue; without it, --verify-existing --apply would fix five profiles and quietly cap fifteen others at 4096.

So the order is: #2162 (this PR's base), then #2160, and only then is --verify-existing --apply safe to run for real. The five profiles here are hand-marked in the meantime, which is also what keeps them protected under the deprecatedBy rule.

Also worth noting the new report line did its job on the first run: gemini-3-pro-image came back as "listed again but still deprecated" — a hand-marked profile the API does list. That one needs a human decision, which is the point of surfacing it rather than letting it sit in unchanged.

The smoke gate validates additions only: a model already in services.json is
accepted without being called, so one that entered correctly and was retired
six months later stays in the catalogue forever. The live test does not catch
it either - it asks the listing endpoint, and providers keep returning models
they have retired, with generateContent among the supported methods, refusing
only when the model is actually called.

--verify-existing calls every active profile and deprecates the ones the
provider reports as gone, carrying over the replacement the provider names when
it names one. Only a message identifying the model as unavailable qualifies; a
rate limit or a permission error never deprecates anything.

That evidence bypasses the authority rule, deliberately. Absence from a listing
means little across sources - an OpenRouter alias missing from a provider's API
is expected - but a direct refusal does not: a profile discovered through
OpenRouter is just as unusable once the provider behind it has retired the
model, and OpenRouter will keep listing it regardless.

The gemini live test now makes one real call per profile instead of trusting
the listing, which is the check it always claimed to be.
Verified with a real generateContent call per profile against a current key,
which is what --verify-existing does: models/gemini-2.5-pro, 2.5-flash,
2.5-flash-lite, 2.5-pro-preview and 2.5-pro-preview-05-06 all answer 404,
'no longer available to new users'. All five are still listed by models.list()
with generateContent among their supported methods, which is why nothing had
noticed.

Google names the replacement for three of them and those go into the migration
notes; the two preview IDs are simply gone with nothing suggested.

Also repointed two existing notes: gemini-2.0-flash and 2.0-flash-lite were
directing people to 2.5-flash and 2.5-flash-lite, both of which are in this
same batch. Every migration target now resolves to a profile that runs.
@dsapandora
dsapandora force-pushed the fix/verify-existing-models branch from f75dbbf to 0e330cb Compare September 1, 2026 15:07
@dsapandora
dsapandora merged commit 0e330cb into fix/deprecate-claude-opus-4-1 Sep 1, 2026
4 checks passed
@dsapandora
dsapandora deleted the fix/verify-existing-models branch September 1, 2026 15:22
@dsapandora

Copy link
Copy Markdown
Collaborator Author

Folding this into #2162 rather than merging twice — the two are the same change split across a stack, and #2162 was already the deprecation-mechanism PR this depends on.

All commits here are now on fix/deprecate-claude-opus-4-1 unchanged (b1202163, 0e330cb7), and #2162 carries Fixes #2154 along with the full context: the live Gemini run, the five retired profiles, and why a direct refusal deprecates regardless of modelSource while absence from a listing does not.

Closing in favour of #2162.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation module:nodes Python pipeline nodes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant