Repository navigation
feat(sync_models): re-verify catalogue models by calling them - #2167
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. 🗂️ Base branches to auto review (2)
Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Team Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
🤖 Internal: Discord sync markerAuto-managed by the Discord notification workflow. Stores the linked Discord message ID and forum thread ID. Do not edit or delete. |
|
Ran All five are still returned by Two things the run turned up that were not in the issue: Two migration notes pointed at models in this same batch. The run must not be applied before #2160 lands. In dry run it wanted to write So the order is: #2162 (this PR's base), then #2160, and only then is Also worth noting the new report line did its job on the first run: |
The smoke gate validates additions only: a model already in services.json is accepted without being called, so one that entered correctly and was retired six months later stays in the catalogue forever. The live test does not catch it either - it asks the listing endpoint, and providers keep returning models they have retired, with generateContent among the supported methods, refusing only when the model is actually called. --verify-existing calls every active profile and deprecates the ones the provider reports as gone, carrying over the replacement the provider names when it names one. Only a message identifying the model as unavailable qualifies; a rate limit or a permission error never deprecates anything. That evidence bypasses the authority rule, deliberately. Absence from a listing means little across sources - an OpenRouter alias missing from a provider's API is expected - but a direct refusal does not: a profile discovered through OpenRouter is just as unusable once the provider behind it has retired the model, and OpenRouter will keep listing it regardless. The gemini live test now makes one real call per profile instead of trusting the listing, which is the check it always claimed to be.
Verified with a real generateContent call per profile against a current key, which is what --verify-existing does: models/gemini-2.5-pro, 2.5-flash, 2.5-flash-lite, 2.5-pro-preview and 2.5-pro-preview-05-06 all answer 404, 'no longer available to new users'. All five are still listed by models.list() with generateContent among their supported methods, which is why nothing had noticed. Google names the replacement for three of them and those go into the migration notes; the two preview IDs are simply gone with nothing suggested. Also repointed two existing notes: gemini-2.0-flash and 2.0-flash-lite were directing people to 2.5-flash and 2.5-flash-lite, both of which are in this same batch. Every migration target now resolves to a profile that runs.
f75dbbf to
0e330cb
Compare
|
Folding this into #2162 rather than merging twice — the two are the same change split across a stack, and #2162 was already the deprecation-mechanism PR this depends on. All commits here are now on Closing in favour of #2162. |
Summary
--verify-existingcalls every profile already in the catalogue, not just new ones, and deprecates the models the provider reports as gone — carrying over the replacement the provider names, when it names one.modelSource, unlike absence from a listing.models.list().Type
fix
Testing
./builder testpassespytest tools/sync_models/test— 69 passed, 12 skipped (no provider keys locally). 10 new: the missing-model detector, the retired-model deprecation path and its migration note, and the guard that a busy or unauthorised provider never deprecates anything.--verify-existingexercised end to end without keys (skips cleanly).ruff check/format --checkclean.Not verified here: the new gemini live test and
--verify-existingitself have not run against a real key — I don't have one. Both are skipped withoutROCKETRIDE_GEMINI_KEY. The nine profiles the issue lists as 404 are therefore not marked in this PR; that data change is best made from an actual--verify-existing --applyrun, which is a one-command follow-up once this merges.Checklist
--verify-existingis opt-in and off by default; nothing changes for existing invocations, including the scheduled workflow.Notes for the reviewer
Why the authority rule is bypassed for this signal, when #2162 argued for keeping it. They answer different questions. A model missing from a source's listing means little across sources — an OpenRouter routing alias will not appear in a provider's native API, and that is expected, not a retirement. A provider refusing to run a model is not ambiguous in the same way:
claude-opus-4-1ismodelSource: openrouter, and it is exactly as unusable through the Anthropic SDK as if it had been discovered natively. OpenRouter will keep listing it indefinitely because OpenRouter still serves it. So absence stays scoped by ownership; a refusal does not.What counts as a refusal is deliberately narrow. Only a message naming the model as gone —
404,not_found_error,no longer available,does not existand similar. A rate limit, a timeout or a permission error must never deprecate a profile, because a single bad run would otherwise wipe out a provider's whole catalogue. Tested in both directions.The migration note keeps the provider's own suggestion. Google's 404 body says "Please update your code to use models/gemini-3.6-flash", and that replacement is extracted into the profile's
migrationfield. A hand-written note already on the profile is never overwritten.Cost. One request per profile, so
--verify-existingis much slower than a normal run — 26 calls forllm_gemini, and they are billed. That is why it is a flag rather than default behaviour. Worth deciding separately whether the scheduled workflow should run it weekly, or on its own slower schedule; I have not wired it into CI here.Linked Issue
Fixes #2154