Repository navigation
feat(server): give the b10621 router a model cache source and INI presets - #1501
Merged
Merged
Conversation
…sets Closes the four deferred manifest entries issue #1438 recorded after PR #1495: POST /models, DELETE /models, GET /models/sse, and --models-preset, all of which needed a model cache source that did not exist. The router pool now reconciles three model sources with b10621's precedence (cache < models-dir < preset). The cache is the mlxcel model store (--model-store-root / MLXCEL_MODELS_DIR / MLXCEL_CACHE_DIR): its snapshots list as removable source "cache" entries, POST /models validates a requested name (non-empty, normalized repo id, synchronous metadata probe, upstream's validate-then-download order) and downloads the MLX snapshot on a background task while the entry lists as "downloading" with per-url progress, and DELETE /models cancels an in-flight download or unloads a running instance, deletes the snapshot with containment re-asserted against the store root, and emits model_remove. GET /models/sse now serves the full b10621 event vocabulary: models_reload, model_status, status_change, download_progress, download_finished, download_failed, model_remove. --models-preset loads b10621 INI presets: [name] sections define models (model= checkpoint directory, hf-repo= cache snapshot) or overlay configuration onto discovered ones, [*] cascades under every model, keys accept long-option or LLAMA_ARG_* spellings, and each section overlays a per-model clone of the router's startup config re-resolved through build_server_config so preset keys and CLI flags cannot drift (explicitly-given CLI flags win, upstream's overlay order). load-on-startup and dedup-cache-models are honored, stop-timeout parses inert, and an untranslated key fails startup naming the key and section. Router mode now also starts from --models-preset alone, matching upstream's model-less router condition for the surfaces mlxcel exposes. Supporting changes: the downloader gains DownloadHooks (per-chunk progress callbacks and cooperative cancellation with a typed DownloadCancelled marker) plus probe_repo for the metadata probe; ServerStartupConfig derives Clone and carries model_store_root; per-model configs are rebuilt per entry, so preset sampling overlays and per-checkpoint generation_config defaults resolve exactly as on the CLI; dispatch resolves preset aliases and addresses the load path by the entry's real name (found live: the alias path previously answered "not found" from ensure_ready). Manifest: POST /models and GET /models/sse flip to by_design with rationale and pinning tests (bare-name default-org expansion as policy; in-process payload residue as architectural, mirroring --models-dir), DELETE /models flips to supported, --models-preset flips to aliased with its refusal divergences recorded. --models-dir itself stays by_design as #1499 recorded. Validated by unit and route tests (fake downloader driving the same hooks: download, progress payloads, failure, cancel-on-delete, dedup hiding, alias routing, auth on the new surfaces) and a real-checkpoint E2E on Apple Silicon: router started with a preset-defined smollm-135m-4bit (load-on-startup, temp overlay, alias chat), a pre-seeded store snapshot deleted through DELETE /models, and mlx-community/SmolLM-135M-Instruct-4bit downloaded twice through POST /models over the real network with 19 download_progress events, download_finished, model_remove, and a chat completion served from the downloaded cache entry. Refs #1438
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Gives the b10621 router the model cache source its four remaining deferred manifest entries needed, and implements
--models-presetINI translation, closing what issue #1438 recorded as remaining after PR #1495.What changed
src/server/router_cache.rs(new): the router's cache source wraps the mlxcel model store (--model-store-root/MLXCEL_MODELS_DIR/MLXCEL_CACHE_DIR) behind aRouterDownloadertrait, so tests drive the full download flow without the network; removal goes through the store's containment-checked deletion.src/server/router_models.rs: the pool reconciles cache,--models-dir, and preset sources with b10621's precedence; entries carry source, aliases, tags, hidden, per-model config, and a transientdownloadingstate;start_downloadruns the fetch on a background task forwarding throttleddownload_progressSSE events;removecancels in-flight downloads, unloads running instances, deletes from disk, and emitsmodel_remove.src/server/router_presets.rs(new): b10621 INI grammar ([name]sections,[*]global cascade, long-option andLLAMA_ARG_*key spellings,versionskipped, unknown keys a startup error), preset-onlyload-on-startup/stop-timeout/dedup-cache-models, and the CLI-wins overlay applied onto a per-model clone of the router's startup config, re-resolved throughbuild_server_configso preset keys and CLI flags cannot drift.src/server/router_server.rs:POST /modelsvalidates (empty, exists, normalized repo id, synchronous metadata probe) then answers{"success": true}while the download runs;DELETE /modelsserves the full removal flow with upstream's exact refusal texts; the model object carries realsource,can_remove,aliases,tags,status.preset, and download progress; dispatch resolves preset aliases and addresses the load path by the entry's real name (found live in the E2E: the alias path previously answered "not found").src/downloader/mod.rs:DownloadHooks(per-chunk progress, cooperative cancellation with a typedDownloadCancelledmarker),download_repo_with_hooks, andprobe_repo; the HEAD size pass also runs when a progress hook is installed so SSE events carry totals.src/server/startup.rs,src/server/cli_input.rs, both binaries:ServerStartupConfigderivesCloneand carriesmodel_store_root; router mode now also starts from--models-presetalone;--models-presetwith a model argument fails startup loudly.compat/llama-server/b10621/router-models.json:POST /modelsandGET /models/sseflip toby_designwith rationale and pinning tests (bare-name default-org expansion as policy; in-process payload residue as architectural, mirroring--models-dir),DELETE /modelsflips tosupported,--models-presetflips toaliasedwith its refusal divergences recorded.docs/llama-server-compat.mdupdated to match.Security
A download path plus a delete path is a new way to write and remove files. Names from HTTP resolve only through the registry or the store's sanitized
<owner>/<name>composition; deletion re-asserts containment under the store root before removing anything; downloads write only under the cache root with the downloader's existing canonicalized-parent checks and symlink-safe tempfiles; the new routes sit behind the router's API-key middleware like every other management route.Test plan
cargo test --lib --features metal,accelerate --profile test-fast server::router(61 tests: discovery confinement, cache listing/collision precedence, download success/failure/cancel event sequences, remove refusals and disk deletion, preset parsing/overlay/dedup/alias routing, auth on all management routes)cargo test --profile test-fast --features metal,accelerate --bin mlxcel-serverand--bin mlxcel(CLI arg structs changed)python3 scripts/ci/check_llama_compat_manifest.pycargo clippy --lib --tests --features metal,accelerate -- -D warnings,cargo fmt --all -- --checkload-on-startup,[*] tempoverlay, chat by alias), pre-seeded store snapshot deleted throughDELETE /models, andmlx-community/SmolLM-135M-Instruct-4bitdownloaded throughPOST /modelsover the real network (19download_progressevents with per-url{done, total},download_finished,model_removeon delete, then a chat completion served from the downloaded cache entry)Closes #1438