Skip to content

feat(server): give the b10621 router a model cache source and INI presets - #1501

Merged
inureyes merged 1 commit into
mainfrom
feature/issue-1438-router-cache-and-presets
Aug 30, 2026
Merged

inureyes merged 1 commit into
mainfrom
feature/issue-1438-router-cache-and-presets

Conversation

@inureyes

Copy link
Copy Markdown
Member

Summary

Gives the b10621 router the model cache source its four remaining deferred manifest entries needed, and implements --models-preset INI translation, closing what issue #1438 recorded as remaining after PR #1495.

What changed

  • src/server/router_cache.rs (new): the router's cache source wraps the mlxcel model store (--model-store-root / MLXCEL_MODELS_DIR / MLXCEL_CACHE_DIR) behind a RouterDownloader trait, so tests drive the full download flow without the network; removal goes through the store's containment-checked deletion.
  • src/server/router_models.rs: the pool reconciles cache, --models-dir, and preset sources with b10621's precedence; entries carry source, aliases, tags, hidden, per-model config, and a transient downloading state; start_download runs the fetch on a background task forwarding throttled download_progress SSE events; remove cancels in-flight downloads, unloads running instances, deletes from disk, and emits model_remove.
  • src/server/router_presets.rs (new): b10621 INI grammar ([name] sections, [*] global cascade, long-option and LLAMA_ARG_* key spellings, version skipped, unknown keys a startup error), preset-only load-on-startup / stop-timeout / dedup-cache-models, and the CLI-wins overlay applied onto a per-model clone of the router's startup config, re-resolved through build_server_config so preset keys and CLI flags cannot drift.
  • src/server/router_server.rs: POST /models validates (empty, exists, normalized repo id, synchronous metadata probe) then answers {"success": true} while the download runs; DELETE /models serves the full removal flow with upstream's exact refusal texts; the model object carries real source, can_remove, aliases, tags, status.preset, and download progress; dispatch resolves preset aliases and addresses the load path by the entry's real name (found live in the E2E: the alias path previously answered "not found").
  • src/downloader/mod.rs: DownloadHooks (per-chunk progress, cooperative cancellation with a typed DownloadCancelled marker), download_repo_with_hooks, and probe_repo; the HEAD size pass also runs when a progress hook is installed so SSE events carry totals.
  • src/server/startup.rs, src/server/cli_input.rs, both binaries: ServerStartupConfig derives Clone and carries model_store_root; router mode now also starts from --models-preset alone; --models-preset with a model argument fails startup loudly.
  • Manifest compat/llama-server/b10621/router-models.json: POST /models and GET /models/sse flip to by_design with rationale and pinning tests (bare-name default-org expansion as policy; in-process payload residue as architectural, mirroring --models-dir), DELETE /models flips to supported, --models-preset flips to aliased with its refusal divergences recorded. docs/llama-server-compat.md updated to match.

Security

A download path plus a delete path is a new way to write and remove files. Names from HTTP resolve only through the registry or the store's sanitized <owner>/<name> composition; deletion re-asserts containment under the store root before removing anything; downloads write only under the cache root with the downloader's existing canonicalized-parent checks and symlink-safe tempfiles; the new routes sit behind the router's API-key middleware like every other management route.

Test plan

  • cargo test --lib --features metal,accelerate --profile test-fast server::router (61 tests: discovery confinement, cache listing/collision precedence, download success/failure/cancel event sequences, remove refusals and disk deletion, preset parsing/overlay/dedup/alias routing, auth on all management routes)
  • cargo test --profile test-fast --features metal,accelerate --bin mlxcel-server and --bin mlxcel (CLI arg structs changed)
  • python3 scripts/ci/check_llama_compat_manifest.py
  • cargo clippy --lib --tests --features metal,accelerate -- -D warnings, cargo fmt --all -- --check
  • Real-checkpoint E2E on Apple Silicon: router started with a preset-defined smollm-135m-4bit (load-on-startup, [*] temp overlay, chat by alias), pre-seeded store snapshot deleted through DELETE /models, and mlx-community/SmolLM-135M-Instruct-4bit downloaded through POST /models over the real network (19 download_progress events with per-url {done, total}, download_finished, model_remove on delete, then a chat completion served from the downloaded cache entry)

Closes #1438

…sets

Closes the four deferred manifest entries issue #1438 recorded after PR #1495: POST /models, DELETE /models, GET /models/sse, and --models-preset, all of which needed a model cache source that did not exist.

The router pool now reconciles three model sources with b10621's precedence (cache < models-dir < preset). The cache is the mlxcel model store (--model-store-root / MLXCEL_MODELS_DIR / MLXCEL_CACHE_DIR): its snapshots list as removable source "cache" entries, POST /models validates a requested name (non-empty, normalized repo id, synchronous metadata probe, upstream's validate-then-download order) and downloads the MLX snapshot on a background task while the entry lists as "downloading" with per-url progress, and DELETE /models cancels an in-flight download or unloads a running instance, deletes the snapshot with containment re-asserted against the store root, and emits model_remove. GET /models/sse now serves the full b10621 event vocabulary: models_reload, model_status, status_change, download_progress, download_finished, download_failed, model_remove.

--models-preset loads b10621 INI presets: [name] sections define models (model= checkpoint directory, hf-repo= cache snapshot) or overlay configuration onto discovered ones, [*] cascades under every model, keys accept long-option or LLAMA_ARG_* spellings, and each section overlays a per-model clone of the router's startup config re-resolved through build_server_config so preset keys and CLI flags cannot drift (explicitly-given CLI flags win, upstream's overlay order). load-on-startup and dedup-cache-models are honored, stop-timeout parses inert, and an untranslated key fails startup naming the key and section. Router mode now also starts from --models-preset alone, matching upstream's model-less router condition for the surfaces mlxcel exposes.

Supporting changes: the downloader gains DownloadHooks (per-chunk progress callbacks and cooperative cancellation with a typed DownloadCancelled marker) plus probe_repo for the metadata probe; ServerStartupConfig derives Clone and carries model_store_root; per-model configs are rebuilt per entry, so preset sampling overlays and per-checkpoint generation_config defaults resolve exactly as on the CLI; dispatch resolves preset aliases and addresses the load path by the entry's real name (found live: the alias path previously answered "not found" from ensure_ready).

Manifest: POST /models and GET /models/sse flip to by_design with rationale and pinning tests (bare-name default-org expansion as policy; in-process payload residue as architectural, mirroring --models-dir), DELETE /models flips to supported, --models-preset flips to aliased with its refusal divergences recorded. --models-dir itself stays by_design as #1499 recorded.

Validated by unit and route tests (fake downloader driving the same hooks: download, progress payloads, failure, cancel-on-delete, dedup hiding, alias routing, auth on the new surfaces) and a real-checkpoint E2E on Apple Silicon: router started with a preset-defined smollm-135m-4bit (load-on-startup, temp overlay, alias chat), a pre-seeded store snapshot deleted through DELETE /models, and mlx-community/SmolLM-135M-Instruct-4bit downloaded twice through POST /models over the real network with 19 download_progress events, download_finished, model_remove, and a chat completion served from the downloaded cache entry.

Refs #1438
@inureyes inureyes added status:review Under review type:enhancement New features, capabilities, or significant additions priority:high High priority area:architecture Architecture and code structure changes area:inference Generation, sampling, decoding (incl. speculative, DRY) area:cli Command-line interface / CLI flags status:done Completed and removed status:review Under review labels Aug 30, 2026
@inureyes
inureyes merged commit 1f2749a into main Aug 30, 2026
12 checks passed
@inureyes
inureyes deleted the feature/issue-1438-router-cache-and-presets branch August 30, 2026 05:10
@inureyes inureyes self-assigned this Aug 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:architecture Architecture and code structure changes area:cli Command-line interface / CLI flags area:inference Generation, sampling, decoding (incl. speculative, DRY) priority:high High priority status:done Completed type:enhancement New features, capabilities, or significant additions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(server): implement b10621 router mode and resolve models-dir semantics

1 participant