feat(server): give the b10621 router a model cache source and INI presets

compat/llama-server/b10621/router-models.json

@@ -120,12 +120,13 @@
       "default": "disabled",
       "description": "path to INI file containing model presets for the router server (default: disabled) (env: LLAMA_ARG_MODELS_PRESET)",
       "discovery": "help-entry-head",
-      "state": "deferred",
+      "state": "aliased",
       "issue": 1438,
-      "test": "src/server/startup.rs",
-      "notes": "Accepted on both binaries with the b10621 spelling and env binding, and refused at startup with a diagnostic: mlxcel cannot yet translate llama-server INI presets into per-model configuration, and loading un-preset models while the operator believes presets apply would be the silent-ignore failure epic #1431 exists to remove.",
+      "test": "src/server/router_presets_tests.rs",
+      "notes": "Implemented since #1438's cache-source PR: the INI file is parsed with upstream's grammar ([name] sections, [*] global preset cascaded under every model, ; and # comments, version key skipped, unknown keys a startup error), keys accept both the long-option spelling without dashes and the LLAMA_ARG_* environment spelling, and each section overlays a clone of the router's own startup config which is re-resolved through the same pipeline as the CLI, so preset keys and CLI flags cannot drift (explicitly-given router CLI flags win over preset values, upstream's own overlay order). Sections define models via model= (checkpoint directory) or hf-repo= (a cache snapshot), or overlay configuration onto a discovered model; the preset-only load-on-startup, stop-timeout, and dedup-cache-models options parse, with load-on-startup and dedup-cache-models honored and stop-timeout inert (in-process unload needs no force-kill escalation). Translated keys: model, hf-repo, alias, tags, ctx-size, parallel, temp, top-k, top-p, min-p, seed, n-predict.",
       "divergence": [
-        "Any configured preset file fails startup with the migration diagnostic; b10621 loads INI presets and applies them per model (#1438)."
+        "Keys outside the translated set fail startup with a diagnostic naming the key and section; b10621 can apply any server option per model. The refusal is the epic's no-silent-ignore rule, and the translated set grows with mlxcel's per-model configurability (#1438).",
+        "A preset hf-repo= must already be a cache snapshot at startup; b10621 downloads a preset's hf-repo on demand when the model loads (#1438)."
       ],
       "rationale": null,
       "mlxcel": {
@@ -173,13 +174,11 @@
       "source": "tools/server/server.cpp",
       "condition": "router-mode",
       "discovery": "source-route-registration",
-      "state": "deferred",
+      "state": "supported",
       "issue": 1438,
-      "test": "src/server/router_server_tests.rs",
-      "notes": "Mounted with upstream's reachable refusal surface for a models-dir pool: a missing name answers 400 \"model must be a non-empty string\", an unknown one 500 \"model name=X is not found\", and a discovered one 500 \"model name=X is not removable (not from cache)\", which is exactly what b10621 answers for its models_dir-sourced entries.",
-      "divergence": [
-        "No cache source exists, so no model is ever removable; b10621 can delete cache-sourced models from disk (#1438)."
-      ],
+      "test": "src/server/router_server_tests.rs::delete_models_removes_a_cache_entry_from_disk",
+      "notes": "Implemented since #1438's cache-source PR: a cache-sourced model is removed from disk (an in-flight download is cancelled first, a loaded instance unloaded first), the entry leaves the list, and the SSE stream carries model_remove. The refusal surface matches upstream exactly: a missing name answers 400 \"model must be a non-empty string\", an unknown one 500 \"model name=X is not found\", and a models-dir or preset entry 500 \"model name=X is not removable (not from cache)\". Deletion is containment-checked against the store root (the same sanitized <owner>/<name> composition and re-assertion the mlxcel rm verb uses), so an HTTP-supplied name can never delete outside the cache.",
+      "divergence": [],
       "rationale": null,
       "mlxcel": {
         "route": "DELETE /models"
@@ -193,14 +192,18 @@
       "source": "tools/server/server.cpp",
       "condition": "router-mode",
       "discovery": "source-route-registration",
-      "state": "deferred",
+      "state": "by_design",
       "issue": 1438,
-      "test": "src/server/router_models_tests.rs",
-      "notes": "Router mode serves the SSE stream with b10621's event envelope ({\"model\", \"event\"[, \"data\"]}) for models_reload, model_status on load begin, and status_change on load/unload transitions.",
+      "test": "src/server/router_models_tests.rs::a_download_emits_the_b10621_event_sequence_and_lands_in_the_cache",
+      "notes": "Router mode serves the SSE stream with b10621's event envelope ({\"model\", \"event\"[, \"data\"]}) and, since #1438's cache-source PR, the full event vocabulary: models_reload, model_status on load begin, status_change on load/unload transitions, download_progress with per-url {done, total} while POST /models fetches into the cache, download_finished / download_failed at each download's end, and model_remove when DELETE /models removes a cache entry.",
       "divergence": [
-        "The event vocabulary is the in-process subset: download_progress / download_finished / download_failed and model_remove never occur (no cache downloads, no removable models), and status_change payloads carry status and failed only, without a child's info/progress blocks (#1438)."
+        "status_change payloads carry no child exit_code and no child /props-derived info block for loaded models: router mode serves models in-process, so there is no child process to exit or to report loading facts over a pipe. Download progress info is served; the child-lifecycle payload fields have no in-process referent (#1438)."
       ],
-      "rationale": null,
+      "rationale": {
+        "kind": "architectural",
+        "reason": "The in-process pool is the same architectural property already argued on --models-dir: models are served inside mlxcel-server rather than as child llama-server processes, so a process exit code and a child's self-reported loading info do not exist to be forwarded. Fabricating an exit_code for a load that never had a process would be less honest than omitting the field.",
+        "revisit_if": null
+      },
       "mlxcel": {
         "route": "GET /models/sse"
       }
@@ -213,14 +216,18 @@
       "source": "tools/server/server.cpp",
       "condition": "router-mode",
       "discovery": "source-route-registration",
-      "state": "deferred",
+      "state": "by_design",
       "issue": 1438,
-      "test": "src/server/router_server_tests.rs",
-      "notes": "Mounted with upstream's validation surface: a missing name answers 400 \"model must be a non-empty string\" and an existing one 400 \"model 'X' already exists\".",
+      "test": "src/server/router_server_tests.rs::post_models_expands_bare_names_to_the_default_org",
+      "notes": "Implemented since #1438's cache-source PR: the name is validated (non-empty, well-formed repo id, synchronous metadata probe against the hub, exactly upstream's validate-then-download order), an existing name answers 400 \"model 'X' already exists\", a missing one 400 \"model must be a non-empty string\", and a valid new one answers {\"success\": true} while a background task downloads the MLX snapshot into the model cache (the mlxcel store that --model-store-root / MLXCEL_MODELS_DIR / MLXCEL_CACHE_DIR resolve). The transient entry is listed with status \"downloading\" and per-url progress, the SSE stream carries download_progress / download_finished / download_failed, and a finished snapshot lists as a removable cache entry.",
       "divergence": [
-        "A new model name answers a 500 diagnostic instead of starting a HuggingFace download into the cache; mlxcel's router has no cache source yet (#1438)."
+        "A bare, owner-less model name expands to mlxcel's default organization (mlx-community/<name>) before validation and download, the same expansion -m <name> and mlxcel download apply, so the cache entry lists under the expanded repo id where b10621 lists the verbatim name (#1438)."
       ],
-      "rationale": null,
+      "rationale": {
+        "kind": "policy",
+        "reason": "mlxcel's model-reference domain is uniform across every surface that accepts a repo id (-m, mlxcel download, the store layout, and POST /models): a bare name means <default-org>/<name> everywhere, and the store's <owner>/<name> layout has exactly one spelling for each snapshot. Accepting a verbatim owner-less id on this one route would create cache entries the rest of mlxcel cannot name. Pinned by post_models_expands_bare_names_to_the_default_org in src/server/router_server_tests.rs.",
+        "revisit_if": "mlxcel's model-reference resolution stops expanding bare names, or the store gains a verbatim-id layout addressable by the other surfaces."
+      },
       "mlxcel": {
         "route": "POST /models"
       }

docs/llama-server-compat.md

@@ -207,9 +207,13 @@ Concretely, on both server binaries: `--no-ui`, `--no-webui`, `--no-agent`, `--n
 `GET`/`POST /tools` and `GET`/`POST /cors-proxy` are mounted as b10621's own disabled-feature stub: 403 with `{"error":{"message":"this feature is disabled","type":"feature_disabled"}}`, byte-shaped like upstream's `res_403`. A client of a llama-server deployment that had these features off sees the identical answer here; a deployment that had them on is the recorded divergence. Because the CORS proxy is refused rather than implemented, its SSRF surface (loopback, link-local, metadata-service and DNS-rebinding reachability) does not exist here by construction. Client-declared `tools` in chat-completion requests are unrelated and keep working: mlxcel parses model output for tool calls, it just never executes anything server-side. A server-side MCP tool loop expressed through the OpenAI Responses built-in tool contract is tracked separately in #1457 as a product feature, outside b10621 compatibility.
 ### `--models-dir` is router mode; the store root moved to `--model-store-root` (#1438)
 
-`--models-dir` / `LLAMA_ARG_MODELS_DIR` now carries b10621's router semantics on `mlxcel-server` and `mlxcel serve`: started without a model argument it serves the router surface over the directory, an in-process model pool where b10621 spawns child processes. Discovery treats each direct subdirectory holding a `config.json` as one model named by the directory; a symlink that resolves outside the models directory is skipped, request `model` names resolve only against the discovered registry, and the management routes sit behind the same API keys as everything else. `GET /models` (and `/v1/models`) reports the b10621 router inventory with `status`, `architecture`, `source: "models_dir"` and `can_remove`, `?reload=1` rescans, `POST /models/load` / `POST /models/unload` answer upstream's exact refusals, `GET /models/sse` streams `{"model", "event", "data"}` status events, `--models-max` (default 4) bounds the loaded set with LRU eviction, and `--models-autoload` plus the per-request `?autoload=` switch control load-on-demand. Requests route by the JSON body's `model` on POST and `?model=` on GET, so `GET /slots?model=X` or `GET /props?model=X` reach the named model's own endpoint, exactly upstream's proxy contract.
+`--models-dir` / `LLAMA_ARG_MODELS_DIR` now carries b10621's router semantics on `mlxcel-server` and `mlxcel serve`: started without a model argument it serves the router surface, an in-process model pool where b10621 spawns child processes. The pool reconciles three sources with b10621's precedence (cache < models-dir < preset): the model cache (the mlxcel store that `--model-store-root` / `MLXCEL_MODELS_DIR` / `MLXCEL_CACHE_DIR` resolve), whose `<owner>/<name>` snapshots list as removable `source: "cache"` entries; the `--models-dir` directory, where each direct subdirectory holding a `config.json` is one model named by the directory (a symlink resolving outside the directory is skipped); and `--models-preset` INI sections. Request `model` names resolve only against the discovered registry (plus preset aliases), and the management routes sit behind the same API keys as everything else. `GET /models` (and `/v1/models`) reports the b10621 router inventory with `status`, `aliases`, `tags`, `architecture`, `source` and `can_remove`, `?reload=1` rescans, `POST /models/load` / `POST /models/unload` answer upstream's exact refusals, `--models-max` (default 4) bounds the loaded set with LRU eviction, and `--models-autoload` plus the per-request `?autoload=` switch control load-on-demand. Requests route by the JSON body's `model` on POST and `?model=` on GET, so `GET /slots?model=X` or `GET /props?model=X` reach the named model's own endpoint, exactly upstream's proxy contract.
 
-**The old mlxcel meaning moved.** `--models-dir` used to name the local model-store root that `-m <owner/name>` resolution and auto-download use; that is now `--model-store-root` on both server binaries (`MLXCEL_MODELS_DIR` still works, and the `mlxcel download` / `list` / `rm` / `generate` subcommands keep their `--models-dir` spelling, which was never a llama-server surface). A server command line that combines `--models-dir` with a model argument fails startup with a diagnostic naming the replacement rather than silently picking either meaning. `--models-preset` is accepted and refused at startup: mlxcel cannot yet translate llama-server INI presets, and pretending otherwise would serve un-preset models silently. `--tags` sets the informational tags the `/v1/models` model object reports.
+`POST /models` downloads a new model into the cache: the name is validated (non-empty, well-formed repo id, then a synchronous metadata probe against the hub, upstream's validate-then-download order; a bare owner-less name expands to `mlx-community/<name>` first, the same expansion `-m` applies), the route answers `{"success": true}`, and a background task fetches the MLX snapshot while the entry lists as `status: "downloading"` with per-url progress. `DELETE /models?model=X` removes a cache-sourced model from disk (cancelling an in-flight download or unloading a running instance first) and answers upstream's exact refusals for everything else; deletion is containment-checked against the store root, so an HTTP-supplied name can never delete outside the cache. `GET /models/sse` streams the full `{"model", "event"[, "data"]}` vocabulary: `models_reload`, `model_status`, `status_change`, `download_progress`, `download_finished`, `download_failed`, and `model_remove`.
+
+`--models-preset` loads b10621's INI presets: `[name]` sections define models (`model =` names a checkpoint directory, `hf-repo =` a cache snapshot) or overlay configuration onto discovered ones, `[*]` is the global preset cascaded under every model, keys accept the long-option spelling without dashes or the `LLAMA_ARG_*` environment spelling, and each section overlays a per-model clone of the router's startup config that is re-resolved through the same pipeline as the CLI (explicitly-given router CLI flags win, upstream's own overlay order). The preset-only `load-on-startup` and `dedup-cache-models` options are honored and `stop-timeout` parses inert. A key outside the translated set (`model`, `hf-repo`, `alias`, `tags`, `ctx-size`, `parallel`, `temp`, `top-k`, `top-p`, `min-p`, `seed`, `n-predict`) fails startup with a diagnostic naming the key and section rather than being silently ignored; giving `--models-preset` together with a model argument fails startup the same way, since presets steer nothing outside router mode.
+
+**The old mlxcel meaning moved.** `--models-dir` used to name the local model-store root that `-m <owner/name>` resolution and auto-download use; that is now `--model-store-root` on both server binaries (`MLXCEL_MODELS_DIR` still works, and the `mlxcel download` / `list` / `rm` / `generate` subcommands keep their `--models-dir` spelling, which was never a llama-server surface). A server command line that combines `--models-dir` with a model argument fails startup with a diagnostic naming the replacement rather than silently picking either meaning. `--tags` sets the informational tags the `/v1/models` model object reports.
 
 Single-model `GET /models` / `GET /v1/models` also moved to b10621's shape: the OpenAI `data` entry carries `aliases` (the full `--alias` list), `tags`, `owned_by: "llamacpp"` (it used to say `"user"`), and a `meta` block of checkpoint facts derived from `config.json` and the safetensors headers (`n_params` unpacks quantized `U32` payloads by their declared bit width), next to b10621's Ollama-compatible `models` block whose `details.format` honestly reads `"safetensors"`.
 

src/bin/mlx_server.rs

@@ -2369,6 +2369,7 @@ fn build_startup_input(mut args: ServerArgs) -> anyhow::Result<ServerStartupInpu
         models_autoload: args.models_autoload,
         models_preset: args.models_preset.clone(),
         tags: args.tags.clone(),
+        model_store_root: args.model_store_root.clone(),
         warmup: args.warmup,
         no_warmup: args._no_warmup,
         temperature: args.temp,

src/commands/serve.rs

@@ -612,6 +612,7 @@ fn build_startup_input(mut args: crate::ServeArgs) -> anyhow::Result<ServerStart
         models_autoload: args.models_autoload,
         models_preset: args.models_preset.clone(),
         tags: args.tags.clone(),
+        model_store_root: args.model_store_root.clone(),
         metrics: args.metrics,
         warmup: args.warmup,
         no_warmup: args._no_warmup,