epic(server): close llama-server b10621 compatibility gaps #1431

Objective

Make mlxcel-server a truthful, testable compatibility target for the official llama.cpp llama-server nightly b10621, without treating successful argument parsing as equivalent to behavior parity and without silently ignoring options whose values have observable semantics.

Frozen reference

• Release: https://github.com/ggml-org/llama.cpp/releases/tag/b10621
• Published: 2026-08-25
• Build: 10621
• Commit: c1d0e7a004015f23bc0233470b747b596f29b264
• macOS arm64 archive: llama-b10621-bin-macos-arm64.tar.gz
• SHA-256: 429c8270608600188035e5e92f7d78dffb7900904fe7dd7e6a84f48068cd13cf

Audit summary

The distributed binary exposes 249 help entries containing 323 distinct long-option spellings and 134 LLAMA_ARG_* / LLAMA_API_KEY environment variables. The current mlxcel help exposes 149 distinct long-option spellings, with 47 exact long-option spelling intersections and 15 exact environment-variable intersections. These raw counts are not a parity score because many upstream aliases map to one feature and many GGML hardware flags are not meaningful on MLX, but they establish that the current “drop-in replacement” surface is incomplete.

The higher-risk findings are semantic collisions rather than missing names: LLAMA_ARG_CACHE_REUSE is parsed as a boolean although upstream defines an integer chunk size; --timeout controls a decode watchdog rather than HTTP read/write timeout; --models-dir names mlxcel's model-store root rather than llama-server router mode; /completions and /embeddings currently select OpenAI-shaped handlers while b10621 uses native llama-server response schemas; --cache-type-k/v use overlapping names with incompatible accepted values; and several hidden flags are accepted and ignored even when the requested value would change upstream behavior.

Compatibility policy

Every b10621 surface must end in exactly one documented state:

1. Supported: the spelling, value domain, default, precedence, and externally observable behavior match.

2. Correctly aliased: a different mlxcel implementation provides equivalent behavior, with a tested translation and no loss of requested semantics.

3. Not applicable: the concept has no MLX/CUDA equivalent; mlxcel accepts only semantically inert forms or rejects the option with an actionable diagnostic, and documents why.

4. Deferred: a linked implementation issue exists with explicit acceptance criteria; the current binary must not claim support by silently ignoring it.

5. By design: the surface is implemented and served, and differs from b10621 on purpose and permanently. Added by feat(compat): add the by_design b10621 compatibility-policy state #1499 after waves 1 through 3 left every sub-issue implemented and seven of them unclosable, because a permanent deliberate difference is none of the first four: not supported because the behavior differs, not aliased because it is not equivalent, not not_applicable because the concept applies and is served, and not deferred because nobody intends to close it. The state carries obligations the validator enforces in full, so that permanence is argued rather than asserted: a rationale object with a kind separating an architectural impossibility from a policy choice, a reason, and a revisit_if naming what would have to change for the decision to be reconsidered, plus a test that pins the behavior mlxcel actually has. Unlike deferred it requires no open issue, since being closed to further work is the point. An entry that cannot meet the bar stays deferred: GET /slots did, because its queue-admission divergence had no permanence decision and no pinning test.

Sub-issues

Ordered by execution phase. Within a phase, items are listed in the order they must be implemented, and depends on records the real producer/consumer edge rather than a scheduling preference. #1456 is a sub-issue of #1435 and is listed here as a node because #1435 cannot close until it lands.

Phase 1

• fix(server): align llama-server b10621 aliases and defaults #1430: low-risk alias, environment, default, request-field, and route drift (merged as fix: align llama-server b10621 compatibility #1454)
• test(server): add a pinned b10621 compatibility manifest and conformance harness #1443: pinned b10621 compatibility manifest and conformance harness; every later sub-issue registers its classification here (merged as test(server): add pinned b10621 compatibility manifest and gates #1458)
• fix: Accept space-separated negative values on server CLI flags #1459: accept space-separated negative values on the server CLI flags (depends on test(server): add a pinned b10621 compatibility manifest and conformance harness #1443)

#1459 was filed during #1443 rather than at epic-authoring time. All 122 value-taking long options on both binaries reject a space-separated negative (--seed -1 gives error: unexpected argument '-1' found), because neither allow_negative_numbers nor allow_hyphen_values is set on either clap command; the pinned b10621 binary accepts that form. Seventeen b10621 options document a negative value, and they are owned by seven different sub-issues, so fixing it once ahead of the chains avoids seven chains editing the same clap setting, three of them concurrently.

Phase 2

• feat(server): align HTTP transport, timeout, CORS, TLS, and SSE semantics #1432: HTTP transport, --timeout collision, CORS, TLS, API prefix, socket, and SSE semantics (depends on test(server): add a pinned b10621 compatibility manifest and conformance harness #1443)
• fix(server): match b10621 multi-key authentication semantics #1437: multi-key and API-key-file authentication semantics (depends on feat(server): align HTTP transport, timeout, CORS, TLS, and SSE semantics #1432)
• fix(server): separate native completion and embedding routes from OpenAI routes #1441: separate the native /completion and /embeddings routes from the OpenAI routes (depends on feat(server): align HTTP transport, timeout, CORS, TLS, and SSE semantics #1432, fix(server): match b10621 multi-key authentication semantics #1437)
• fix(server): translate model-source flags and reject unsupported GGUF semantics #1434: model-source translations and hidden compatibility flags (depends on test(server): add a pinned b10621 compatibility manifest and conformance harness #1443)
• fix(server): reject unsupported GGML runtime flags and KV formats explicitly #1445: reject unsupported GGML runtime flags and KV formats explicitly (depends on fix(server): translate model-source flags and reject unsupported GGUF semantics #1434)
• fix(server): align chat-template, reasoning, and output-parsing semantics #1447: chat-template, reasoning, assistant-prefill, and output-parsing behavior (depends on fix(server): translate model-source flags and reject unsupported GGUF semantics #1434)
• fix(inference): align context, batching, RoPE, and runtime override semantics #1450: context, batching, RoPE, SWA, keep, and common runtime override semantics (depends on test(server): add a pinned b10621 compatibility manifest and conformance harness #1443)
• feat(server): implement b10621 prompt-cache and slot-scheduling semantics #1453: prompt-cache, KV-reuse, slot-selection, and continuous-batching semantics (depends on fix(inference): align context, batching, RoPE, and runtime override semantics #1450)

Phase 3

• feat(server): add b10621 tokenization, template, infill, and native embedding utilities #1442: tokenize, detokenize, apply-template, token-counting, native embedding, and infill utilities (depends on fix(server): separate native completion and embedding routes from OpenAI routes #1441, fix(server): align chat-template, reasoning, and output-parsing semantics #1447)
• fix(server): align embedding and reranking mode, pooling, and normalization #1452: embedding and reranking modes, pooling, normalization, and route gates (depends on fix(server): separate native completion and embedding routes from OpenAI routes #1441, feat(server): add b10621 tokenization, template, infill, and native embedding utilities #1442)
• fix(server): classify projector flags and align multimodal media handling #1451: projector flag classification and multimodal media handling (depends on fix(server): translate model-source flags and reject unsupported GGUF semantics #1434, fix(server): align chat-template, reasoning, and output-parsing semantics #1447)
• fix(server): align b10621 audio transcription route semantics #1446: shared audio-transcription route semantics (depends on fix(server): separate native completion and embedding routes from OpenAI routes #1441, fix(server): classify projector flags and align multimodal media handling #1451)
• feat(inference): complete b10621 sampling and grammar semantics #1436: missing sampling algorithms, sampler ordering, grammar controls, and request fields, reusing feat(sampling): add locally typical sampling (typical_p) #1377 for typical_p (depends on fix(server): separate native completion and embedding routes from OpenAI routes #1441, fix(inference): align context, batching, RoPE, and runtime override semantics #1450)
• feat(inference): map b10621 speculative flags to MTP and DFlash semantics #1433: speculative-decoding flag groups mapped onto MTP and DFlash (depends on fix(server): reject unsupported GGML runtime flags and KV formats explicitly #1445, fix(inference): align context, batching, RoPE, and runtime override semantics #1450, feat(inference): complete b10621 sampling and grammar semantics #1436)
• feat(inference): implement or explicitly reject control-vector options #1449: control-vector loading, scaling, and layer-range semantics, or explicit rejection (depends on fix(inference): align context, batching, RoPE, and runtime override semantics #1450, feat(server): implement b10621 prompt-cache and slot-scheduling semantics #1453)

Phase 4

• feat(server): align props, slots, metrics, health, and slot persistence #1440: /props, /slots, /metrics, slot persistence, and server lifecycle controls (depends on fix(server): match b10621 multi-key authentication semantics #1437, fix(server): align embedding and reranking mode, pooling, and normalization #1452, feat(server): implement b10621 prompt-cache and slot-scheduling semantics #1453)
• feat(server): implement b10621 router mode and resolve models-dir semantics #1438: router mode and the conflicting --models-dir meaning (depends on fix(server): translate model-source flags and reject unsupported GGUF semantics #1434, feat(server): align props, slots, metrics, health, and slot persistence #1440)
• feat(server): implement multi-adapter LoRA request and hot-swap semantics #1439: LoRA multi-adapter loading, per-request scaling, and /lora-adapters hot-swap (depends on feat(server): implement b10621 router mode and resolve models-dir semantics #1438)
• feat(server): implement resumable streams and chat-completion control #1444: resumable stream lookup/delete semantics and realtime chat reasoning control (depends on fix(server): separate native completion and embedding routes from OpenAI routes #1441, fix(server): align chat-template, reasoning, and output-parsing semantics #1447)
• feat(server): add Vertex AI (GCP) predict route adapter for llama-server b10621 compatibility #1456: Vertex AI (GCP) predict route adapter (depends on fix(server): separate native completion and embedding routes from OpenAI routes #1441, feat(server): add b10621 tokenization, template, infill, and native embedding utilities #1442)
• feat(server): classify and implement optional UI, tools, MCP, and GCP surfaces #1435: Web UI, built-in tools, MCP, agent, static-path, and GCP compatibility classification (depends on feat(server): align HTTP transport, timeout, CORS, TLS, and SSE semantics #1432, feat(server): add Vertex AI (GCP) predict route adapter for llama-server b10621 compatibility #1456)
• fix(server): align logging, introspection, completion, and preset flags #1448: logging, prompt-log security, cache listing, shell completion, and built-in preset flags (depends on fix(server): translate model-source flags and reject unsupported GGUF semantics #1434)

Phase 5

• feat(compat): add the by_design b10621 compatibility-policy state #1499: decide and implement the compatibility-policy state for a permanent, deliberate divergence (depends on feat(server): implement resumable streams and chat-completion control #1444, feat(server): add b10621 tokenization, template, infill, and native embedding utilities #1442, fix(server): align embedding and reranking mode, pooling, and normalization #1452)

This phase exists because waves 1 through 3 left every sub-issue implemented but seven of them unclosable. It is the last structural work the epic needs; after it the remaining sub-issues are either closable or genuinely unfinished, and the difference becomes visible rather than conflated.

Follow-ups filed during implementation

These were not in the original decomposition. Each was found by measuring against the pinned binary while a sub-issue was being implemented, and each owns manifest entries, so the epic's definition of done covers them too.

• fix(server): honor string stop sequences on the MLX serving path #1466: fix(server): honor string stop sequences on the MLX serving path
• fix(server): implement assistant prefill, reasoning-budget message, and completion echo #1470: fix(server): implement assistant prefill, reasoning-budget message, and completion echo
• fix: Close the remaining b10621 context and batching gaps #1472: fix: Close the remaining b10621 context and batching gaps
• feat: Close the remaining b10621 prompt-cache and slot-scheduling gaps #1473: feat: Close the remaining b10621 prompt-cache and slot-scheduling gaps
• feat: Close the native completion response and request-field gaps #1477: feat: Close the native completion response and request-field gaps
• feat(inference): complete the remaining b10621 sampling and grammar surfaces #1485: feat(inference): complete the remaining b10621 sampling and grammar surfaces

Execution grouping and what actually ran

Phases execute as three concurrent chains, each in one long-lived worktree so the MLX C++ build and the Rust artifact cache are amortized across the chain rather than rebuilt per issue. Waves 1 and 2 are complete; wave 3 has not started.

Two corrections to the plan this table originally carried, both established by measurement rather than by reading:

The chain C order was wrong. It began at #1436, but #1436's scope names top-n-sigma and says to reuse #1377 for typical_p, while #1377 was blocked on #1375, which introduces the batch-safe row-filter hook both reuse and which was not in the tree at all. #1375 also already owned top-n-sigma outright, so implementing it inside #1436 would have duplicated a live issue. The chain therefore ran #1375 and #1377 first. Neither is a sub-issue of this epic and neither owns a manifest shard.

The claim that #1436 required C++ was false. This section previously said the sampler filter chain lives in src/lib/mlx-cpp/turbo/sampling.cpp behind fused_sample / fused_sample_xtc and that new stages could not be added from Rust alone. Every stage landed in Rust with zero commits under src/lib/mlx-cpp/ or src/lib/mlxcel-core/cpp/. Top-n-sigma and typical_p are row-wise, history-free and RNG-free logit masks, so they compose from MLX graph ops ahead of the single ffi::fused_sample dispatch, lazily fuse into the same evaluation, and keep config_supports_fused_batch true. What pushed XTC into C++ was its RNG plumbing, not the fact that it is a filter. No sub-issue has bumped the pinned MLX commit.

Definition of done

• A machine-readable manifest covers every b10621 help entry, environment variable, registered route, and native request field with one compatibility-policy state and a linked test or issue.
• CI can compare the pinned manifest against both mlxcel serve --help and mlxcel-server --help, including hidden compatibility options and environment precedence.
• HTTP conformance tests cover route method/path, authentication exposure, accepted request shapes, status codes, streaming event termination, and response envelopes for all claimed endpoints.
• No option with observable b10621 semantics is accepted and silently ignored.
• README and operator documentation describe the verified compatibility boundary and link to the manifest rather than making an unqualified drop-in claim.
• Real-checkpoint smoke tests exercise one dense text model and one VLM for shared chat/completion/tokenization behavior; feature-specific checkpoints cover embeddings, reranking, speculative decoding, and LoRA where applicable.

Reference sources

• Arguments: https://github.com/ggml-org/llama.cpp/blob/c1d0e7a004015f23bc0233470b747b596f29b264/common/arg.cpp
• Routes: https://github.com/ggml-org/llama.cpp/blob/c1d0e7a004015f23bc0233470b747b596f29b264/tools/server/server.cpp
• Native schema: https://github.com/ggml-org/llama.cpp/blob/c1d0e7a004015f23bc0233470b747b596f29b264/tools/server/server-schema.cpp
• Server reference: https://github.com/ggml-org/llama.cpp/blob/c1d0e7a004015f23bc0233470b747b596f29b264/tools/server/README.md