feat: Close the native completion response and request-field gaps #1477

Part of #1431.

Problem / Background

This continues #1441 (merged as PR #1467 and PR #1476) and carries the four divergence strings that remain on the POST /completion and POST /completions entries in compat/llama-server/b10621/routes.json:154-197. PR #1467 separated the native /completion and /embeddings routes from the OpenAI routes so /completions and /embeddings answer the b10621 native shape while /v1/completions and /v1/embeddings stay OpenAI compatible, and PR #1476 added the response_fields projection, the stream_options block, the whole n_predict value domain including -1 and the verified 0 prompt-only case, the real prefill figures on every streaming frame, and two corrected timings derivations. That work is merged, correct, and truthfully classified: deferred is epic #1431's fourth compatibility state, "a linked implementation issue exists with explicit acceptance criteria; the current binary must not claim support by silently ignoring it". This issue is the tracked continuation of that state, not a regression report against either PR.

It has to exist as its own issue for a mechanical reason. scripts/ci/check_llama_compat_manifest.py --check-issues-open resolves every deferred entry's issue through gh issue view and fails when it is not OPEN (scripts/ci/check_llama_compat_manifest.py:445-468 and scripts/ci/check_llama_compat_manifest.py:624-640), and that flag is exactly what the llama-compat manifest CI job runs (.github/workflows/ci.yml:244-273). Closing #1441 while the two route entries still point at it turns that job red on every open pull request in the repository. Repointing the entries here lets #1441 close against what it delivered.

Current Behavior

Four divergence strings, identical on both route entries, quoted verbatim from the manifest as it stands today. If the implementation restates rather than removes one, this is the text it is restating.

`generation_settings` echoes the settings mlxcel actually resolved rather than b10621's full task_params block, so keys mlxcel has no analogue for are absent (#1441).
`id_slot` is always -1: the continuous-batching scheduler exposes no stable per-request slot number, so exposing one is #1440's scope, not this entry's (#1440).
`tokens_cached` reports the prompt tokens mlxcel's KV prefix cache supplied for this request, while b10621 reports the slot's cache occupancy after it, which is `tokens_evaluated + tokens_predicted - 1`: a 51-token prompt answered with 4 tokens measured `tokens_cached: 54` on the pinned binary against mlxcel's cache-supplied count (#1441).
`return_tokens`, `return_progress`, `verbose`, `n_indent`, `t_max_predict_ms` and `n_cmpl` above 1 are refused with a diagnostic rather than honored (#1441).

The four differ in character and are not one piece of work. One is unimplemented scope, one is a one-field formula change, one shrinks on its own as other sub-issues land, and one belongs to a different issue entirely.

1. Six native request fields are refused rather than honored

This is the substantive one, and it is unimplemented scope rather than an unavoidable platform difference: #1441's own scope text asked for "tokens/prompt when requested" and "progress reporting". reject_unsupported_native_fields (src/server/routes/native_completion.rs:515-562, called at src/server/routes/native_completion.rs:131) accepts each field at its inert value and returns a 400 invalid_request_error naming the field otherwise: n_cmpl at 1, n_indent at 0, t_max_predict_ms at 0 or below, and return_progress / verbose / return_tokens at false. The fields are declared on NativeCompletionRequest at src/server/types/request.rs:1132-1146. Their manifest entries are field:n_cmpl (routes.json:529-548), field:n_indent (:549-566), field:return_progress (:606-623), field:return_tokens (:624-641), field:t_max_predict_ms (:696-713) and field:verbose (:732-749), all currently not_applicable with a "no equivalent" justification in notes that stops being true the moment a field is honored.

return_tokens. b10621 fills the top-level tokens array with the raw generated token ids and leaves it [] otherwise. A capture of the pinned binary answering The capital of France is with 8 tokens returned tokens: [12095, 13, 576, 6722, 315, 15344, 374, 21718] against tokens_predicted: 8. mlxcel's response and chunk types already carry the field (src/server/types/native_completion.rs:173 and src/server/types/native_completion.rs:206) and always send it empty: the response builder forwards outcome.tokens at src/server/routes/native_completion.rs:289 and every call site passes Vec::new() (src/server/routes/native_completion.rs:221, :442, :480). The ids exist: Sequence::generated_tokens: Vec<i32> at src/server/batch/sequence.rs:246. Nothing carries them out. GenerationResult (src/server/model_provider.rs:218) has no id vector, and the native streaming callback is FnMut(String, Option<TokenLogprobData>) (src/server/model_provider.rs:1416), where TokenLogprobData.token_id (src/lib/mlxcel-core/src/sampling.rs:1151-1158) is populated only when logprobs are requested, so it cannot be the carrier for a request that did not ask for logprobs.

return_progress. b10621 opens the stream with frames that carry prompt_progress: {total, cache, processed, time_ms} and empty content. Captured from the pinned binary on a 1-token prompt: "prompt_progress":{"total":1,"cache":0,"processed":0,"time_ms":0} on the first frame and "prompt_progress":{"total":1,"cache":0,"processed":1,"time_ms":8} on the second, both with tokens_predicted: 0, before the first content frame. mlxcel already has the type (PromptProgress, src/server/types/native_completion.rs:155-161) and the chunk field (src/server/types/native_completion.rs:214), hardwired to None at src/server/routes/native_completion.rs:456. The scheduler emits exactly one GenerateEvent::Prefill(PrefillStats) per request (src/server/model_provider.rs:146-180, sent from src/server/batch/sequence.rs:364-369) when the first token is stamped, which supplies total, cache and time_ms but only the terminal processed == total.

verbose. CORRECTED against the pinned binary (2026-08-30): b10621 attaches NO __verbose block to the native /completion response. __verbose is emitted only from the OAI-compat shapes, at server-task.cpp lines 404, 452, 521 and 1098, each inside a to_json_oaicompat* builder, and the native response IS to_json_non_oaicompat(). Captured with {"verbose": true} against the pinned binary, the top-level key set is identical to the same request without it: content, generation_settings, has_new_line, id_slot, index, model, prompt, stop, stop_type, stopping_word, timings, tokens, tokens_cached, tokens_evaluated, tokens_predicted, truncated. So upstream accepts the field on this route and ignores it, and mlxcel refusing it with a 400 is the divergence. The work is to accept it inert, not to build a block.

n_indent. b10621 stops the completion when a generated line's indentation falls below the given number of whitespace characters, for fill-in-the-middle clients. mlxcel has no such stop rule; the only per-request stop machinery is the stop-string matcher reached through Sequence::stream_decoded_text (src/server/batch/sequence.rs:387).

t_max_predict_ms. b10621 bounds the prediction phase per request, measured from the first token, and the timeout fires only once a newline has been generated. mlxcel bounds a stalled decode with the server-wide --decode-timeout / MLXCEL_DECODE_TIMEOUT watchdog, which cannot be narrowed per request.

n_cmpl above 1. CORRECTED against the pinned binary (2026-08-30): the clamp is a schema limit that REFUSES rather than clamping, and its upper bound is the server's own slot count. Running the pinned binary with -np 1, {"n_cmpl": 2} answers 400 Field 'n_cmpl': Value must be between 1 <= value <= 1, but got 2, which is what mlxcel already does; running it with -np 4, the same request returns a JSON array of two result objects each carrying its own index. So mlxcel's refusal is byte-equivalent to upstream's at one slot and diverges only above it, which is what the terminal state has to describe. mlxcel serves one completion per request and has no path that returns an array.

2. tokens_cached uses a different definition

build_native_response sets tokens_cached: outcome.cached_tokens at src/server/routes/native_completion.rs:305, which is the prompt-prefix-cache count fed from GenerationResult::cached_tokens on both the non-streaming path (src/server/routes/native_completion.rs:224) and the final streaming frame (src/server/routes/native_completion.rs:483). b10621 reports the slot's cache occupancy after the request instead. Four independent measurements on the pinned binary agree on tokens_evaluated + tokens_predicted - 1: 5 + 8 - 1 = 12 on a limit stop, 10 + 91 - 1 = 100 on an eos stop, 8 + 4 - 1 = 11 on a word stop whose matched string was excluded from content, and the 51 + 4 - 1 = 54 already recorded in the manifest. This is one field on one line and is independently fixable.

3. generation_settings is a subset

native_generation_settings (src/server/routes/native_completion.rs:322-348) emits 19 keys: seed, temperature, top_k, top_p, min_p, xtc_probability, xtc_threshold, repeat_penalty, presence_penalty, frequency_penalty, dry_multiplier, dry_base, dry_allowed_length, dry_penalty_last_n, dry_sequence_breakers, stop, max_tokens, n_predict, stream. b10621 echoes its whole task_params block, 49 keys, a strict superset of mlxcel's 19. The 30 absent keys are dynatemp_range, dynatemp_exponent, top_n_sigma, typical_p, repeat_last_n, mirostat, mirostat_tau, mirostat_eta, adaptive_target, adaptive_decay, n_keep, n_discard, ignore_eos, logit_bias, n_probs, min_keep, grammar, grammar_lazy, grammar_triggers, preserved_tokens, chat_format, reasoning_format, reasoning_in_content, generation_prompt, samplers, speculative.types, timings_per_token, post_sampling_probs, backend_sampling, lora. This one shrinks on its own: #1436 owns the sampler side (typical_p, top-n-sigma, dynamic temperature, Mirostat, adaptive-p, sampler ordering, ignore-EOS, grammar and preserved tokens, logit_bias, n_probs, min_keep, repeat_last_n) and #1433 owns speculative.types. Re-measure the key set after those land rather than chasing the individual keys here.

4. id_slot is always -1, and is NOT in scope here

build_native_response sets id_slot: -1 at src/server/routes/native_completion.rs:293 and the streaming frames do the same at src/server/routes/native_completion.rs:444. mlxcel's continuous-batching scheduler exposes no stable per-request slot number, and slot exposure is #1440's scope (observability-and-slots.json, owner [1440]). This issue must not implement it, must not claim it, and must not remove its divergence string. It is listed here only because it sits on the same two manifest entries, and it is why those entries stay deferred even after everything this issue owns is closed, unless #1440 lands first.

Proposed Solution

Carry the token ids out of the scheduler for return_tokens. Add an opt-in id channel rather than turning on logprobs internally: extend GenerationResult with the generated ids and give the native streaming path per-frame ids, populated only when the request asked for them so the decode hot path does not clone an id vector for every request. Sequence::generated_tokens is already the accumulator, so the non-streaming half is a move rather than new bookkeeping. The frame-level half must ride with the emitted piece through Sequence::stream_decoded_text (src/server/batch/sequence.rs:387), not be re-derived from the text, because that funnel holds back an ambiguous tail while a stop string is being matched and a frame's text can therefore span several tokens or none.

Emit real progress frames for return_progress. The one-shot GenerateEvent::Prefill gives a terminal snapshot only. Chunked prefill already exists (prefill_chunk_size, threaded from src/server/model_provider.rs:670 and its sibling constructors), so per-chunk progress is reachable by emitting a progress event per prefill chunk and mapping it onto PromptProgress in the native route. If the implementation emits only the terminal snapshot, the entry keeps a restated divergence naming the missing intermediate frames rather than flipping to supported.

Capture verbose before implementing it. Run the pinned binary with "verbose": true, record the __verbose key set, and fill only the keys mlxcel can source honestly. Invent no value for a key with no analogue: a key mlxcel cannot fill is omitted, exactly as generation_settings already handles the same problem. If the resulting block is a strict subset, the entry stays refused or is restated with the absent keys named; it does not become supported.

Implement n_indent and t_max_predict_ms as stop rules. Both are per-request generation bounds with observable stop metadata, so both need the stop_type upstream reports when they fire, measured rather than assumed. n_indent belongs next to the stop-string matcher on the sequence; t_max_predict_ms is a per-request deadline checked in the decode loop, gated on the "a newline has been generated" condition upstream applies, and distinct from the server-wide --decode-timeout watchdog, which stays as it is.

Decide n_cmpl on evidence, not on ambition. Honoring it means the response becomes a JSON array, the streaming frames carry a per-completion index, and the value is clamped to the upstream range. If the batch scheduler cannot serve N completions for one request within this issue, the correct outcome is not_applicable with a refreshed justification and a refreshed refusal test, not a partial array shape. Note the scope overlap: #1436's scope bullet mentions "multiple completions", but field:n_cmpl lives in routes.json, whose owner set gates who may edit it, so the manifest entry is this issue's to flip while any sampler-stage work stays in sampling-and-grammar.json.

Change tokens_cached to the upstream formula. Set it to tokens_evaluated + tokens_predicted - 1 with a saturating subtraction, and leave timings.cache_n fed from outcome.cached_tokens, which is a different quantity: NativeTimings documents cache_n as the cache-supplied prompt tokens (src/server/types/native_completion.rs:52-56) and derives prompt_n = tokens_evaluated - cache_n from it (src/server/types/native_completion.rs:101). Conflating the two would silently break prompt_n, which PR #1476 verified against the binary.

Leave generation_settings alone for now beyond re-measuring it after #1436 and #1433, and leave id_slot to #1440.

Scope

In scope: src/server/routes/native_completion.rs (the refusal block, the response builder, the streaming frame builder), src/server/types/native_completion.rs (the progress and token id shapes if they need to change), src/server/types/request.rs (only if a field's parsing has to change), src/server/model_provider.rs and src/server/batch/sequence.rs (the id and progress channels), src/server/routes/native_route_tests.rs and src/server/types/native_completion_tests.rs (tests), the six field:* entries and the two route entries in compat/llama-server/b10621/routes.json, and docs/llama-server-compat.md where the native-route section describes the boundary.

Out of scope: id_slot and slot exposure (#1440); the sampler stages behind the absent generation_settings keys (#1436) and speculative.types (#1433); the OpenAI routes under /v1/, whose behavior must not change; compat/llama-server/b10621/pin.json's reference block and counts; every other shard.

Implementation Notes

• Reuse: PromptProgress and the tokens field already exist on the native types, Sequence::generated_tokens already accumulates the ids, GenerateEvent::Prefill / PrefillStats is the existing prefill-observation channel, and Sequence::stream_decoded_text is the single funnel every serving path uses to reach the client. Extend these rather than adding a parallel path; build_native_response is deliberately the one function both the streaming and non-streaming shapes are built through, and that must stay true.
• Manifest gate coupling: the manifest validator requires a deferred entry's issue to be OPEN, so this issue must not be closed while either route entry still points at it. Repointing the two entries also requires this issue's number in pin.json shards.routes.owners, because check_shard_ownership (scripts/ci/check_llama_compat_manifest.py:423-442) fails an entry whose issue is not an owner of its own shard.
• State choice is constrained: a non-empty divergence on a supported entry is a hard gate error (scripts/ci/check_llama_compat_manifest.py:247-273, asserted again from the Rust side in supported_entries_record_no_divergence_from_b10621, src/server/llama_compat_tests.rs:310). Each entry this issue touches must end as supported (empty divergence, an mlxcel claim, a named conformance test), aliased, or not_applicable with refreshed notes and a named diagnostic test. Do not assume supported is the target for all six fields.
• Conformance tests read the manifest: manifest_route_claims_match_the_mounted_router (src/server/llama_compat_tests.rs:145) and manifest_native_field_claims_match_native_completion_request (src/server/llama_compat_tests.rs:230) assert the claim halves against the real router and the real request struct, and check_test_function_exists (scripts/ci/check_llama_compat_manifest.py:391-420) rejects a file::fn pointer that names no such function. A manifest edit and the code change land together or the gate fails.
• Edge cases: tokens_predicted == 0 (the verified n_predict: 0 prompt-only case) makes the upstream tokens_cached formula underflow, so measure what the binary reports there before choosing between tokens_evaluated - 1 and tokens_evaluated, and use a saturating subtraction either way. A word stop excludes the matched string from content while still counting its tokens in tokens_predicted, so measure whether upstream includes those ids in tokens under return_tokens. A recorded stream capture shows intermediate frames carrying per-token ids while the final frame carried tokens: [], but the request body of that capture was not recorded, so re-measure with return_tokens both false and true before deciding whether the per-frame ids are unconditional. A fully cache-hit prompt reports prompt_n: 1 against a larger tokens_evaluated, so a progress frame must not assume processed starts at 0 for the uncached part. A cancelled or failed request must still terminate the stream with a frame the client can act on, which the current error arm at src/server/routes/native_completion.rs:491-501 already does.
• Error handling: a field that stays refused keeps its 400 invalid_request_error naming the field and the alternative, in the existing envelope. A field that becomes honored must be validated at its value domain before generation starts and refused with the same envelope when out of range, never silently clamped. No field may be accepted and silently ignored, which is epic epic(server): close llama-server b10621 compatibility gaps #1431's Definition of Done.
• The b10621 capture set from fix(server): separate native completion and embedding routes from OpenAI routes #1441 lives in a session scratchpad and is not durable. Re-capture from the pinned binary rather than trusting a copied number; pin.json carries the archive URL and its sha256 for a fresh fetch.

Acceptance Criteria

• tokens_cached on both POST /completion and POST /completions equals tokens_evaluated + tokens_predicted - 1 (saturating at 0), verified against the pinned binary for a limit stop, an eos stop, a word stop, and the n_predict: 0 prompt-only case, with timings.cache_n and timings.prompt_n unchanged and still matching the binary on the same requests.
• return_tokens: true returns the generated token ids in the top-level tokens array on both routes and both shapes, and return_tokens: false (and its absence) returns []; for a request with no stop string the id count equals tokens_predicted and detokenizing the ids through POST /detokenize reproduces the emitted content; for a word stop the ids match what the binary returns on the same request rather than an assumption about the matched string; the per-frame streaming behavior matches the binary as re-measured.
• return_progress: true emits prompt_progress: {total, cache, processed, time_ms} frames with empty content ahead of the first content frame, with total equal to the prompt length and a final progress frame at processed == total, and emits none when the field is absent or false; if only the terminal frame is emitted, the route entry carries a restated divergence naming the missing intermediate frames instead of claiming support.
• verbose: true is accepted and changes nothing in the native response, matching the pinned binary, whose __verbose block exists only on the OAI-compat shapes; the entry is supported on that evidence rather than carrying a block mlxcel invented.
• n_indent stops the completion at upstream's indentation rule with the stop_type the binary reports for it, and n_indent: 0 is unchanged behavior; if the rule is not implemented, field:n_indent stays not_applicable with refreshed notes and a refusal test that asserts the diagnostic.
• t_max_predict_ms bounds the prediction phase per request from the first token, fires only after a newline has been generated, reports the stop_type the binary reports, and does not change the server-wide --decode-timeout behavior; if it is not implemented, field:t_max_predict_ms stays not_applicable with refreshed notes and a refusal test.
• n_cmpl: 2 either returns a JSON array of two result objects with per-completion index on the streaming frames and upstream's clamping, or field:n_cmpl stays not_applicable with refreshed notes recording that the refusal is byte-equivalent to upstream's own at one slot and a refusal test; no partial array shape ships.
• generation_settings is re-measured against the pinned binary after feat(inference): complete b10621 sampling and grammar semantics #1436 and feat(inference): map b10621 speculative flags to MTP and DFlash semantics #1433 have landed, and the route entries' divergence string is either removed or restated naming the keys that are still absent at that point.
• id_slot is untouched: the route entries keep their id_slot divergence string naming feat(server): align props, slots, metrics, health, and slot persistence #1440 as its owner.
• The two route entries in compat/llama-server/b10621/routes.json carry no divergence string owned by this issue: the generation_settings string, the tokens_cached string and the refused-fields string are each removed or restated with fresh evidence and an owning issue named, and any restated string names an issue that is open.
• Each of the six field:* entries ends as supported with an mlxcel claim and a named conformance test, or not_applicable with notes that describe the current refusal and a named diagnostic test; none is supported with a non-empty divergence.
• Every field that becomes honored is reachable through the real route on a running server against a real checkpoint, not only through a unit test on the request struct: the acceptance evidence is an HTTP response from mlxcel-server, compared against the same request answered by the pinned binary.
• docs/llama-server-compat.md describes the resulting native-route boundary, including any field that stays refused and why.
• python3 scripts/ci/check_llama_compat_manifest.py --check-issues-open passes, and the full gate (make verify) passes.

Verification

# The pinned b10621 reference (build 10621, commit c1d0e7a00). The scratchpad copy is
# session-local; pin.json carries the archive URL and sha256 for a fresh fetch.
B10621=a local extraction of the pinned archive (see `pin.json` for its URL and SHA-256)
DYLD_LIBRARY_PATH="$B10621" "$B10621/llama-server" --version   # expect: build 10621, commit c1d0e7a00
DYLD_LIBRARY_PATH="$B10621" "$B10621/llama-server" -hf ggml-org/Qwen3-0.6B-GGUF --port 18621 &

# mlxcel on the real checkpoint.
cargo build --release --features metal,accelerate
./target/release/mlxcel-server -m models/mlx/qwen3-0.6b --port 18622 &

# Same native request against both, field by field.
REQ='{"prompt":"The capital of France is","n_predict":8,"return_tokens":true}'
curl -s localhost:18621/completion -H 'content-type: application/json' -d "$REQ" | python3 -m json.tool
curl -s localhost:18622/completion -H 'content-type: application/json' -d "$REQ" | python3 -m json.tool
# Repeat for /completions, for {"return_progress":true,"stream":true}, {"verbose":true},
# {"n_indent":4}, {"t_max_predict_ms":50}, {"n_cmpl":2}, and {"n_predict":0}.

# Gates.
make verify-llama-compat
GH_TOKEN=$(gh auth token) python3 scripts/ci/check_llama_compat_manifest.py --check-issues-open
cargo test --workspace --profile test-fast --features metal,accelerate
cargo clippy --workspace --all-targets -- -D warnings
cargo fmt --all -- --check

A pass is: every honored field's mlxcel response matches the pinned binary's for that field on the same request (model id, timing values, and id_slot normalized away), every refused field returns the 400 naming it, the manifest validator reports no error with --check-issues-open, and the workspace test total is unchanged apart from the tests this issue adds.

Technical Considerations

• Epic: epic(server): close llama-server b10621 compatibility gaps #1431. Predecessor: fix(server): separate native completion and embedding routes from OpenAI routes #1441 (PR feat(server): separate the native completion and embedding routes #1467, PR feat(server): honor response_fields, stream_options and real prefill timings #1476). Related: fix(server): honor string stop sequences on the MLX serving path #1466 (stop-sequence half of the native route), feat(server): align props, slots, metrics, health, and slot persistence #1440 (id_slot and slot exposure), feat(inference): complete b10621 sampling and grammar semantics #1436 (sampling keys in generation_settings), feat(inference): map b10621 speculative flags to MTP and DFlash semantics #1433 (speculative.types).
• Upstream reference: tools/server/server.cpp at ggml-org/llama.cpp commit c1d0e7a004015f23bc0233470b747b596f29b264, https://github.com/ggml-org/llama.cpp/blob/c1d0e7a004015f23bc0233470b747b596f29b264/tools/server/server.cpp, with the request schema at https://github.com/ggml-org/llama.cpp/blob/c1d0e7a004015f23bc0233470b747b596f29b264/tools/server/server-schema.cpp.
• The /v1/completions and /v1/embeddings OpenAI shapes must not move. PR feat(server): separate the native completion and embedding routes #1467 already published the breaking change for the non-/v1 paths; this issue adds fields inside the native shape and must not introduce a second migration.