Repository navigation
fix(server): separate native completion and embedding routes from OpenAI routes #1441
Description
Parent
Problem
The native /completion response and streaming envelope expose only a subset of b10621 fields. More seriously, /completions is routed to an OpenAI-shaped handler in mlxcel while b10621 treats it as a native llama-server alias. /embeddings has the same native-versus-OpenAI collision.
Scope
- Align native
/completionrequest parsing, default resolution, cache controls, token-return options, stop metadata, timings, and streaming termination. - Route
/completionsto the native handler and retain OpenAI behavior exclusively under/v1/completions. - Route
/embeddingsaccording to the b10621 native schema and retain OpenAI behavior under/v1/embeddings. - Add compatible native fields including resolved generation settings, tokens/prompt when requested, cached/truncated state, stop type, stopping word, and progress reporting.
- Align reverse-prompt stopping and special-token output controls, including their interaction with streaming and returned token IDs.
- Match error status/envelopes and client-disconnect cancellation.
- Publish a migration note because the current non-
/v1routes return different schemas.
Acceptance criteria
- Route method/path tests prove native and OpenAI handlers are no longer conflated.
- Non-streaming and streaming golden-schema tests cover normal length, EOS, stop-string, cancellation, cache hit, and truncation outcomes.
- Reverse-prompt and special-token tests distinguish emitted text, stop metadata, raw token IDs, and detokenization behavior.
- Unknown native request fields are rejected or reported consistently rather than silently ignored when they affect semantics.
-
/v1/completionsand/v1/embeddingsremain OpenAI compatible. - Existing non-
/v1behavior change is documented as breaking.
Validation
- Compare the same prompt against b10621 and mlxcel using a real checkpoint, normalizing only backend-specific timing and model identifiers.
Activity
test(server): add pinned b10621 compatibility manifest and gates
inureyes commentedon Aug 27, 2026
Partial implementation landed in #1467. This issue stays OPEN: closing it would strand manifest entries that are not honestly resolvable yet, and
scripts/ci/check_llama_compat_manifest.py --check-issues-openfails the whole repository on adeferredentry whose issue is closed.Done in #1467:
/completionand/completionsboth reach the native handler and answer b10621's full native object (index,content,tokens,id_slot,stop,model,tokens_predicted,tokens_evaluated,generation_settings,prompt,has_new_line,truncated,stop_type,stopping_word,tokens_cached,timingswithcache_nleading); the streaming form ends with that same object and no[DONE]sentinel;/embeddingis mounted for the first time and/embeddingsanswers upstream's bare array of{index, embedding}with the nestedembedding;/v1/completionsand/v1/embeddingsare unchanged and stay OpenAI compatible;n_predictgains themax_tokensandmax_completion_tokensaliases; andn_cmplabove 1,n_indent,t_max_predict_ms,return_progress,verboseandreturn_tokensare refused with a diagnostic naming the field instead of being silently ignored. The key sets, orders and thestop_typevocabulary were captured from the pinned b10621 binary serving real checkpoints.Remaining, in the order a follow-up should take them:
ServerGenerateOptions::stop_sequencesand are never read; only thexla-iree-gated worker consumes them. Measured onmodels/mlx/qwen3-0.6b:/completionwith and without"stop":["5"]returns byte-identical text. Until that lands,stop_typehere can never reportwordandstopping_wordis always empty, which is what keepsPOST /completionandPOST /completionsdeferred.generation_settingsreports the settings mlxcel resolved rather than upstream's fulltask_paramsblock, andid_slotis always-1because the continuous-batching scheduler exposes no stable per-request slot number. Both are recorded as divergences on the two route entries.field:response_fields,field:stream_optionsandfield:stream_options.include_usageare untouched and stilldeferred.return_tokensimplementation.GET /health/GET /v1/health/GET /models/GET /v1/modelsentries are filed under this issue only becauseroutes.json's owner set inpin.jsonis[1441, 1442, 1452]; theirdivergencetext names feat(server): implement b10621 router mode and resolve models-dir semantics #1438 and feat(server): align props, slots, metrics, health, and slot persistence #1440 as the real owners. Whether to add those issues to that owner set and repoint the entries is a call for the epic owner, so feat(server): separate the native completion and embedding routes #1467 changed neither.fix(compat): reconcile manifest ownership and fix trace perplexity
fix(server): honor string stop sequences on the MLX serving path
inureyes commentedon Aug 27, 2026
PR #1476 closes the implementable remainder and this issue deliberately stays OPEN, for the same reason it stayed open after #1467: two manifest entries filed under it are still
deferred, and the validator requires a deferred entry's issue to be OPEN.Flipped to
supportedinroutes.json, each verified against the pinned b10621 binary serving a real checkpoint:field:response_fields,field:stream_options,field:stream_options.include_usage,field:timings_per_tokenandfield:n_predict. That covers acceptance criterion "unknown native request fields are rejected or reported consistently" for the four fields that were simply unimplemented, and then_predict: 0prompt-only case is now measured rather than assumed: the binary emits exactly one token for it despite its schema description saying none, and mlxcel emits the same one token on the same prompt.Still
deferredunder this issue:POST /completionandPOST /completions, with three divergences.generation_settingsis mlxcel's resolved set rather than upstream's fulltask_paramsblock.tokens_cachedcounts the prompt tokens mlxcel's prefix cache supplied while b10621 reports the slot's cache occupancy after the request (tokens_evaluated + tokens_predicted - 1), which is newly recorded by that PR and was not on the list before. And the six fields mlxcel refuses rather than honors, each of which has its ownnot_applicableentry and does not itself block this issue.The
id_slotdivergence line moved: exposing a stable per-request slot number is #1440's scope, so the line now names #1440 as its owner. The entries themselves stay filed here, because their other divergences are this issue's.The remaining
generation_settingsandtokens_cachedwork is response-body fidelity with no dependency on the field work, so it probably wants its own issue rather than keeping this one open indefinitely. That call belongs to the epic owner and no follow-up has been filed.fix(server): cover GenerateEvent::Prefill in the xla-iree test helpers
feat(server): honor response_fields, stream_options and real prefill …
fix(compat): move the native completion remainder to #1477
inureyes commentedon Aug 28, 2026
Closing against what this issue delivered.
PR #1467 moved
/completionsand/embeddingsto the native handlers, mounted/embeddingfor the first time, gave the native completion object b10621's full sixteen-key set, and ended streaming on that object with no[DONE]sentinel. PR #1476 then addedresponse_fields,stream_optionsandinclude_usage, corrected the timings derivations against the pinned binary, maden_predict: -1work where it had been rejected with HTTP 422, and verified then_predict: 0prompt-only case. Five field entries moved tosupported.What this issue did not finish now belongs to #1477, and the eight manifest entries that still described it were repointed there in #1478, so no
deferredentry references this issue and the compatibility gate stays green. The substantive remainder is the six native request fields that are refused rather than honored, two of which (return_tokensandreturn_progress) this issue's own scope named; #1477 owns them with per-field acceptance criteria.id_slotstays recorded but names #1440, because exposing a stable per-request slot number is that issue's scope.generation_settingsshrinks on its own as #1436 and #1433 land their options, so #1477 re-measures it rather than chasing it now.feat(server): align props, slots, metrics, health, and slot persisten…
feat(compat): add the by_design b10621 compatibility-policy state
feat(server): separate the native completion and embedding routes (la…