Repository navigation
fix(server): re-inject generated reasoning into content-only history - #2116
Merged
Merged
Conversation
inureyes
added
type:bug
inureyes
added a commit
that referenced
this pull request
Oct 5, 2026
Templates such as AI21 Jamba-Reasoning keep a thinking instruction on an earlier user turn only when the following assistant message carries reasoning_content. A client that echoes only content (the OpenAI SDK default) re-rendered that turn shorter than it was generated, so the history-boundary snapshot never prefixed the follow-up and every turn missed the prompt cache. The chat route now records the exact reasoning_content each plain reply returned (streaming and non-streaming, including replies cut off inside the thinking block) in a bounded store and fills it back into a later content-only assistant turn before rendering, as if the client had echoed it. The key covers model id, template_sig, resolved session, a digest of every preceding message as the client sent it, and the trimmed reply content, so any edit misses and a miss changes nothing. Echoed reasoning always wins. The store keeps only a 32-byte key and the text, is capped at 16 MiB and 4096 entries with LRU eviction (MLXCEL_REASONING_ECHO_MAX_BYTES, 0 disables), and runs only while the prompt cache is on. Validated with new unit and route tests (the route test fails with the store disabled), the chat_request, prompt_cache and routes filters, clippy and fmt. On GB10, jamba-v0.1-4bit content-only 3-turn chat went from cached_tokens 0/0/0 to 0/41/89; qwen3-0.6b-4bit is unchanged. Closes #2110
inureyes
force-pushed
the
fix/issue-2110-reasoning-reinjection
branch
from
October 5, 2026 14:19
1fbbc15 to
6e27ae7
Compare
This was referenced Oct 5, 2026
inureyes
added a commit
that referenced
this pull request
Oct 6, 2026
…#2121) PR #2116 re-injected server-generated reasoning into content-only assistant turns on /v1/chat/completions only, so Jamba-Reasoning missed the prompt cache on every turn of /v1/responses and /v1/messages, and the #1144 warm-up never ran for it: the probe rendered the reply without reasoning and the unfilled request, failed the extension check, and bailed. - /v1/responses and /v1/messages fill a render-only copy of the translated request and record the reply after it finishes, non-streaming and streaming, through new render_request and record_reply helpers. Responses records the reasoning item (previous_response_id chains replay no reasoning item, so they are covered too); Messages records the thinking block only when extended thinking was enabled, since otherwise the client never receives it. - The chat routes record before the warm-up. record now returns whether it stored the trace, and the warm-up renders the filled request plus the reply with that reasoning only when stored. render_next_turn_history takes the reply reasoning and picks the raw or typed render path with the same predicate as prepare_chat_request_with_cache. The streamed flush tail now reaches the warm-up reply. Route tests for both endpoints fail with MLXCEL_REASONING_ECHO_MAX_BYTES=0; warm-up tests check the target prefixes the filled follow-up. On GB10 with jamba-v0.1-4bit, non-streaming cached tokens went 0/0/0 to 0/52/102 (responses) and 0/51/101 (messages), and chat turn 2 hit the 91-token warm-up entry. Closes #2118
18 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A client that echoes only
content(the OpenAI SDK default) re-rendered Jamba-Reasoning's earlier user turn without its thinking instruction, so every follow-up missed the prompt cache. While the prompt cache is on,/v1/chat/completions(streaming and non-streaming) now records the exactreasoning_contenteach plain reply returned and fills it back into a later content-only assistant turn before rendering, as if the client had echoed it.Soundness
template_sig, resolved session (prompt_cache_key,user, or the anonymous bucket), a digest of every preceding message as the client sent it, and the trimmed reply content. A trace is only found by a request that could itself have produced it; editing the reply, any earlier message, kwargs, tools or session misses, and a miss changes nothing.<think>content are never filled.MLXCEL_REASONING_ECHO_MAX_BYTES(0 disables) is documented.cache_prompt: false,--reasoning-format none/deepseek-legacy, and--skip-chat-parsing. Not wired into/v1/responses,/v1/messages, prompt inspection, or the router front.Validation
"Before" is the same binary with the store disabled (
MLXCEL_REASONING_ECHO_MAX_BYTES=0); the issue records 0 on 33c4505. qwen3.5-0.8b-4bit hit max_tokens in thinking on every turn and was identical both ways (0/0/36).server::chat_request,server::prompt_cache,server::routes,reasoningfilters, clippy-D warnings, fmtCloses #2110
Closes #2110