[adrysn]

Summary

• Streaming /v1/chat/completions and /v1/responses now send tool-call arguments as they are generated for Hermes / Qwen JSON, Mistral [ARGS] and the [TOOL_CALLS] [..] list, Llama 3 JSON (bare or after <|python_tag|>), and Qwen3-Coder XML. Every other format keeps the one-shot end-of-stream emission.
• IncrementalToolCallStreamer (src/server/tool_calls/incremental.rs plus incremental/) reads the raw decoded text once, holding only a bounded tail (a partial marker or undecided name; long held values bail out and the final emission sends the rest). It skips thinking blocks and reuses the final parser's tool filter (resolve_tool_name), match_qwen_name, the shared Mistral header regex, the call caps, and a coerce_xml_param-locked string predicate.
• Behavior change outside streaming: for Hermes / Mistral / Llama 3 (and Granite, which delegates to Hermes), an object-valued arguments is now returned as the model's own text, not a re-serialization (new json_scan module), on both streamed and non-streamed replies. This is what makes the Chat deltas concatenate to exactly the final arguments. One parity expectation changed ({"city": "Seoul"}).
• End of generation: parse_tool_calls stays authoritative. plan_final matches streamed calls by index and name; a match gets only the unsent rest of its arguments. On Responses, .done and output_item.done carry the parsed arguments.

Tool-call id. Start carries the id the call keeps. A Mistral-list id is used only when it precedes name, validated with is_valid_model_call_id and de-duplicated as try_mistral_nemo does; otherwise the id is minted at Start, and a model id arriving later is ignored (comment on note_model_id). The final emission writes the streamed id back into the parsed calls, so Chat never mints a second id and the completed Responses object carries the streamed call_id.

Disagreement. A different count, name, or a non-prefix of the final arguments logs tracing::warn! with the request / response id. No further delta is sent for that call. The parsed call is emitted one-shot, under a fresh index when the streamed call used its slot. An unconfirmed Responses item is closed as incomplete. Named tool_choice mismatches keep their index slot and are never streamed; count_tool_calls_reaching_client still decides finish_reason.

Changes during review

• Char-boundary panics (CRITICAL / HIGH) (f29a87dc, b06f234e). step_outer stepped one byte at a time and sliced, so any non-ASCII prose panicked the per-token callback. On Chat this happened inside the cb_state lock. step_json (ObjectStep::Invalid) and the Mistral ValueLead had the same defect. All three now step whole characters, with regression tests on Korean and emoji inputs.
• Panic containment (b06f234e). IncrementalToolCallStreamer::feed runs under catch_unwind. A caught panic drops that fragment's events, rolls the newest call back to what the client received, stops incremental streaming for the request, and logs at error level. The final plan then sends the rest from the authoritative parse. A test injects a panic and checks the result.
• Responses output parity (5897a60f). For a streamed request that parses tool calls, response.completed and the stored response (previous_response_id) are now built from the items actually streamed: the same rs_ / msg_ / fc_ ids and output_index order, and each message segment as its own item. Before, the final object put calls first, merged the message segments, and minted new fc_ ids. Test: responses_completed_output_matches_the_streamed_items. call_position replaces linear scans with a binary search.
• Docs (44959fc7): the raw-arguments behavior change, panic containment and Responses parity are now in CHANGELOG.md, docs/server-features.md and docs/responses-api.md.

Measured (security review): 1.2M fuzz runs (every format, multibyte and control characters, random chunkings) found no panics after the fixes. 150k Qwen string-value runs: deltas always concatenate to valid JSON equal to the final parse. Throughput is 28 to 130 ns per 4-character token, including 1 MB prose, 1 MB argument values and 20k short calls. Held buffers are capped (256 B names and headers, 1 MiB values, 1024 calls).

Not in this PR: bare Qwen <function= calls without <tool_call> and several calls inside one <tool_call> are not streamed (the final parse still sends them). The number of Hermes blocks is uncapped, matching the final parser. Responses streaming still ignores max_tool_calls, as before.

Test plan

• cargo test --profile test-fast -p mlxcel --lib with server::tool_calls::, server::routes::, server::responses, server::muse_atem, server::router_front
• cargo clippy -p mlxcel --lib --tests -- -D warnings, cargo fmt --check
• Unit tests feed each case per character and under other chunkings and compare against parse_tool_calls: Hermes, two-call Mistral [ARGS], Qwen3-Coder string + int (deltas before </parameter>), named tool_choice, thinking, unknown tools, Mistral-list ids, bounded holds.
• Route tests drive the real handlers through scripted_streaming_for_route_tests (the muse_atem tests replay the route by hand) and show several argument chunks for one call on both endpoints.
• Not run here (no Metal or CUDA): cargo test --workspace --features metal,accelerate and the integrated Qwen3-Coder-30B-A3B-Instruct-4bit OpenAI SDK check.

Resolves #2396

[adrysn]

Security and performance review

Fixed in b06f234

• HIGH: two more char-boundary panics on untrusted model output. The JSON call reader consumed one byte of a rejected non-ASCII character (<tool_call>{"arguments"🎉: ...), and the Mistral bracket reader did the same for a multi-byte scalar value ([TOOL_CALLS]f[ARGS]日). Both now consume the whole character. Regression tests added.
• HIGH: IncrementalToolCallStreamer::feed now runs under catch_unwind. On a panic it drops the fragment's events, rolls the newest call back to what the client received, and stops streaming. The final plan sends the rest from the authoritative parse, so a panic can no longer poison the Chat cb_state mutex or unwind through the callback.

Verified, no change needed

• Fuzz: 1.2M runs over all formats with multibyte, control and marker fragments under per-char, whole and random chunkings, with no panic after the fix. Qwen string fragments always concatenate to valid JSON equal to the final parse.
• Throughput (test-fast, 4-char tokens): 28-130 ns per token across 1 MB of prose, 1 MB argument values (Hermes, Qwen, Mistral), 20k short calls, 1 MB of whitespace and 1 MB of [ nesting. This is far below the per-token decode cost, so the per-char Vec in step_outer is not material.
• Memory: names and headers are capped at 256 B (Mistral, Qwen), JSON names at 512 B, ids at 128 B, and held values at 1 MiB. Qwen and Mistral calls and params are capped at 1024. The JSON scanner is iterative with a counter, so deep nesting cannot overflow the stack.

Remaining (LOW, not fixed)

• Hermes <tool_call> count is uncapped, as in the final parser. ResponsesToolCallStream::item_for and plan_final scan calls linearly, which is O(calls^2) over a reply with thousands of calls. The cost is bounded by output length. Calls are pushed in increasing index order, so a binary search would fix it.