Repository navigation
feat(server): implement resumable streams and chat-completion control #1444
Description
Parent
Problem
b10621 provides resumable stream creation/lookup/deletion and realtime chat-completion control, including reasoning-budget updates. mlxcel supports ordinary SSE responses and responses retrieval/cancellation, but those are not aliases for the b10621 stream lifecycle.
Scope
- Implement compatible stream identifiers, buffering, lookup, replay cursor, completion, expiration, and DELETE semantics.
- Implement chat-completion control requests for fields b10621 permits to change during generation, including reasoning budget.
- Define authorization ownership so one API key or tenant cannot inspect/control another stream.
- Bound retained events and stream lifetime; handle process shutdown, client disconnect, and model error deterministically.
- Keep the OpenAI responses retrieve/cancel surface separate unless an exact translation is demonstrated and tested.
Acceptance criteria
- A disconnected client can resume without missing or duplicating committed events.
- Lookup and delete return compatible status codes for active, completed, expired, unknown, and unauthorized streams.
- Control updates have defined event-order boundaries and reject immutable or invalid fields.
- Memory usage is bounded under abandoned and slow-consumer streams.
- Ordinary non-resumable SSE and OpenAI responses retrieval/cancellation remain regression-tested.
Validation
- Add integration tests that disconnect at multiple event boundaries, replay, update a reasoning budget, delete the stream, and attempt cross-key access.
Activity
test(server): add pinned b10621 compatibility manifest and gates
feat(server): implement b10621 resumable streams and completion contr…
inureyes commentedon Aug 28, 2026
PR #1491 implemented and merged every acceptance criterion of this issue: the resumable-stream lifecycle (GET/DELETE /v1/stream, POST /v1/streams/lookup) with upstream's buffer, TTL, GC and status-code semantics on all five streaming surfaces, POST /v1/chat/completions/control with the reasoning_end action, the reasoning_control request field (manifest state supported), per-key stream and control ownership, bounded retention, and integration tests covering disconnect-and-resume at byte granularity, cross-key access, DELETE cancellation, and the control contract.
The issue stays open deliberately: the four route entries in compat/llama-server/b10621/streams-and-realtime.json remain deferred against this issue because the cross-key authorization ownership this issue mandates is an externally observable divergence from b10621, which treats every configured API key as equivalent. The manifest's divergence rule forbids supported for a diverging entry, and the divergence is permanent by design, so this issue continues to own those entries, the same way #1442, #1446 and #1452 hold theirs.
feat(compat): add the by_design b10621 compatibility-policy state
inureyes commentedon Aug 30, 2026
Closing: this issue's implementation merged, and the entries that kept it open now carry a state that describes them.
The work for resumable streams and chat-completion control landed and was verified at the time. What remained were manifest entries recording behavior that differs from llama-server b10621 on purpose and permanently, which none of the four original states could express: not
supportedbecause the behavior differs, notaliasedbecause it is not equivalent, notnot_applicablebecause the concept applies and is served, and notdeferredbecause nobody intended to close the gap. The compatibility gate requires adeferredentry's issue to stay open, so this issue was pinned open by a vocabulary gap rather than by unfinished work.#1499 added the
by_designstate for exactly that case, and PR #1500 moved this issue's entries into it. Each carries arationaleobject the validator requires in full: akindseparating an architectural impossibility from a policy choice, areason, and arevisit_ifnaming what would have to change for the decision to be reconsidered. Each is pinned by a test, not by a note.Verified before closing: this issue now owns zero
deferredentries, so closing it cannot redden thellama-compat manifestjob for other pull requests. That failure mode happened twice earlier in epic #1431 and is what the check exists to prevent.test(server): add pinned b10621 compatibility manifest and gates
feat(server): implement b10621 resumable streams and completion contr…