You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[feature] Measure pre-answer latency for streamed requests #1230
A streaming client can disconnect while Switchyard is still waiting for upstream headers or the first SSE event. In #1168, an operator reported 499s during slow prefills but could not tell whether the client had seen any response. Existing handling_duration_ms does not separate upstream phases, and switchyard.total_latency_ms measures full turns only.
A loopback reproduction with a valid streamed response showed the same overall wait when delaying upstream headers versus delaying the first SSE event. Both paths returned HTTP 200, while a client disconnect before either milestone produced a 499 without a phase-specific wait.
Proposed solution
Record a per-attempt histogram from upstream dispatch to response headers, including non-2xx attempts and retries.
Record a per-request histogram from arrival to the first framed downstream SSE event.
For streaming 499s, log the elapsed pre-answer wait as a censored duration, not a completed first-event sample. Log the pending upstream phase when an attempt is abandoned, with the caller correlation ID when supplied.
Use configured model IDs and fixed phase names only. Do not add a setting or include request content in telemetry.
Alternatives considered
Full-turn latency cannot distinguish prefill from generation, and completed-only timing omits disconnects. Sending early headers or keepalives changes HTTP behavior and is separate from this measurement work.
Scope notes
The upstream client owns attempt timing; the server owns downstream framing and 499 logging. This is additive telemetry with no public API, request, response, or configuration change. Related to #1168; it does not implement keepalives.
Activity