Skip to content

[feature] Measure pre-answer latency for streamed requests #1230

Open

Description

@cpakkamisaac-sae

Problem

A streaming client can disconnect while Switchyard is still waiting for upstream headers or the first SSE event. In #1168, an operator reported 499s during slow prefills but could not tell whether the client had seen any response. Existing handling_duration_ms does not separate upstream phases, and switchyard.total_latency_ms measures full turns only.

A loopback reproduction with a valid streamed response showed the same overall wait when delaying upstream headers versus delaying the first SSE event. Both paths returned HTTP 200, while a client disconnect before either milestone produced a 499 without a phase-specific wait.

Proposed solution

  • Record a per-attempt histogram from upstream dispatch to response headers, including non-2xx attempts and retries.
  • Record a per-request histogram from arrival to the first framed downstream SSE event.
  • For streaming 499s, log the elapsed pre-answer wait as a censored duration, not a completed first-event sample. Log the pending upstream phase when an attempt is abandoned, with the caller correlation ID when supplied.

Use configured model IDs and fixed phase names only. Do not add a setting or include request content in telemetry.

Alternatives considered

Full-turn latency cannot distinguish prefill from generation, and completed-only timing omits disconnects. Sending early headers or keepalives changes HTTP behavior and is separate from this measurement work.

Scope notes

The upstream client owns attempt timing; the server owns downstream framing and 499 logging. This is additive telemetry with no public API, request, response, or configuration change. Related to #1168; it does not implement keepalives.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions