91 second cuttoff - kills slower systems #1168

Activity

1. modtim

   added

   bugSomething isn't workingSomething isn't working

   on Oct 28, 2026
2. ting-hong-shieh commented on Oct 28, 2026

   

   ting-hong-shieh

   on Oct 28, 2026

   Contributor

   More actions

   Could you share a few details so I can reproduce the cutoff?

   • Your Switchyard version or commit, how it was installed, and the command used to start it.
   • The relevant llm_clients, targets and route configuration, with credentials and private endpoint details removed. In particular, is timeout_ms set anywhere?
   • Which caller and model server you use, whether the request is streaming, and whether it goes through another proxy or gateway. Does the cutoff happen while new tokens are still arriving, or while waiting for the first token / after a pause? Is the 91 seconds measured from the start of the request?
   • The exact client error or HTTP status and the corresponding Switchyard error, with credentials and request content removed. Does the same request complete when sent directly to the model server?

   Current main already supports a per-client timeout_ms, which is unbounded when omitted. I’ll check delayed responses and ongoing streams against a local mock upstream to see whether this reproduces without a configured deadline.
3. modtim commented on Oct 29, 2026

   

   modtim

   on Oct 29, 2026

   Author

   More actions

   ▎ Correction from our side: we re-tested on the routes that were failing, under real load. Switchyard held a non-streaming
   ▎ request for 461s and streamed for over 520s without cutting off. The disconnects (HTTP 499) came from our client timing out
   ▎ at ~300s, not from Switchyard. Our earlier 91s report was wrong — sorry for the noise and thanks for jumping on it. One
   ▎ optional feature question: on slow backends the first upstream byte can take 5+ minutes, and Switchyard sends nothing until
   ▎ then. Could it send response headers plus an Anthropic event: ping keepalive before the first upstream chunk? That would
   ▎ stop idle clients dropping on long prefills.
4. ting-hong-shieh commented on Oct 29, 2026

   

   ting-hong-shieh

   on Oct 29, 2026

   Contributor

   More actions

   Thanks for re-testing and clarifying the original report. My local probes at b0fa4fb2 also completed 105-second delayed responses and streams with timeout_ms omitted.

   For the keepalive request, could you share:

   • The client/SDK and version, plus its timeout settings with credentials removed. Is the ~300-second limit an idle/read timeout that resets when data arrives, or a deadline for the whole request? Pings may help with the former; they would not extend the latter.
   • Whether this is a stream: true request to /v1/messages, and whether another proxy or gateway sits between the client and Switchyard.

   For maintainers: would sending early headers and Anthropic pings while waiting for the upstream be an acceptable direction? Returning HTTP 200 before the upstream status is known means a later upstream failure would need to be reported through the stream rather than its original HTTP status. I'd like to confirm that behavior before preparing a patch.
5. modtim commented on Oct 29, 2026

   

   modtim

   on Oct 29, 2026

   Author

   More actions

   Thanks. Answers in order:

   1. Client, version, timeouts

   • The client is the Claude Code CLI 2.1.310, using its bundled Anthropic SDK. ANTHROPIC_BASE_URL points straight at
     Switchyard.
   • All the data we sent was collected with API_TIMEOUT_MS unset, so the client's default timeout applied.
   • From the counts:
     • stream: false: the ~300s limit acts as a whole-request deadline. These cuts all land at 300.1–300.4s, whether or not
       anything was happening. Pings wouldn't help with this one. We've since set API_TIMEOUT_MS=1800000, and those cuts are
       going away.
     • stream: true: we see no fixed cutoff. Most drops come well before 290s, and a cluster falls at 310–400s, which matches
       slow prefill on loaded backends (first upstream byte up to 414s). That pattern fits an idle/read timeout, but we can't
       prove it. Switchyard's 499 log lines have no time-to-first-byte field, so we can't separate idle drops from normal client
       cancels. That field would help.

   2. Path

   • Both kinds of request go to POST /v1/messages, and the keepalive question is about the stream: true ones.
   • There's no other proxy or gateway in the path. The client connects to a Docker port published straight into the Switchyard
     container.

   3. Early headers + pings

   • Yes, we're fine with that direction. We'd like:
     • opt-in (per route or by request header, default off);
     • stream: true only.
   • If the upstream fails after the 200 and headers are sent, report it in-stream as an Anthropic event: error
     (overloaded_error / api_error, keeping the upstream status in the message if you can), then close.
   • Non-streaming behaviour stays as it is today.