Skip to content

Feature: Flush response headers immediately for streaming requests #40

Closed

Description

@qcnhy

Problem

When a Chat Completions provider has high time-to-first-token (e.g., GLM-5.2 with large context can take 20-90s before first token), codex-relay does not flush the HTTP response headers until the first SSE event arrives.

This causes client-side idle timeouts ("stream disconnected before completion: idle timeout waiting for SSE") because the client sees no HTTP response at all during the TTFT period.

Current workaround

Proxy scripts must:

  1. Send HTTP 200 + headers to the client immediately (before forwarding to codex-relay)
  2. Inject SSE keepalive comments (": keepalive\n\n") every 5 seconds during TTFT
  3. Stop keepalives once real output begins

This is ~40 lines of keepalive management that would be unnecessary if codex-relay flushed headers early.

Proposed solution

codex-relay could:

  1. Send HTTP 200 + Content-Type: text/event-stream headers immediately when accepting a streaming request
  2. Optionally send SSE keepalive comments while waiting for the upstream's first token

Environment

  • codex-relay 0.5.1
  • GLM-5.2 via Chat Completions (TTFT 5-90s for large context)

Activity

  1. mountain commented on Jul 11, 2026

    @mountain
    Collaborator

    Thanks for the report. The streaming path already yields response.created before opening the upstream request, so the relay can flush 200 + text/event-stream without waiting for the provider's response headers. It also had Axum SSE keepalive enabled, but was using Axum's 15-second default interval.

    I have tightened the keepalive interval to 5 seconds and made the comment explicit as : keepalive. I also added an end-to-end regression test with a mock upstream that waits 7 seconds before returning its HTTP headers. The test verifies that:

    • the relay response headers arrive within 1 second
    • response.created arrives immediately
    • a keepalive comment arrives at 5 seconds, before the upstream headers

    The full test suite passes (168 non-ignored tests; 12 live-provider tests ignored).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions