Skip to content

fix(logs): keep the Codex WebSocket error wrapper's 5xx status and record the terminal frame kind - #6974

Merged
lidge-jun merged 1 commit into
devfrom
codex/6740-ws-error-wrapper-status
Oct 12, 2026
Merged

lidge-jun merged 1 commit into
devfrom
codex/6740-ws-error-wrapper-status

Conversation

@lidge-jun

@lidge-jun lidge-jun commented Oct 12, 2026 •

Copy link
Copy Markdown
Owner

Summary

On the canonical Codex upstream WebSocket, the backend can end a turn with an error frame shaped like {"type":"error","status":503,"error":{...}}, sent after response.created. The native Codex client uses that status as the HTTP error status (codex-rs/codex-api/src/endpoint/responses_websocket.rs, map_wrapped_websocket_error_event). Unless the error code was a recognized overload or rate-limit class (#6889), OpenCodex dropped the status and logged a synthetic 502 upstream_server_error with failureCause=upstream-fault. That label hides whether the upstream refused the turn (503) or failed it (500). It is the row that #6740 has been collecting for days, and the reporter could not tell whether those 502s were the same admission refusal they saw as an SSE 503 server_is_overloaded.

  • Wrapper status in the log row. A bare error frame with an unrecognized code now records the 5xx its wrapper states: a top-level integer status or status_code, one of the two and not both, which is the same rule as the pre-commit rejection projection. A wrapped 503 becomes 503 / server_is_overloaded / upstream-declined, and a wrapped 500 becomes 500 / upstream-fault. The status stays provisional, so a genuine terminal still replaces it. A wrapper 4xx is not adopted, because 401, 403, 402 and 429 would move credential and quota routing. Every 5xx is already the same transient class in classifyCodexUpstreamOutcome, so account health, the fix(codex-pool): stabilize account health and reuse WS per thread under concurrency #6860 failure window and rotation do not change.
  • Terminal kind in codexWsStage. CodexWsStageRecord gains terminalEvent, which records the normalized response.completed / response.failed / response.incomplete / error frame that ended a plain exchange before any synthetic tail. It also gains terminalErrorStatus, the wrapper status of a terminal error frame. Both fields are content-free and closed. They are null on native-control exchanges and on legacy rows, and malformed persisted values read as null without dropping the rest of the stage. This is the "terminal event type as observed before any synthetic tail" that the maintainer triage on [Bug] Intermittent pooled Codex upstream response.failed 502 surge after v2.80 install (Windows; transport/egress isolated) #6740 asked reporters to collect.
  • Unchanged behavior. The client-facing terminal, the HTTP head, and retry and replay behavior are unchanged; unknown codes still reach the client as upstream_server_error. Buffered JSON callers keep their own presentation status, as documented.

Docs: structure/dashboard-and-usage.md, structure/transports/responses-wire-shapes.md, docs-site/.../reference/management-api.md. The plan and audit are in devlog/_plan/261012_6740_ws_error_status/.

This makes the #6740 failures classifiable; it does not fix them. Source tracing points at an upstream error frame that OpenCodex relabels as 502, not a proxy transport failure. The upstream root cause still needs Windows rows from real accounts, and I did not reproduce it here.

Refs #6740

Verification

  • bun test tests/responses/sse-failed-tail.test.ts tests/responses/ws-failure-stage.test.ts tests/usage/usage-log-ws-stage.test.ts tests/usage/upstream-failure-diagnostics.test.ts tests/responses/ws-upstream.test.ts tests/routing/combo-stream-preflight.test.ts tests/responses/ws-ambiguous-resend.test.ts: 413 pass, 1 skip, 0 fail.
  • The new regressions fail without the change. With src reverted, 8 new cases fail. The queue-overflow capture test fails with the earlier capture position, and the native-control test fails without the !nativeControl guard.
  • bun run typecheck, bun run structure:check, bun run privacy:scan and bun scripts/file-size-ratchet.ts all pass.
  • bun run test:changed (macOS, Bun 1.4.0; it selected 1,672 files because usage/log.ts and codex-ws-wire.ts are widely imported): 36,512 pass, 107 skip, 37 fail and 3 unhandled errors, all in unrelated files: service install state, Lab live probe, live audio, native Codex/Grok toggles, restart replacement. Those 11 files pass on their own, 430/430, both with this change and on the unchanged base, so the failures depend on load from the concurrent full run. Exact-head CI covers the rest.
  • No Windows runner or real-account traffic was used. Whether the backend's frames carry status, and which value, remains to be observed.

Checklist

  • Scope stays focused and avoids unrelated cleanup.
  • Docs or release notes were updated when needed.
  • Security-sensitive changes were reviewed for secrets, auth, and unsafe defaults.

Summary by CodeRabbit

  • Bug Fixes
    • Streaming Codex errors with an unrecognized error code now record an explicit wrapper server status (500–599) when exactly one valid status is provided. If no eligible status is present, the recorded status remains 502.
    • Recorded WebSocket stage details now include the terminal event and applicable error status. Missing or invalid values are stored as null without dropping the stage record.
    • Client-facing error classification and retry behavior remain unchanged.
  • Documentation
    • Updated references to explain how streaming and buffered requests record statuses for unknown upstream errors.

…cord the terminal frame kind

A bare upstream error frame with an unrecognized code reached request logs as a
synthetic 502 upstream_server_error even when its WebSocket wrapper stated the
status, which the native client maps directly. Adopt the wrapper's top-level
status or status_code (one of the two) provisionally when it is 5xx; 4xx stays
excluded because it would move credential and quota routing, and every 5xx is
the same transient health class. Record terminalEvent and terminalErrorStatus on
the content-free Codex WebSocket stage record so a row shows which terminal frame
the upstream actually sent before any synthetic tail. Client terminals, retry
and replay behavior are unchanged.

Refs #6740
@lidge-jun
lidge-jun requested a review from Ingwannu as a code owner October 12, 2026 07:14
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 12, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-12T07:18:26.870281Z 6fc5e22 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@coderabbitai

coderabbitai Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Repository: lidge-jun/opencodex/.coderabbit.yaml
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: 1ca99000-006e-41b5-9dac-202f8c5749cc

📥 Commits

Reviewing files that changed from the base of the PR and between 8eb1fc3 and 6fc5e22.


📒 Files selected for processing (12)
  • devlog/_plan/261012_6740_ws_error_status/010_plan.md
  • docs-site/src/content/docs/reference/management-api.md
  • src/server/request-log-terminal-status.ts
  • src/server/responses/codex-ws-exchange.ts
  • src/server/responses/codex-ws-wire.ts
  • src/usage/log.ts
  • structure/dashboard-and-usage.md
  • structure/transports/responses-wire-shapes.md
  • tests/responses/sse-failed-tail.test.ts
  • tests/responses/ws-failure-stage.test.ts
  • tests/usage/upstream-failure-diagnostics.test.ts
  • tests/usage/usage-log-ws-stage.test.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 7 remain after this review.



📝 Walkthrough

Walkthrough

Streaming bare WebSocket errors can use an eligible wrapper 5xx as the recorded status when no recognized structured error determines it. WebSocket stage records now include terminal event and error status fields, which normalize to null when absent or invalid.

Changes

Codex WebSocket diagnostics

Layer / File(s) Summary
Wrapper status selection
src/server/responses/codex-ws-wire.ts, src/server/request-log-terminal-status.ts, tests/usage/upstream-failure-diagnostics.test.ts, docs-site/src/content/docs/reference/management-api.md, structure/dashboard-and-usage.md, structure/transports/responses-wire-shapes.md, devlog/_plan/261012_6740_ws_error_status/010_plan.md
For bare error events, a single top-level integer status or status_code can supply status when it is 500–599 and no recognized structured error takes precedence. Tests cover exclusions and status classification. Documentation and the plan describe the streaming and buffered behavior.
Terminal metadata capture and normalization
src/server/responses/codex-ws-wire.ts, src/server/responses/codex-ws-exchange.ts, src/usage/log.ts, tests/responses/sse-failed-tail.test.ts, tests/responses/ws-failure-stage.test.ts, tests/usage/usage-log-ws-stage.test.ts, structure/transports/responses-wire-shapes.md
Stage records capture the terminal event and wrapper status for terminal error frames. Log normalization sets missing or invalid fields to null without dropping the stage. Tests cover terminal types, relay queue overflow, native-control exchanges, and legacy or malformed records.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix


Merge Risk: ⚪ Minimal · up to 6fc5e

No actionable issue remains in the supplied evidence; the change is mergeable after normal checks.

Pre-merge checks | Passed 4 | Failed 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 30.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 10 functions across 8 files. (4 skipped: … Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly and concisely describes both primary changes: preserving the Codex WebSocket error wrapper's 5xx status and recording the terminal frame kind.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

Docstring coverage is 30.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 10 functions across 8 files. (4 skipped: 4 unsupported.)


  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Commit to this branch
  • Create a new PR


🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown
Contributor

✅ Deterministic PR hygiene checks passed.

@Ingwannu Ingwannu left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Source/contract review at 6fc5e22f90a7fd63b4fadabb001147ad38012c44 (PR base 8eb1fc3de365b019f9bacdcaacc6b7d284bd8d48; trusted current dev a0191a1b6f).

The change keeps the important separation: an unrecognized bare error can preserve an exclusive integer wrapper status/status_code only as a streaming log 5xx; known error classes keep precedence, a genuine terminal supersedes provisional status, and the client failed-tail/replay behavior is unchanged. Terminal-kind observation precedes relay-queue rejection, native-control records remain unobserved, and old/malformed persisted terminal fields normalize to null without dropping the rest of the stage. The added fixtures target those distinctions, not just the normal success path.

No concrete blocker found in these reviewed paths. I did not run the fixtures locally in this pass. Existing exact-head Cross-platform CI 38216613266 and target run 38216613252 succeeded; a later cancelled metadata run is not an additional successful check. No live Windows/backend capture, scan, replay or approval was performed.

Please continue to describe this as diagnostic fidelity, not resolution of #6740 or automatic recovery for #6969. The stage record is evidence for the next investigation, never a resend grant; the mapped docs correctly retain that restriction.

@lidge-jun
lidge-jun merged commit e4af84a into dev Oct 12, 2026
60 of 63 checks passed
@lidge-jun

Copy link
Copy Markdown
Owner Author

Maintainer integration into dev (actor @lidge-jun, admin); head 6fc5e22f90 bound with --match-head-commit, no unresolved threads. Exact-head CI 26 pass / 8 skipped. Independent review GPT-6.1-Sol high (different family from the Claude builder): PASS, no findings. On dev 49ae1f7 merged with this head: typecheck, structure:check, privacy:scan, ratchet and diff --check 0; seven WS/usage/combo test files 413 pass / 1 skip / 0 fail. Privacy: only a closed terminal-event value and a bounded integer status are captured. #6740 stays open for real-account Windows rows.

@lidge-jun
lidge-jun deleted the codex/6740-ws-error-wrapper-status branch October 12, 2026 11:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants