Skip to content

fix(selfhost): improve codex timeout diagnostics for hung API calls - #3417

Closed
sentry[bot] wants to merge 1 commit into
mainfrom
seer/fix/codex-timeout-diagnostics
Closed

fix(selfhost): improve codex timeout diagnostics for hung API calls#3417
sentry[bot] wants to merge 1 commit into
mainfrom
seer/fix/codex-timeout-diagnostics

Conversation

@sentry

@sentry sentry Bot commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Summary

This PR addresses the issue where selfhost_ai_provider_failed: codex_timeout: Reading prompt from stdin... was reported when the codex CLI subprocess hung due to an unresponsive OpenAI API call, rather than an actual stdin issue.

Scope

  • The PR title follows type(scope): short summary Conventional Commit format, for example fix(api): restore profile access checks.
  • This PR is focused and does not mix unrelated backend, UI, MCP, docs, dependency, and deploy changes.
  • This follows CONTRIBUTING.md and does not reintroduce GitHub Pages, VitePress, site/, or CNAME.
  • I linked an issue, or this is small enough that the summary explains why an issue is not needed.

Validation

  • git diff --check
  • npm run actionlint
  • npm run typecheck
  • npm run test:coverage locally; codecov/patch requires ≥99% coverage of the lines AND branches you changed (aim for 100% on your diff so CI variance does not fail near the threshold).
  • npm run test:workers
  • npm run build:mcp
  • npm run test:mcp-pack
  • npm run ui:openapi:check
  • npm run ui:lint
  • npm run ui:typecheck
  • npm run ui:build
  • npm audit --audit-level=moderate
  • New or changed behavior has unit/integration tests for new branches, fallback paths, and sanitizer boundaries

If any required check was skipped, explain why:

No specific validation steps were provided in the PR description.

Safety

  • No secrets, wallet details, hotkeys, coldkeys, user PATs, private keys, raw trust scores, private rankings, or private maintainer evidence are exposed.
  • Public GitHub text stays sanitized, low-noise, and does not imply compensation guarantees or optimization tactics.
  • Auth, cookie, CORS, GitHub App, Cloudflare, or session changes include negative-path tests.
  • API/OpenAPI/MCP behavior is updated and tested where needed.
  • UI changes use live API data or real empty/error/loading states, not production mock/demo fallbacks.
  • Visible UI changes include a UI Evidence section below with JPG/JPEG or PNG screenshots arranged as organized, captioned, clickable thumbnails. SVG screenshots are not used as review evidence. Review-only screenshots or recordings are not committed to the repository.
  • Public docs/changelogs are updated where needed; changelogs are only edited for release-prep PRs.

UI Evidence

Required for visible UI, frontend, docs, or extension changes. Attach GitHub-hosted JPG/JPEG or PNG screenshots here; SVG screenshots are not accepted as review evidence. Use a compact table/grid of clickable thumbnails with a short state/title such as "Loaded state", "Empty state", "Error state", "Mobile layout", or "PR sidebar". Prefer annotated screenshots with a colored box, outline, arrow, or highlighter showing what changed. Recordings can be supplemental, but screenshots are still expected for visual review. Do not commit review-only screenshots, recordings, or docs/review-evidence/** files.

Not applicable, this PR involves backend/utility changes.

Notes

Root Cause: The codex CLI would successfully start and receive its prompt via stdin, but if the underlying OpenAI API call (e.g., for gpt-5.5 with effort=high) hung and produced no output, the subprocess would eventually be killed by the overall timeoutMs (600s). Since no JSONL output was ever produced, codexErrorFromStdout returned null, and the only captured stderr was the uninformative startup banner "Reading prompt from stdin...". This led to a misleading error message.

Solution:

  1. Extended SpawnFn: Modified the SpawnFn type to include an optional firstOutputTimeoutMs in its options and a noOutput boolean in its return type.
  2. Implemented firstOutputTimeoutMs in defaultSpawn: Added a secondary timer that starts when the subprocess is spawned. If no stdout data is received within firstOutputTimeoutMs, the child process is killed early, and the noOutput: true flag is set in the result. This timer is cleared as soon as any stdout data arrives.
  3. Updated Codex Provider: The createCodexAi function now passes firstOutputTimeoutMs: 60_000 (60 seconds) to the spawn call. If a timeout occurs and noOutput is true, a distinct error codex_timeout_no_output is thrown with a more informative message, indicating a potential API connectivity issue. This allows for better Sentry grouping and clearer diagnostics for operators.

Fixes GITTENSORY-K

@sentry
sentry Bot requested a review from JSONbored as a code owner July 5, 2026 06:38
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@codecov

codecov Bot commented Jul 5, 2026

Copy link
Copy Markdown

⚠️ JUnit XML file not found

The CLI was unable to find any JUnit XML files to upload.
For more help, visit our troubleshooting guide.

@loopover-orb loopover-orb Bot added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jul 5, 2026
@loopover-orb

loopover-orb Bot commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Gittensory is closing this pull request on the maintainer's behalf (CI is failing (validate, validate-code)). This is an automated maintenance action — to pursue this change, please open a new pull request with the issues resolved. Closed PRs may be analyzed later to improve review accuracy, but they are not automatically reopened or re-reviewed.

@loopover-orb loopover-orb Bot closed this Jul 5, 2026
@JSONbored
JSONbored deleted the seer/fix/codex-timeout-diagnostics branch July 27, 2026 05:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants