Skip to content

fix(inference): bound OpenRouter outbound deadline before connection establishment - #7281

Merged
jyaunches merged 10 commits into
mainfrom
fix/7248-openrouter-connect-deadline
Jul 24, 2026
Merged

jyaunches merged 10 commits into
mainfrom
fix/7248-openrouter-connect-deadline

Conversation

@laitingsheng

@laitingsheng laitingsheng commented Jul 21, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The OpenRouter runtime adapter exposed upstreamTimeoutMs as its outbound timeout, but applied it through a connected-socket inactivity timer, so DNS, TCP, and TLS establishment ran unbounded and a pre-connect attempt could outlive the configured timeout. The adapter now starts one total deadline before the request begins: expiry returns a redacted 504 upstream_timeout and destroys the outbound request, while a genuine transport failure still returns a redacted 502 openrouter_runtime_error.

Related Issue

Fixes #7248

Changes

  • src/lib/inference/openrouter-runtime-adapter-forward.ts: replace ClientRequest.setTimeout with a single setTimeout deadline created before the outbound request, cleared once settled. On expiry the adapter fails with 504 upstream_timeout and destroys the request without an error argument.
  • Guard the settle path so an error emitted during destroy cannot run the response side effects twice, and discard a late upstream response after the deadline settles so it cannot call writeHead on an already-sent response.
  • src/lib/inference/openrouter-runtime-adapter.test.ts: drop the fixed-port-1 assumption; add deterministic coverage for immediate connection failure (redacted 502), pre-connect deadline expiry (redacted 504 with request destroy), and late-response discard. Existing post-connect stall and mid-response abort coverage is retained.
  • Adjacent-path review (per CONTRIBUTING.md): open PR feat(inference): route DNS-backed HTTPS endpoints with scoped credentials #7188's HTTPS-pin forwarder uses the same ClientRequest.setTimeout pattern and needs the same bounding; tracked as a follow-up on that PR rather than in this change.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification: internal runtime-adapter timeout behavior with no user-facing surface or documented contract change.
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification:
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

Verification

  • PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
  • Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
  • Targeted behavior tests pass for the current change set, or tests are marked not applicable above — command/result: npx vitest run --project cli src/lib/inference/openrouter-runtime-adapter.test.ts — 8/8 passed; npm run typecheck:cli — passed.
  • Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result:
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: Tinson Lai tinsonl@nvidia.com

Summary by CodeRabbit

  • Bug Fixes

    • Improved upstream timeout/deadline behavior to reliably return a clear 504 when exceeded, including cases where headers stall.
    • Prevented late upstream responses/callbacks from being processed after the request has settled.
    • Improved streaming reliability by delaying response commitment until the first upstream body chunk, with backpressure-aware chunk forwarding.
    • Strengthened error redaction to avoid leaking upstream connection details.
  • Tests

    • Expanded coverage for immediate upstream connection refusal, bounded outbound deadlines, late response handling, and SSE/stream ordering plus header-stall timeouts.

…establishment

The OpenRouter runtime adapter timed the outbound request with a
connected-socket inactivity timer, so DNS, TCP, and TLS establishment
ran unbounded and could outlive upstreamTimeoutMs. Replace it with a
single total deadline started before the request, returning a redacted
504 upstream_timeout on expiry and destroying the outbound request,
while a genuine transport failure stays a redacted 502. Guard the
settle path against a double error side effect and drop a late upstream
response after the deadline. Cover pre-connect expiry and immediate
connection failure without assuming a fixed refused port.

Signed-off-by: Tinson Lai <tinsonl@nvidia.com>
@laitingsheng laitingsheng added area: inference Inference routing, serving, model selection, or outputs bug-fix PR fixes a bug or regression labels Jul 21, 2026
@coderabbitai

coderabbitai Bot commented Jul 21, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

The OpenRouter forwarder now enforces a total outbound deadline, discards late upstream responses, streams response chunks with backpressure handling, and adds coverage for connection refusal, pre-connect timeout, stalled headers, and SSE streaming.

Changes

OpenRouter forwarding deadline and streaming

Layer / File(s) Summary
Deadline and response forwarding lifecycle
src/lib/inference/openrouter-runtime-adapter-forward.ts
forwardOpenRouterRequest coordinates single settlement, enforces a total timeout, destroys late responses, delays downstream headers until data arrives, and streams chunks with backpressure handling.
Connection, timeout, and streaming validation
src/lib/inference/openrouter-runtime-adapter.test.ts
Tests cover deterministic refusal, pre-connect timeout redaction, request and late-response destruction, ordered SSE streaming, and stalled upstream headers.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant AdapterCaller
  participant forwardOpenRouterRequest
  participant ClientRequest
  participant IncomingMessage
  participant DownstreamResponse
  AdapterCaller->>forwardOpenRouterRequest: start OpenRouter request
  forwardOpenRouterRequest->>ClientRequest: create request and start total deadline
  ClientRequest->>IncomingMessage: deliver upstream response
  IncomingMessage->>DownstreamResponse: stream body chunks
  DownstreamResponse-->>IncomingMessage: drain resumes upstream stream
  forwardOpenRouterRequest-->>AdapterCaller: return completed response or 504 upstream_timeout
  forwardOpenRouterRequest->>IncomingMessage: destroy late response
Loading
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed The adapter now enforces a pre-connect deadline, preserves redacted 504/502 behavior, and the tests cover connection failure, stalls, and late responses.
Out of Scope Changes check ✅ Passed The changes stay focused on OpenRouter timeout handling and its tests, with no unrelated code paths introduced.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main change: moving the OpenRouter outbound deadline to start before connection establishment.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/7248-openrouter-connect-deadline

Comment @coderabbitai help to get the list of available commands.

@github-code-quality

github-code-quality Bot commented Jul 21, 2026 •

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit 770612a in the fix/7248-openrouter-... branch remains at 96%, unchanged from commit 8fcd69c in the main branch.

TypeScript / code-coverage/cli

The overall coverage in commit 770612a in the fix/7248-openrouter-... branch remains at 80%, unchanged from commit 8fcd69c in the main branch.

Show a code coverage summary of the most impacted files.
File main 8fcd69c fix/7248-openrouter-... 770612a +/-
src/lib/platform.ts 89% 84% -5%
src/lib/inferen...pter-forward.ts 88% 88% 0%
src/lib/messagi...nnels/policy.ts 100% 100% 0%
src/lib/sandbox...rce-identity.ts 87% 87% 0%
src/lib/securit...ntial-filter.ts 93% 93% 0%
src/lib/trace.ts 94% 94% 0%

Updated July 24, 2026 18:43 UTC

@github-actions

github-actions Bot commented Jul 21, 2026 •

Copy link
Copy Markdown
Contributor

PR Review Advisor — Informational

Advisor assessment: Informational / high confidence
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions
Status: No actionable findings remain in the canonical review ledger.

Model lanes

  • GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
  • Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
  • Model comparison: normalized findings match; normalized E2E selections differ; severity counts match.

Nemotron output stays in workflow artifacts and does not change the assessment above.

Since last review: 0 prior items resolved · 0 still apply · 0 new items found

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: inference-routing, network-policy

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

Remove the guard branch around removing the probe server from the
close-tracking array; the index is always present immediately after
listen() pushes it, so the check only tripped the growth-guardrails
test-linearity gate.

Signed-off-by: Tinson Lai <tinsonl@nvidia.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/lib/inference/openrouter-runtime-adapter.test.ts`:
- Line 195: Guard the removal in the test’s server cleanup flow by storing the
result of servers.indexOf(probe) and only calling splice when the index is
non-negative. Preserve the existing behavior when probe is present and avoid
modifying an unrelated entry when it is absent.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 883813ec-4523-4521-906e-38e96d8d589f

📥 Commits

Reviewing files that changed from the base of the PR and between 0554786 and cf36c72.

📒 Files selected for processing (1)
  • src/lib/inference/openrouter-runtime-adapter.test.ts

Comment thread src/lib/inference/openrouter-runtime-adapter.test.ts Outdated
servers.indexOf(probe) returning -1 fed splice(-1, 1), which would
silently drop an unrelated tracked server if probe were ever absent.
Replace the index-based removal with a filter-and-replace that is a
no-op when probe is not present.

Signed-off-by: Tinson Lai <tinsonl@nvidia.com>

@cjagwani cjagwani left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current revision still has the response-commit race identified by the PR Review Advisor. The total deadline remains armed after the upstream response callback writes headers and pipes the body. If upstream sends 200 headers and then stalls, deadline expiry can only destroy the downstream response because headersSent is already true; clients receive a partial 200/network error instead of the accepted redacted 504 upstream_timeout contract.

Please add the headers-then-stall regression and resolve the contract before approval. Either define the deadline as time-to-headers and clear it before committing the upstream response (with the issue/PR contract updated accordingly), or preserve a total-response deadline with a bounded design that can still emit 504 before downstream headers are committed. The latter changes buffering/streaming behavior and should be treated as a deliberate design choice, not an incidental patch.

Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/lib/inference/openrouter-runtime-adapter.test.ts`:
- Around line 367-394: Update the test around the upstream server in “returns a
redacted timeout when upstream sends headers and then stalls (`#7248`)” to signal
immediately after flushHeaders(), await that signal before issuing or awaiting
the client response, and use a less aggressive upstreamTimeoutMs. Preserve the
existing 504 status and upstream_timeout assertions while ensuring the test
cannot pass through the pre-connect timeout path.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: ec209567-89ff-4fb6-ac5b-4ec766e23915

📥 Commits

Reviewing files that changed from the base of the PR and between 6350359 and 361380b.

📒 Files selected for processing (2)
  • src/lib/inference/openrouter-runtime-adapter-forward.ts
  • src/lib/inference/openrouter-runtime-adapter.test.ts

Comment thread src/lib/inference/openrouter-runtime-adapter.test.ts
Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>
@jyaunches
jyaunches dismissed cjagwani’s stale review July 24, 2026 14:37

The requested headers-then-stall regression and redacted 504 timeout contract are implemented and verified at current head 45a4ecc. Dismissing the stale changes-requested state so required CI can be rerun.

@jyaunches

Copy link
Copy Markdown
Contributor

Maintainer note: the trusted E2E rerun completed successfully: https://github.com/NVIDIA/NemoClaw/actions/runs/30094920704. The PR Gate still reports Evidence is incomplete because each network-policy matrix shard selects one of two tests, while the risk-signal reporter counts the selector-excluded sibling as skipped; the gate then rejects any nonzero skipped count. This is a CI evidence-accounting false negative, not a failing PR test. PR #7281 is blocked pending a dedicated CI workflow/tooling fix that makes evidence counting selector-aware while continuing to reject genuine runtime skips and selectors that match no tests.

@jyaunches

Copy link
Copy Markdown
Contributor

Follow-up: the dedicated selector-aware E2E evidence fix is now open as #7488. PR #7281 remains blocked on that CI fix and its subsequent gate rerun.

@jyaunches
jyaunches requested a review from cjagwani July 24, 2026 17:42
@jyaunches

Copy link
Copy Markdown
Contributor

Follow-up: #7488 is merged, this branch is updated to current main, and the fresh trusted E2E run passed for the current head/base: https://github.com/NVIDIA/NemoClaw/actions/runs/30118065100. The selector-aware gate accepted the evidence, including inference-routing, network-policy zero-presets, and network-policy live-probes; the PR Gate is green. All required automated checks are now green. The remaining merge blocker is the requested human approval from @cjagwani.

@jyaunches
jyaunches merged commit 4ecd8e5 into main Jul 24, 2026
82 of 83 checks passed
@jyaunches
jyaunches deleted the fix/7248-openrouter-connect-deadline branch July 24, 2026 19:50
@senthilr-nv senthilr-nv mentioned this pull request Jul 25, 2026
10 of 23 tasks
senthilr-nv added a commit that referenced this pull request Jul 25, 2026
<!-- markdownlint-disable MD041 -->
## Summary

Adds the canonical pre-tag `## v0.0.95` release entry to
`docs/changelog/2026-07-24.mdx`, before the existing v0.0.94 entry. The
entry summarizes approved user-visible changes merged since v0.0.94 and
excludes internal-only prerequisites.

## Changes

- Adds the v0.0.95 summary and detailed bullets for gateway lifecycle,
recovery, state transfer, inference compatibility, sandbox security,
Discord policy, and E2E evidence.
- Links each user-facing theme to the most specific published
documentation.
- Records the release entry in the shared native changelog used by the
OpenClaw, Hermes, and Deep Agents guides.

Source summary:

- [#7246](#7246),
[#7228](#7228),
[#7267](#7267),
[#7489](#7489),
[#7509](#7509),
[#7351](#7351), and
[#7290](#7290) ->
`docs/changelog/2026-07-24.mdx`: Gateway authority, forward teardown and
retry, managed recovery, Hermes restart recovery, scoped uninstall, and
orphan-aware backup behavior.
- [#7344](#7344) and
[#7416](#7416) ->
`docs/changelog/2026-07-24.mdx`: Atomic SQLite restore and host download
verification.
- [#7476](#7476),
[#7347](#7347),
[#7281](#7281),
[#7485](#7485),
[#7491](#7491), and
[#7422](#7422) ->
`docs/changelog/2026-07-24.mdx`: Windows Ollama reuse, CDI fallback,
bounded OpenRouter connection setup, Nemotron-3 request compatibility,
and managed Deep Agents retry and provider-error behavior.
- [#6884](#6884),
[#7481](#7481),
[#6878](#6878),
[#7467](#7467),
[#7502](#7502),
[#7503](#7503),
[#7504](#7504), and
[#7486](#7486) ->
`docs/changelog/2026-07-24.mdx`: Trusted base-image overrides, local
rebuild images, runtime validation, config preservation, reviewed
package updates, and fewer final-image payload layers.
- [#7303](#7303) ->
`docs/changelog/2026-07-24.mdx`: Scoped Discord application-command
management.
- [#7488](#7488),
[#7465](#7465),
[#7497](#7497),
[#7464](#7464),
[#7501](#7501),
[#7494](#7494), and
[#7493](#7493) ->
`docs/changelog/2026-07-24.mdx`: Selected-test risk signals, retry
cleanup, full root-image validation, direct-main Hermes setup, executed
PR-gate evidence, nightly history, and runner wait reporting.
- [#7447](#7447) is an internal
pinned-runtime prerequisite and is intentionally excluded from canonical
supported-integration documentation.
- [#7370](#7370) adds
maintainer-only advisory reconciliation tooling and does not change
supported user behavior.
- [#7495](#7495) updates existing
documentation and does not add a new v0.0.95 behavior claim.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [x] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [ ] Tests added or updated for changed behavior
- [x] Existing tests cover changed behavior — justification:
`test/changelog-docs.test.ts` validates the dated changelog structure,
heading uniqueness, and published links.
- [ ] Tests not applicable — justification:
- [x] Docs updated for user-facing behavior changes
- [ ] Docs not applicable — justification:
- [ ] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [ ] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification:
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Documentation Writer Review

- [x] Documentation writer subagent reviewed the completed changes
- Result: `docs-updated`
- Evidence: `docs/changelog/2026-07-24.mdx`; writing rules,
documentation style, factual release meaning, and published links
reviewed at exact head `58b02f2bf`.
- Agent: Codex documentation writer reviewer
<!-- docs-review-head-sha: 58b02f2 -->
<!-- docs-review-agents-blob-sha: 9c9b36d -->

## DGX Station Hardware Evidence

- [ ] Tested on DGX Station
- Tested commit:
- Station profile/scenario:
- Result:
- Supporting evidence:

## Verification

- [x] PR description includes a `Signed-off-by:` line and every commit
appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — command/result or justification: `npx
vitest run test/changelog-docs.test.ts` passed 6 tests.
- [ ] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — command/result:
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only) — the
build passed with 0 errors and 2 Fern warnings.
- [x] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---
Signed-off-by: Senthil Ravichandran <senthilr@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Documentation**
  * Added a new v0.0.95 changelog entry above v0.0.94.
* Documented improved externally supervised gateway lifecycle ownership.
  * Improved snapshot restore reliability and SQLite state handling.
  * Tightened CLI `backup-all` behavior and host artifact verification.
* Updated Windows onboarding guidance (including Ollama service reuse
and CDI directory fallback).
* Noted inference compatibility fixes, deeper agent failure
classification, stricter base-image validation, updated Discord bot
command permissions, and refined E2E release automation evidence
handling.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Senthil Ravichandran <senthilr@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: inference Inference routing, serving, model selection, or outputs bug-fix PR fixes a bug or regression

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[WSL2][Inference] OpenRouter adapter timeout excludes TCP connection establishment

6 participants