[ananthsub]

What changed and why

uvicorn's workers share one listening socket, and each request goes to whichever worker accepts the connection. Session state lives in the memory of the worker that created the session, and nothing keeps a session's later requests on that worker.

Measured with the stateful counter resources server (example_session_state_mgmt) and 200 concurrent episodes through /run:

Workers	Reward 1.0	Reward 0.0
1	200	0
4, before this change	7 to 27	173 to 193
4, after this change	200	0

Servers on main with this shape:

• swe_rebench runs 8 workers. It stores each session's sandbox in an in-memory dict at /seed_session and pops it at /verify. A /verify on another worker finds nothing, the patch is recorded as empty, and the reward is 0.
• Agents with 4 workers are affected wherever they keep per-session state across requests.
• CLI harnesses in sandboxes reach a resources server's tools over /mcp. They send the signed MCP session token minted at /seed_session, and no Gym session cookie. With 4 workers and 200 concurrent counter sequences over MCP, 65 of 200 ended with the right count.

One rule

With num_workers > 1, resources servers and agent servers route every request the same way:

1. Find the request's session ID.
2. Look up the session's claim, which names the worker that created the session.
3. Forward the request to that worker's private Unix socket, unchanged, and stream its reply back unchanged. If this worker holds the claim, or the session has no claim, the request is handled here.

A request that arrives over a private socket is always handled where it arrives, so nothing is forwarded twice. If the claimant's socket is gone, because its worker exited and uvicorn replaced it, the request gets a 410 instead of running silently against empty state.

The cookie, the MCP session token and the body only supply the session ID:

Request	Where its session ID comes from
Any request to a resources server with the session cookie	The cookie's session_id (SESSION_ID_KEY)
A resources tool call over /mcp, or a request with an MCP session token and no cookie	The token's sid
A typed resources /seed_session or /close_session	The body's resources_session_id
Any request to an agent server with the session cookie	The cookie's agent_session_id
An agent session seed or close (/v1/agent_sessions, /v1/agent_sessions/close)	The body's agent_session_id

Seed and close routes are matched bare or under a rollout prefix. Some details of the rule:

• One ID per session. A typed resources seed now gives its cookie its resources_session_id as the session_id, so the seed, its close and every request in between name the session by one ID. Handlers that read request.session[SESSION_ID_KEY] keep working, and see the same ID the body carries. An agent session already had one ID.
• Two IDs are refused. If a seed's or close's cookie names a different session than its body, the reply is 409, as the agent seed already answered. This also holds with one worker, where the same step gives a typed seed its ID in the cookie.
• New sessions. A resources request with no session ID gets a new ID from the router, which the session middleware uses. A session start (/seed_session, or another route in checkpoint_session_starts) claims it before the handler runs, whether or not the handler reads the session, because its reply's cookie names the session. This covers an agent's own /run seeding with its untyped body, which the default seed_session answers without reading the session. Any other request claims its new ID only if the handler read or wrote the session, before the reply leaves the worker, so the client's next request through any worker finds the claim; this covers a server that creates a session implicitly on its first tool call. A request whose handler never touches the session, such as a health check, /aggregate_metrics or a checkpoint control route, leaves no claim and gets no session cookie, so no cookie ever names a session nobody claims. If another worker claimed the same random ID meanwhile, the request gets 409 rather than being served twice.
• Cookies whose claim ended. A request whose cookie names a session nobody claims, because its claim ended after the window or it came from before a restart, is claimed by the worker that serves it if the handler uses the session, so the session's later requests reach that worker.
• No owner in cookies or tokens. Cookies and MCP session tokens no longer carry the creating worker's ID. A cookie or token minted by an earlier version of this change is read for its session ID only.

Claims

A claim is a file in the server's runtime directory, /tmp/ng-<pid>-<random>/session-claims/, named by a hash of the session ID and holding the claimant's worker ID. The worker that creates a session creates its claim with a hard link from a complete temporary file, which fails if the claim exists, so two workers can never both claim an ID and a reader never sees a partial claim.

Claims are created on every path that creates a session:

• a typed resources seed, with or without a cookie;
• an agent session seed;
• a session start, whatever its handler does, and any other new resources session whose handler used the session, as above;
• a request whose cookie names a session nobody claims, if its handler used the session;
• a close of an ID nobody has claimed, so a delayed seed reaches the worker that recorded the close and is refused, as on one worker.

Every claim ends:

Session	Claim ends
Agent session	When the agent drops the session's last record: its close receipt, or a failed seed nobody closed, session_close_retry_window_seconds after it was written
Resources session, open	Not while it is open: from a successful /seed_session (or another route in checkpoint_session_starts, such as a Gymnasium /reset) until its /close_session, its /verify, or checkpoint_session_ended. A /verify that says a close follows (x-ng-checkpoint-close-follows, which single_agent_turn sends) leaves it open for that close
Resources session, not open	session_close_retry_window_seconds after the last request for it: after its close or verify, after a failed seed, or for a session that never had a seed
Session of a worker that exited	When the server exits, with the whole runtime directory; until then its requests get 410

• session_close_retry_window_seconds is a new resources server setting, default 300 seconds, as agents already had. It covers the caller's retried and cleanup closes, which reach the claimant within the window.
• A claim is never released while a request for its session is being served, and only its claimant removes it. Each worker sweeps its own expired claims in the background, so a claim ends even when no more requests arrive.
• Rollback. An agent seed that creates nothing releases its claim when the seed returns. An agent seed whose state was set up but cannot be registered, for example because the checkpoint participant refuses a retired attempt, now releases that state with the close hook and drops its claim at once; if the release fails, the state is kept as a failed seed for the caller's close. A failed resources seed keeps its claim for the window, because the router cannot tell whether the server kept partial state, and the caller's cleanup close must reach it.
• ClosedResourcesSessionRecords in nemo_gym.base_resources_server keeps a closed resources session for the same window, counted from its first close, so the record never outlives the claim. While the claim lasts, a late seed reaches the worker that remembers the close and is refused; after it ends, every worker treats the ID as new. NOOA keeps its closed sessions in it, and its tool routes refuse a cookie whose counter is gone.
• An agent drops a failed seed that nobody closed once the window has passed, running its close hook in a background task with a fresh context, under the session's own rollout. If the hook raises, the record keeps its state for a later close and the release is tried again a window later.
• With one worker there are no private sockets and no claims.

sequenceDiagram
    participant C as Caller
    participant B as Worker B
    participant A as "Worker A (claimant)"
    C->>B: POST /verify with session cookie
    B->>B: read session_id from the cookie
    B->>B: read the session's claim: worker A
    B->>A: same request over A's private socket
    A->>A: SessionMiddleware and handler with A's state
    A-->>B: reply with A's Set-Cookie
    B-->>C: reply streamed back unchanged

Loading

sequenceDiagram
    participant E as "Environment server"
    participant B as Worker B
    participant A as Worker A
    E->>A: POST /seed_session, resources_session_id s, no cookie
    A->>A: claim s for A, seed it
    A-->>E: 500, no Set-Cookie
    E->>B: POST /close_session, resources_session_id s, no cookie
    B->>B: read s from the body, claim of s: worker A
    B->>A: same request over A's private socket
    A->>A: close releases the partial state
    A-->>E: 200, through B

Loading

Also in this change

• Large seeds. Seeds and closes of any size are routed: the router reads the whole body, as the app does anyway, and passes it on without another copy.
• Agent sessions on several workers. On main, an agent server refuses to seed an environment server session when num_workers > 1. That guard is removed, and Codex and NOOA drop their own copies of it, so the harnesses (Codex, Hermes, NOOA, OpenClaw, OpenCode and Pi) all seed sessions on several workers, like Simple Agent. Each harness's tests check that a seed with two workers opens its session. The harness READMEs and the Hermes SWE-bench Pro guide no longer ask for one worker.
• Logging. The private socket's uvicorn server leaves uvicorn's process-wide loggers as the worker configured them and writes no access lines of its own, so uvicorn_logging_show_200_ok logs each forwarded request once, on the worker that received it.
• CI. A session-routing-e2e job in unit-tests.yml runs NEMO_GYM_MULTIWORKER_E2E=1 pytest tests/e2e/session_routing when the routing code, or the server code it wires into, changes.
• Docs. The resources server guide gains a section on state with several workers, with the rule, the table of session ID sources, and a claim's lifetime.
• example_session_state_mgmt gains the module-level app that multi-worker uvicorn imports.

How it fits together

• The router sits outside Starlette's SessionMiddleware and reads the cookie by decoding it with the same secret. A forwarded request never reaches the receiving worker's SessionMiddleware, so the reply carries only the claimant's Set-Cookie.
• Routing is on for SimpleResourcesServer and SimpleResponsesAPIAgent through routes_sessions_to_owner, which a server that keeps no per-session state can set to False. Model servers keep no session state and are not routed.
• A claimant knows its own claims in memory, so a request for one of its sessions reads no file. Another worker reads the claim file, which costs about 10 microseconds against a forwarding hop of about 1 millisecond, so claims are not cached.
• The forwarding client is aiohttp with one UnixConnector per claimant socket; a claimant whose socket refuses connections has its client discarded.
• The private server runs on the worker's own event loop, after the app's startup, without running the app's lifespan again or adding its own date and server headers.
• The checkpoint stack (feat(checkpoint): resources, agent, and environment servers with several workers #3909) will turn its restore of sessions into claim rewrites, and route an agent's own /run calls that carry no session ID by the worker their header names, which the router handles before it falls back to handling the request locally.

Related work

This fix targets main and does not depend on the rest of the partial-rollout checkpointing stack (#3886 onward; #3882 to #3885 have merged). #3909 builds on it to support environment, agent, and resources servers with several workers. The adapter PR #4790, stacked on this one, moves more resources servers onto ClosedResourcesSessionRecords, whose name and interface are unchanged.

Issue

None known. The bug was found while measuring multi-worker behavior for partial-rollout checkpointing. This is the fix for main, separate from that work.

Validation

Run on this branch, on the current main:

• pytest tests/unit_tests/test_session_routing.py: passed. In-process workers share a real socket directory. The tests check, table-driven where it helps:
  • each source of the session ID (resources cookie, MCP token, typed seed and close bodies, agent cookie, agent seed and close bodies) routes to the worker that created the session, and a typed seed gives its cookie its ID;
  • each path that creates a session creates a claim on the worker that created it, including a seed whose handler never reads the session, while a request whose handler keeps no session (a health check, /aggregate_metrics, a control-style POST) and an agent request without a session create none and get no session cookie, a new session is claimed before its reply arrives, a cookie whose claim ended is claimed again by the worker that uses it, a minted ID another worker claimed meanwhile gets 409, and sessions minted on two workers at once are distinct;
  • a typed session keeps its claim while open and releases it a window after its close, a new session a window after its verify or checkpoint_session_ended, a session started without a seed a window after its last request, and a failed seed a window after the cleanup close, all without another request;
  • agent claims end with the close receipt or the failed seed, and an agent seed that creates nothing, or whose session cannot be registered, rolls its claim back;
  • a seed whose cookie names another session gets 409, with one worker and with several, and a seed forwarded after its claim moved gets 409;
  • a session, an agent session or an MCP token of a worker that exited gets 410;
  • a request over a private socket is handled where it arrives, with method, path, query and headers unchanged and only the claimant's Set-Cookie;
  • a cookie or MCP token from an earlier version, with a worker ID in it, routes by its session ID;
  • one worker writes no claims;
  • races: two workers claiming the same IDs, a seed and a close of one ID on two workers at once, a claim released while a request is forwarded, and a restarted server with a fresh runtime directory;
  • a typed seed with 9 MiB of task data reaches its claimant, and a seed without a valid typed ID is served where it lands and claimed there;
  • the private socket leaves uvicorn's process-wide loggers as the worker configured them.
• pytest tests/unit_tests -n 16, in a venv with CI's extras: passed.
• The codex_agent, opencode_agent, openclaw_agent, nooa_agent and hermes_agent tests, each in that agent's own environment: passed. The pi_agent tests: passed except one test that runs the installed Pi CLI, which fails the same way on main because of the local Node.js version.
• The simple_agent, example_session_state_mgmt, example_single_tool_call and single_agent_turn tests: passed.
• NEMO_GYM_MULTIWORKER_E2E=1 pytest tests/e2e/session_routing: passed.
• NEMO_GYM_MULTIWORKER_STRESS=1 pytest tests/e2e/session_routing/test_session_routing_stress.py: passed; see below.
• pre-commit run --files <changed files>: passed.

Rollout evidence

• Real model: feat(checkpoint): resources, agent, and environment servers with several workers #3909's real-model checkpoint runs carry this PR's routing exactly. With Qwen/Qwen3-4B-Instruct-2507 on vLLM 0.29.0 and two workers per server, every rollout that continued after a crash and restore, and every restarted rollout, finished with reward 1.0 in three runs in a row. See feat(checkpoint): resources, agent, and environment servers with several workers #3909's rollout evidence.

The opt-in e2e test runs real server processes against a fake OpenAI-compatible backend, with the counter resources server at 4 workers behind the /run relay (legacy_agent) and Simple Agent: every /run episode, every seed-then-verify session and every MCP sequence scores correctly, and killing a worker gives exactly its sessions a 410.

The opt-in stress test (tests/e2e/session_routing/test_session_routing_stress.py) runs the NOOA counter resources server at 8 workers and Simple Agent at 4 workers behind single_agent_turn, against the same fake backend, with session_close_retry_window_seconds set to 5 seconds:

Check	This change	Previous version of this change
Concurrent episodes through single_agent_turn, episode requests	4,000 of 4,000 score 1.0 in 16 s; 10,000 of 10,000 in 44 s	the same
Concurrent episodes through the flat-row wrapper (single_agent_turn_legacy)	4,000 of 4,000 in 16 s; 10,000 of 10,000 in 47 s	the same
Seeds that fail with 500 and no cookie, then a cookieless cleanup close and a new attempt	1,000 of 1,000 (2,500 of 2,500) correct	the same
Seeds whose first replies are dropped or cut off, then retried without a cookie	1,000 of 1,000 (2,500 of 2,500) correct	the same
A cleanup close before its delayed seed	every delayed seed refused	the same
Sessions created on two workers, and live sessions left after every close	none	none
Untyped seeds whose handler never reads the session (as an agent's own /run sends), then five tool calls and an end that read it, each on a new connection	1,000 of 1,000 correct, none split, none left	5 of 1,000 correct: 992 sessions were created on two or more workers
MCP sequences with only the session token, 4 workers	1,000 of 1,000 (2,500 of 2,500)	the same
Claims left 6 s after the run ended (window 5 s)	none	3,391 at 4,000 episodes, still there after 35 s: claims ended only when a later seed or close reached the same worker
A worker killed with sessions open	its sessions get 410 on every request and close; every other session finishes correctly; its claims go with the runtime directory at shutdown	the same

Routing's cost, over one kept-alive connection to the 8-worker server against the same requests to a 1-worker copy: a tool call took 1.9 ms at the median on one worker and 3.1 ms when the accepting worker forwarded it, so forwarding adds about 1.2 ms at the median and about 0.4 ms at p99 (8.8 ms against 9.2 ms). The claim lookup alone, opening and reading a small file, takes about 8 microseconds at the median.

With a real model: the checkpoint stack's opt-in tests ran Qwen/Qwen3-4B-Instruct-2507 on vLLM with two workers per agent, resources and environment server (#3909), so every session's requests were routed to the worker that holds it, and every rollout finished with reward 1.0. They ran against the previous version of this change and will run again on this one with #3909.

Pending:

• a swe_rebench run at 8 workers with real sandboxes;
• a multi-worker agent that keeps per-session state across requests.

Compatibility

• Model servers, and agent servers with one worker, are unchanged. A resources server with one worker writes no claims and opens no socket. Its typed seeds now give the cookie their resources_session_id, and a typed seed or close whose cookie names another session gets 409.
• Cookies and MCP session tokens no longer carry a worker ID. Handlers that read request.session[SESSION_ID_KEY] and the MCP endpoint's token check are unaffected.
• On a resources server with several workers, a request without a session cookie or token whose handler uses the session starts a claimed session, which ends a window after its last request unless it is a session start. Handlers see no difference.
• An agent server's routing key is agent_session_id. An agent that keeps state across requests by the cookie's session_id instead, outside agent sessions, is not routed by it, and should keep one worker.
• Agent servers with num_workers > 1 now accept agent session seeds instead of raising Agent sessions require num_workers=1.
• A request for a session whose worker is gone gets a 410 instead of running against empty state. A cookie minted before a multi-worker server restarted names a session no worker claims, so it is handled where it lands, and claimed there if the handler uses the session.
• A request that lands on another worker than its claimant costs one local Unix-socket hop, which is about (N-1)/N of a session's requests after its first. Forwarded requests no longer see the caller's client address.
• A server that remembers closed IDs longer than session_close_retry_window_seconds refuses a delayed seed through another worker only within the window; ClosedResourcesSessionRecords keeps the two in step. State keyed by an ID that the request carries nowhere in the table above is not routed.
• A session that is seeded and then never closed or verified keeps its claim as long as the server runs, as the server keeps its state.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[ananthsub]

A session whose seed never read the session was not claimed. The previous version claimed a new resources session only if its handler read or wrote request.session, but it always sent a cookie naming the new ID. The default SimpleResourcesServer.seed_session never reads the session, so an agent's own /run seed got a cookie for a session nobody claimed. Every later request with that cookie was then served on whichever worker accepted it. Servers that keep per-session state behind that seed split their sessions across workers: math_with_code keeps each session's Python REPL in _sessions[session_id], litmus_agent creates its sandbox lazily per worker, and tavily_search splits its per-session metrics. On two in-process math_with_code workers, x = 41 on one worker and then print(x + 1) with the same cookie on the other failed with name 'x' is not defined.

This version applies one rule: a session ID a client can name is a claimed session.

• A session start (/seed_session, or another route in checkpoint_session_starts) claims its new ID before its handler runs, whatever the handler does.
• Any other request with a new ID claims it only if the handler used the session, as before. If the handler did not, the reply carries no session cookie, so /aggregate_metrics, control routes and health checks still leave nothing behind.
• A request whose cookie names a session nobody claims, because the claim ended after session_close_retry_window_seconds or the cookie predates a restart, is claimed by the worker that serves it if the handler uses the session, with a 409 if another worker claimed it meanwhile. Its lifetime then follows the same rules as any other resources session.

Tests that fail on the previous version: a seed through the default seed_session on one worker, then calls with its cookie through the other worker, which now reach the seeding worker; a request whose handler keeps no session, which now gets no session cookie; and a cookie whose claim ended, which is now claimed again by the worker that uses it. A seed whose body the app refuses now keeps a claim for the window, since its reply's cookie names its session. The same math_with_code check now finds x on the second call. The stress test gains a phase of untyped seeds that never read the session, each followed by tool calls on new connections that do.

This version also adds a comment on the router's per-claimant ClientSession pools: they are not Gym's shared aiohttp client because each needs a UnixConnector bound to its claimant's socket.

[ananthsub]

/ok to test bbe9e7a