fix(core): isolate server signals and drain agent sessions on shutdown by RyaliNvidia · Pull Request #5141 · NVIDIA-NeMo/Gym

RyaliNvidia · 2026-10-27T18:02:56Z

What does this PR do?

Isolate managed servers in their own process sessions so an interrupt reaches Uvicorn once and permits lifespan cleanup. Keep the logging tee alive, wait within one shared configurable grace period, signal all surviving process groups with SIGKILL, then reap within a separate shared bound. Descendants remain owned even after their wrapper exits.

Drain agent sessions concurrently on FastAPI shutdown through the same close-by-ID operation as HTTP close. Preserve partial canceled seeds, failed-seed cleanup, shielded continuation closes, checkpoint state and retry receipts. Unconfirmed cleanup retains handles and reports session IDs.

Incorporates main 3bf99705a905f9aefb18b03875a003189ff1edbf, retaining its capability/stop routes, cancellation behavior, MCP timeout settings and rollout-claim ownership. The extracted close operation releases main's rollout claim while preserving new-versus-cached receipt cookie behavior.

Part of #3160; complements #3735. This does not close the broader lifecycle issue.

Configuration and compatibility

server_shutdown_timeout_seconds: shared finite nonnegative server grace, default 1 second; zero requests immediate escalation. Reaping has a separate shared bound. Telemetry, sidecar and head-server teardown remain outside this grace.
session_close_timeout_seconds: base agent close timeout and total shutdown-drain budget, default 60 seconds. Existing Hermes/Stirrup 30-second overrides remain.
Server grace must exceed session drain plus Uvicorn/process overhead; docs include a 70-second server / 60-second session example. An outer platform deadline must allow both.
Timeout does not establish remote termination. Cleanup state remains available for retry; ordinary prompts, verification and scoring are unchanged.

Cleanup entered after a caller has already caught cancellation now completes normally. Each close/release records the task's cancellation count at entry and rejects only a newly requested cancellation. This preserves bounded shutdown and cancellation-suppression protection while allowing Nooa's caller-cancellation cleanup to close both Agent and Resources sessions. No Nooa test expectations were changed.

Validation

After integrating main's server-runtime module split, 83 lifecycle/shutdown checks and the complete 11 Nooa environment checks passed again, including both unchanged interruption cases. Independent final review passed 12 targeted checks; final hooks and whitespace checks passed. The earlier full Nooa wrapper result below was recorded before this namespace-only integration.
Fixed the required Nooa caller-cancellation CI failure: the previous nonzero-count guard incorrectly treated an already-handled cancellation as a new interruption, stopping cleanup before Resources close.
Six focused cancellation cases cover normal close and failed-seed release, handled caller cancellation, and newly suppressed cancellation with and without an inherited cancellation. Focused retry/activation suites: 66 passed. Broader lifecycle/session/checkpoint run: 278 passed. Independent verifier: 12 targeted checks passed; simplifier found no worthwhile simplifications.
Restoring the old guards in memory makes both handled-cancellation regressions fail. The original cancellation still propagates; its count is not reset. New cancellation still retains retryable state and prevents success receipts.
The unchanged Nooa interruption cases (episode timeout and caller cancellation): 2 passed. The complete actual isolated Nooa wrapper: 423 passed, 8 skipped, using the exact pinned core/CLI/bench source revision 051472343211914222e24ce36d8752f4e86bbe43. A same-commit archive was used after Git fetching stalled; temporary setup constraints allowed Nooa's declared Rich minimum while preserving the model SDK pins, including OpenAI 2.44.0. No repository dependency policy or Nooa test assertions changed.
pre-commit run --all-files, Ruff/format and whitespace checks passed. The private guardwords scan was skipped because its catalog was unavailable.
Earlier validation of the complete main integration remains applicable: all four isolated Pi/OpenCode/Codex/OpenClaw wrappers passed (1,752 passed, 182 skipped); the one macOS process-memory test failure also reproduced on untouched main. Fern validation passed with zero errors and an unauthenticated redirect-check warning.

Live model/sandbox rollout: N/A for this lifecycle correction; checks use synthetic fixtures and local subprocesses. No live provider or remote sandbox qualification is claimed. No dependency pins, timeout settings, shields, or conformance expectations were changed. No uncancel() calls were added. Commits include DCO sign-off. No comment replies or trust-approval comments were posted.

Proposed maintainer labels: bug, area:orchestration, needs-review.