[tdene]

apptainer's per-container session tmpfs has a size capped by sessiondir max size. SWE commonly exceeds the default value, with a simple git reset --hard usually killing the episode.

This PR backs the writable layer with a directory overlay that is freshly created under TMPDIR and removed when the container exits.

Episode results currently land next to the server sources and accumulate forever. This PR adds results_root to be able to specify a different root directory - and avoid disk overflow - and preserve_episode_artifacts which can clean up stale episode results if set to False.

[tdene]

/ok to test d68e7bc

[tdene]

/claude review

[claude]

RISK — overlay dirs leak on the exact failure mode they're meant to protect against (disk exhaustion).

trap 'rm -rf "$OVERLAY_DIR"' EXIT TERM INT only fires on those three signals. But this project's own teardown for timed-out / OOM episodes is SIGKILL, not SIGTERM: _kill_container_tree sends SIGKILL (app.py:2317, 2322) and _finish_container_command calls process.kill() (SIGKILL) on timeout (app.py:2494), as does _kill_active_command (app.py:2539). SIGKILL cannot be trapped, so the shell dies without running the trap and $OVERLAY_DIR under $TMPDIR is orphaned.

Blast radius: long training runs are precisely the ones that repeatedly hit agent/eval timeouts and the memory watchdog. Each killed episode leaves a full SWE working-tree overlay in $TMPDIR; these accumulate until $TMPDIR fills — the same "filesystem fills mid-run" failure preserve_episode_artifacts was added to prevent, but this path isn't governed by that knob and is never reaped.

Fix: track the overlay path Python-side and rmtree it in _kill_active_command / the timeout branch after the process is reaped, and/or add a startup sweep of stale apptainer_overlay_* dirs under $TMPDIR (the fixed prefix exists for exactly this). The trap alone only covers clean exits, not the kill paths that dominate at scale.

[claude]

SHIP WITH CARE

Sandbox-hardening + episode-scratch-lifecycle change to the SWE agent harness. Net improvement: the --writable-tmpfs → TMPDIR-backed directory overlay swap fixes real episode kills (working trees exceeding the capped tmpfs), the /etc/hosts and git safe.directory fixes are correctly made non-fatal, and preserve_episode_artifacts / results_root give operators a way to keep long runs from filling the shared FS. Tests cover the new paths well, including a stub-apptainer integration test for overlay lifecycle. No verifier/scoring or async-HTTP surface is touched — scores are unaffected.

One reliability finding worth resolving before this rides a long training run (inline on app.py:3362):

• RISK — overlay dirs leak under SIGKILL teardown. The overlay cleanup is a shell trap ... EXIT TERM INT, but the harness kills timed-out/OOM episodes with SIGKILL (_kill_container_tree, process.kill()), which is untrappable. Orphaned overlays accumulate in $TMPDIR until it fills — the same disk-exhaustion failure preserve_episode_artifacts guards against, but on a path that knob does not cover. Fix: reap the overlay Python-side in the kill/timeout branches and/or sweep stale apptainer_overlay_* at startup.

NOTE (author's call): results_root / preserve_episode_artifacts are new TypedDict-ish config fields with Python-side defaults on the Pydantic SWEBenchWrapperConfig. Per CLAUDE.md, defaults belong in the exemplar YAML — but the existing fields on this config (container_formatter, swebench_tests_timeout, etc.) already carry Python defaults, so this matches local convention rather than deviating from it. Not a blocker.

[claude]

RISK — overlay dirs leak on the exact path they're meant to protect against (disk exhaustion).

trap 'rm -rf "$OVERLAY_DIR"' EXIT TERM INT only fires on those three signals. But this project's own teardown for timed-out / OOM episodes is SIGKILL, not SIGTERM: _kill_container_tree sends SIGKILL (app.py:2317, 2322) and _finish_container_command calls process.kill() (SIGKILL) on timeout (app.py:2494), as does _kill_active_command (app.py:2539). SIGKILL cannot be trapped, so the shell dies without running the trap and $OVERLAY_DIR under $TMPDIR is orphaned.

Blast radius: long training runs are precisely the ones that hit agent/eval timeouts and the memory watchdog repeatedly. Each killed episode leaves a full SWE working-tree overlay in $TMPDIR. These accumulate until $TMPDIR fills — the same "filesystem fills mid-run" failure the preserve_episode_artifacts knob was added to prevent, but this path isn't governed by that knob and never gets reaped.

Fix: add a startup sweep of stale apptainer_overlay_* dirs under $TMPDIR (they carry a fixed prefix for exactly this reason), and/or add KILL handling by tracking the overlay path on the Python side and rmtree-ing it in _kill_active_command / the timeout branch after the process is reaped. The trap alone covers clean exits, not the kill paths that dominate at scale.

[tdene]

Thanks for the review, you're absolutely right. I've addressed this in my follow-up commit.

[tdene]

/ok to test 5eb218e

[tdene]

/claude review

[claude]

SHIP — LGTM, no reliability concerns.

Reviewed the SWE-bench harness sandbox-hardening + episode-cleanup change end to end:

• persistent_dir rmtree is correctly scoped (app.py:3585). It removes a per-episode directory ({instance_id}_{ts}_{uuid}, app.py:3447), not the shared results root. _inner_responses returns a fully-materialized NeMoGymResponse (metadata is in-memory JSON, no post-return file reads) before cleanup runs, and it's gated on not preserve_episode_artifacts — default True, so existing behavior is unchanged. Failed episodes take the except path and keep their dir + traceback.err, matching the documented contract.
• Overlay reap is env-consistent. Shell ${TMPDIR:-/tmp} and Python os.environ.get("TMPDIR") or "/tmp" resolve identically (both map unset/empty → /tmp), and both execute in the same runner_ray_remote process, so the reap targets the same path the launch shell created. The dirname is generated in Python and threaded onto ExecuteContainerCommandArgs/ActiveContainerCommand so the SIGKILL teardown paths (timeout branch of _finish_container_command, _kill_active_command) can reap by name where the shell's EXIT trap never fires.
• No leak on any exit path. The finally reap gates on returncode is not None; both normal completion and timeout-then-kill set it. Reap racing the shell's own trap is harmless (ignore_errors=True). Per-execution UUIDs prevent agent/eval overlay collisions.
• /etc/hosts and git safe.directory writes are made non-fatal — correct, these were a real episode-abort source under the overlay's uid remapping.
• Test coverage is strong: both kill paths parametrized, overlay lifecycle via stub apptainer, results_root relocation, and preserve_episode_artifacts both ways.

No public API, async, or scoring-correctness surface touched.

[oyilmaz-nvidia]

/review

[svcnvidia-nemo-ci]

SHIP WITH CARE — two RISK findings remain in the cleanup paths.

Reviewed both changed files and traced container teardown, Ray result handling, and response materialization. The earlier SIGKILL overlay-leak counterexample is addressed by the Python-side cleanup and covered by the new teardown regression cases.

Validation was static source/test inspection only; tests and model smoke rollouts were not executed. Suggested changes below require author verification.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

[svcnvidia-nemo-ci]

RISK — Preserve diagnostics for handled execution failures

Failure / blast radius: With preserve_episode_artifacts=False, an agent timeout or watchdog kill reaches the worker's exception handler, which records metrics and returns None rather than raising. _inner_responses then constructs a normal response, so this branch deletes the failed episode's logs. Eval failures and golden-patch eval failures take the same route. This contradicts the new retention contract and removes diagnostics precisely for users enabling cleanup during long runs.

Correction: Distinguish a completed benchmark attempt from a caught execution failure. One small implementation is to add execution_failed: bool = False to SWEBenchMetrics, and assign metrics.execution_failed = True before persisting metrics in the two process_single_datapoint exception handlers and the _run_golden_patch_verification exception handler. Replace this cleanup block with:

if not self.config.preserve_episode_artifacts:
    metrics = SWEBenchMetrics.model_validate_json(response.metadata["metrics"])
    if not metrics.execution_failed:
        await asyncio.to_thread(rmtree, params.persistent_dir, ignore_errors=True)

The invariant is that caught execution errors retain their episode directory even though the API returns a response. Do not use resolved=False as the failure discriminator: a valid unsuccessful patch or ordinary no-patch result should still be eligible for cleanup. The new metric flows through the existing metrics serialization and inherited verify-response model.

Preservation / proposed acceptance checks (not run):

• Exercise timeout, nonzero-exit/OOM, and golden-eval exception handlers: the response still returns as before, but execution_failed=True and the episode/log files remain with preservation disabled.
• A normal unresolved or no-patch attempt with execution_failed=False is still removed when preservation is disabled.
• Retain the exists() is preserve assertion in the new cleanup test, updating its mocked response to carry real metrics JSON.
• Run the outer exception case with preservation disabled and assert that its directory and traceback.err survive; preservation enabled must retain all episodes.

[svcnvidia-nemo-ci]

RISK — Run recursive deletion outside the async event loops

Failure / blast radius: A completed agent with a large populated directory overlay now calls _reap_overlay_dir synchronously here. Its recursive filesystem traversal blocks the Ray worker's event loop, including the concurrently running eval container's memory watchdog. _kill_active_command does the same. With preservation disabled, the new deletion in responses additionally blocks every concurrent request on the server loop while the large episode tree is removed. Cleanup should delay only the operation awaiting it, not unrelated requests or watchdog polling.

Correction: Keep the synchronous helper, but replace both async callers' direct _reap_overlay_dir(...) calls with:

await asyncio.to_thread(_reap_overlay_dir, active_command.overlay_dirname)

Likewise, replace the endpoint's direct rmtree(...) with:

await asyncio.to_thread(rmtree, params.persistent_dir, ignore_errors=True)

Keep the existing returncode guard, kill/wait ordering, and preservation/failure-retention conditions. Await the thread rather than launching fire-and-forget cleanup, so teardown still guarantees deletion has finished before returning. This preserves the prior SIGKILL fix, TMPDIR resolution, and per-command isolation.

Proposed acceptance checks (not run): Use a deletion stub blocked on a threading.Event; while teardown awaits it, assert that another coroutine/heartbeat can run, then release it and assert teardown finishes only afterward. Cover both worker cleanup methods and endpoint cleanup. Retain the SIGKILL tests' assertions that the process is reaped and its overlay is absent, plus the normal-exit lifecycle assertion that the overlay exists during execution and is absent afterward.