[macandro96]

What does this PR do?

ToolSimulationAgent currently reports its in-flight /run episodes as restart-only. This adds continuation through the current Gym checkpoint session API, so a rollout can resume a model wait or reuse an already completed response instead of generating it again after recovery.

This ports the ToolSimulationAgent recovery idea from #3412 to main's AgentSessionParticipant / LegacyRun contract. It does not cherry-pick that PR's older boundary protocol or its broader prefix-checkpoint stack. Main already has the shared checkpoint infrastructure, so this PR targets main independently of the NeMo RL stack.

• Record boundaries before resource seeding, model generation, verification, and returning the result.
• Park late model replies while a checkpoint is open. Retain a completed self-dispatch reply until /run records its next boundary, covering the reply-in-flight race and duplicate delivery.
• Export/restore the reply and completion marker, retain them across a second checkpoint, and reject stale attempts or a late duplicate after /run has consumed the reply.
• Declare the single-step argument-comparison verifier stateless with replayable verification. Its base, search, SWE, toolcall-schema, and Droid variants share this implementation.
• Respect the resource server's seed capabilities. Unsupported resources still make the episode restart-only; direct /v1/responses without an owning /run also remains restart-only.

Checkpointing remains opt-in. The non-checkpointed API, tool-call output, configured skip-verification reward, and comparator reward semantics are preserved. This adds no native agent-session API and does not execute the requested tools. The model participant owns any generation-prefix coordinates; this agent persists the boundary that reissues the matching call. It does not save tokens or a KV cache itself.

Validation

• 115 server tests passed: existing ToolSimulationAgent and argument-comparison verifier tests, plus 11 checkpoint tests.
• 110 shared agent/resources checkpoint tests passed.
• pre-commit run --all-files passed. The guardwords hook reported the private catalog unavailable and skipped that private-catalog check.
• All new code has NVIDIA SPDX headers; the commit has DCO sign-off.

The checkpoint tests exercise the actual agent, participant control plane, on-disk save/restore, and real comparator with a controlled model response. They cover two successive restores during a model wait, a completed reply still on its way to /run, verification-only replay, a saved final result, an unclaimed restore saved again, stale attempts, skip-verification, restart fallbacks, and the real verifier's seed capability headers.

What does this PR do?

Validation

Rollouts

Compatibility and benchmark impact

Checklist

• I have read the contributing guidelines.
• The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs.
• Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable).
• Documentation added or updated, or N/A with justification.
• Pre-commit checks pass locally (pre-commit run --all-files) (so CI lint/format checks pass).
• New source files include the required Apache-2.0 SPDX header.
• All commits have DCO sign-off (git commit -s) (so the DCO check passes).

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[macandro96]

/ok to test e3989f1

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-4798-e3989f140f51.docs.buildwithfern.com/nemo/gym

[zyzhou5]

LGTM after the fixes

[github-actions]

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/39851661791