[waple0820]

Closes #2610

Builds on #2611 (both touch BaseVerifyResponse); please review that one first.

#2114 landed the rollout correlation contract, but two gaps keep incident attribution manual.

It is one-directional. BaseSeedSessionResponse is an empty model and BaseVerifyResponse carries no identifier, so the environment-side handle — the container, browser context or provider session that actually consumed quota — never reaches the training side. #2122 states this explicitly: "No rollout identifier is added to public request schemas."

It is gated on model-call capture. server_utils.py:341-348 only applies the rollout prefix to resources servers when observability_enabled is true, and that flag defaults off, so current_rollout_id() is None inside resources servers exactly in the runs where an incident happens.

What this changes

class BaseSeedSessionResponse(BaseModel):
    env_session_id: Optional[str] = None   # container / browser context / provider session

class BaseVerifyResponse(BaseVerifyRequest):
    env_session_id: Optional[str] = None   # same handle, returned with the score

plus a rollout_correlation_enabled global-config key that turns on the resources-server prefix without turning on capture. It defaults to false, so nothing changes for existing runs — happy to flip the default if you would rather have correlation on by default.

Why it matters

When a run degrades, the training side has rollout ids, the provider has environment session ids, and the only join key is a timestamp. Reconstructing which orphaned sessions belonged to which aborted rollouts across our own incident (1172 of 1280 rollouts over 20 steps) was done by hand; with a returned session id it would have been a groupby. This is also the environment-side half of #238.

No training framework needs a transport change: verl and NeMo-RL already carry the whole verify response as full_result.

Terminology note: this is exclusively the environment session created by /seed_session, not the vLLM router KV-cache affinity of #2570 / #2347 / #2369.

Tests

Round trip through seed and verify, plus test_rollout_correlation.py: correlation is off by default, and the new flag prefixes resources-server calls without full observability.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[waple0820]

Rebased onto main and out of draft.

This PR used to be stacked on #2611 and carried its commit, so the diff looked larger than the change. Unstacked — the two are independent now and can land in either order.

One connection to the error-handling design in #2750 that was not in the original description: the ReplayPolicy contract there puts POST at before_delivery by default, which is right for a request that performs work. But /seed_session allocates a resource, and under that rule a response lost on the way back leaves the session alive with the caller holding no id for it — nothing a release or a sweeper can act on. A caller-minted env_session_id makes /seed_session eligible for deduplicated instead: replay is safe, and a lost response is still releasable. Detail in #2750 (comment).

The field here is optional and defaults to absent, so it does not require that policy change — it just makes it possible.

[ananthsub]

This PR should remain diagnostic observability work for #2610/#2831. The provider env_session_id is opaque correlation metadata; it must not become the allocation, deduplication, authorization, or cleanup identity. Those semantics belong to caller-created _ng_session_id and _ng_session_close_token in #2609. Rebase this after #2609 establishes that distinction.

[waple0820]

@ananthsub agreed on the distinction, and it is now stated at the definition rather than only in the PR body:

# Diagnostic correlation only. It is not the allocation, deduplication, authorization
# or cleanup identity - those belong to the caller-created `_ng_session_id` and its
# close capability (#2609), which exist precisely because a provider handle is learned
# from this response and is therefore unavailable when the response is lost.
env_session_id: Optional[str] = None

Two tests hold it: absent by default on both ends, and surviving the seed-to-verify round trip. The PR is 2 files, +36/-1 — the rollout_correlation_enabled half was dropped when #2783 made correlation independent of the observability gate.

Ready whenever #2609 lands; nothing here conflicts with it.

[ritaneves]

@ananthsub what's the status of this, should we merge?

[waple0820]

Closing this. Since #3717 the seed request carries episode_id, and the same id comes back on the episode result, so an environment can log its provider handle against the rollout directly, without depending on capture being enabled. That covers what this PR was after on the protocol Gym is moving to, and adding fields for the legacy /run path alone doesn't seem worth it. #3932 does this for interactive_browser. Thanks @ritaneves for checking in.