Generation-safe policy model checkpoint cuts #3218

Parent issue

Sub-issue of #3024.

Problem

The current experimental checkpoint participant can close policy-model admission and wait for whole accepted HTTP requests, but that is too coarse for long generations. A safe checkpoint needs a precise token boundary: every pre-cut token must be durable in the framework token store, post-cut tokens must remain outside the checkpoint, and Gym's token-free lineage must reference the same boundary.

A completed call must also become checkpoint-safe when its custody record is durable, even if its HTTP response has not yet been delivered to the caller.

Scope

• Separate response lifetime from generation-cut safety in admission tickets.
• Define framework-neutral active-generation inventory, cut, and acknowledgement contracts.
• Fence response egress until the checkpoint disposition is known.
• Record durable success and durable failure explicitly.
• Aggregate cut acknowledgements across managed server workers.
• Validate token-store coordinates and lineage identity before commit.
• Cover deterministic active-prefix and completed-before-egress races with a fake backend.

Real vLLM, Megatron inference, and framework token-store implementations are tracked as dependent children. Experimental implementation remains under nemo_gym._checkpoint.

Acceptance criteria

• New requests cannot enter the frozen cut.
• Every admitted call is classified as no-generation, durably completed, durably prefix-cut, durably failed, or excluded.
• Prepare does not wait only for HTTP response delivery.
• Duplicate cut/ack operations are idempotent.
• Partial multi-worker acknowledgement and deadline expiry fail closed.
• Gym lineage and external token-store receipts identify exactly the same token boundary.
• Abort/resume cannot publish a stale response or lineage root.