[svcnemo-autobot]

Background

This is the nightly main → dev sync. It merges the oldest 100 commits of dev..main, 897649d0..19deef67.

• dev base: 87effd07995c95b21e1a53bcfcc11dabbff8c485
• slice tip on main: 19deef67f910c96c213f33b33b30277be8b94d6d
• dev has since advanced to c0d3b03f ([data] Support Nemotron-SFT, RP2, and CodeForge datasets in the varlen loader; prefetch packed global batches #5771). git merge-tree merges it into this head with no conflict, so the branch was not re-merged.

What changed

The branch is 102 commits ahead of dev:

• 100 original main commits, unchanged and in order (checked against the slice list).
• 1 merge commit, 1eaec4752693a60f5d8a85baa065e0ab15312c9a, with parents 87effd07 (dev) and 19deef67 (main).
• 1 squashed repair commit, 4a534df611eeb12250fad0b09a1a9d2eccf83fec: 14 files, +363/−38. Its patch is identical to the previous repair. It covers:
  • the scratch buffer for the inference mamba cache
  • megatron_fsdp experimental symm-mem allocation
  • the per-iteration RNG reset in moe_perf
  • teardown of the RL unit-test fixtures
  • test doubles and cudagraph test isolation
  • restoring main's optional sock argument on the dynamic text-gen server, which the Thread socket argument through MInf API #6513 callers use. A new unit test covers it.

Details

• Rebuilt on dev 87effd07, picking up 6 dev commits: [dev] Restore expert wgrad overlap with shared experts #7390, [dev] Fix a race in in-place fused MLA RoPE with remove_interleaving=False #7789, [dev] Share the fused compact DSA indexer workspace across CSA layers #8104, [dev] Fix fused MLA RoPE key/value split for non-power-of-two head dims #7803, [dev] Use binary search for the THD row lookup in fused MLA RoPE #7806 and [dev] chained optimizer fix for cuda graph #8103.
• Conflicts: the earlier hand resolutions carry over, including the 58 merge files and test_random.py. The rebuild added one new conflict, in tests/unit_tests/fusions/test_mla_yarn_rope_apply.py. It was resolved as a union that keeps main's test_inplace_forward_backward and dev's test_kv_split_non_power_of_two_head_dims.
• Restorations: none.
• Dependency files: .github/CODEOWNERS, pyproject.toml, uv.lock and docker/Dockerfile.ci.dev are identical to dev.
• Impact of the dev changes:
  • ChainedOptimizer takes its tensor path only for a Tensor norm, and every producer returns a float.
  • Fused MLA RoPE: the THD binary search is integer-identical, the kv-split constexprs are unchanged for power-of-two dims, and tl.debug_barrier is a race fix.
  • dsa_compact_indexer_workspace_sharing defaults to True.
  • The early-wgrad flag is used only by 3 unit tests.
• Pre-push advisory (from the nightly-sync skill): 2715 finding lines and 0 guidance warnings, all dispositioned.
• Lint: black, isort and ruff are clean, and pylint scores 10.00 on the 13 changed files.
• Remerge-diff: git show --remerge-diff 1eaec475 shows 60 files.
• Signatures: both bot commits are SSH-signed, and GitHub reports them verified.
• Review:
  • The strict-review findings F1–F5 are in inference code that came from main. Fixes are in separate PRs against main: fix(inference): AUT-2539 keep PP stages consistent when only some hold recurrent #8250, fix(dist-checkpointing): AUT-2539 key the PG distribution cache by the full proce #8251, fix(inference): AUT-2539 hold back unstable reasoning in streamed chat deltas #8254, fix(inference): AUT-2539 reply to a KV handoff import that fails before admission #8255, fix(inference): AUT-2539 handle string prompts and rejections in KV hand #8258 and fix(inference): AUT-2539 fail impossible KV handoffs on arrival and honor aborts #8267.
  • The dev conflict raised in FDecaYed's review is resolved here.
  • Codecov reported 70.14% patch coverage on the previous head, against an 80% target.
• DCO: a maintainer override is needed, because the imported commits keep their original authors.
• Full-suite labels: Run MBridge tests was requested through the approved mechanism.

Tested

Local H100 runs used image megatron-lm@sha256:2cbc1243aead76636d4a558e5822bbb5466c4361b324f8396e1a95ec4c1b55ed. The final sweep ran on 4a534df6, with rows chosen by what the dev changes affect.

• Unit, all passed:
  • tests/unit_tests/**/*.py
  • models/**
  • transformer/**
  • transformer/moe/**
  • a2a_overlap/**
  • test_multi_latent_attention.py: 13 skips, all declared in the test file
  • test_dsv4_hybrid_attention_cp_cuda_graph.py: 3 skips, which need SM100+ or are covered under CP2
• Functional, all passed:
  • tp1_pp2_dsv4_hybrid_mhc_mtp
  • tp2_pp2_mla
  • tp2_pp2_dsa: smoke run only, because no golden values exist
• Not re-run:
  • The other 23 unit and 78 functional rows keep earlier passes, because their inputs are unchanged.
  • common/moe_perf and ..._logitsmatch are left to the full-suite CI.
• CI on this head: pending.

Ordered 100 source SHAs

897649d 14346b6 15c83d2 81fe7c7 8534490 fad6111 b2f2888 5436289 a34880f 3cd8cf3 3ce4b4a 301e0d6 5714a20 fecb29f 8ba5c60 281075d 2a75ac1 87bf520 487bb39 09921c0 2119719 87cc590 d7430d5 f7b2209 69e282e 3d0956f 0ee415b 94939b8 f9cc544 de4a82e d918967 d6d48cc ad3ecd9 6c34429 d347eef 4dc5c96 c8b2a4a 41ba4d9 cac6b7b 814300c 78ad108 883533e 0615ac9 a4417a5 669d6ed 51f6a39 e79110c e60e42f 9e96f45 c95b0da b164d74 c374e5b d262dbe 34c3741 32e7eda 62a6173 afb55dd 43c0e02 8638022 51270a3 fcc3319 dd8c825 39d0896 e4ba3a3 7cdba42 d7e2c13 574e04e 59ff171 35ae46c 5d50b16 32756e3 ae5ec14 cfb8e46 dcb2bfd bd72caf c9676f4 3126ff2 74a4894 0980313 54c7918 d5a019a aaa4dfd aeae5d7 5f9b2bb 342faa4 e5542e8 bf51cf4 0961f12 e7d58db 29361d6 c452d5a 3ad8c2a 540f420 1d69364 afa707e cab7116

[svcnvidia-nemo-ci]

/ok to test 2e5f599

[ko3n1g]

/review reset=true mode=strict

[svcnvidia-nemo-ci]

Strict review — incomplete

Findings: CRITICAL: 11, IMPORTANT: 1, SUGGESTION: 0. Overall risk: significant. Eleven concrete findings from review 5529654466 remain unresolved. One additional checkpoint-cache collision is source-validated below. The inline comments contain F1–F5; the remaining findings follow here.

Verified fix: prior F9's HTTP-server signature mismatch is resolved. The real server accepts sock, and the new tests bind both serving APIs to that signature and exercise pre-bound sockets. That finding is closed.

Why the remaining tasks cannot be completed here

The all-changed-file inspection and whole-scope cross-component validation tasks remain pending. This broad review covers 343 changed paths and a 1,879,821-byte diff, including 290 omitted diff sections requiring head/base reconstruction. Substantial kernel, test, and configuration coverage remains, alongside full-context and added-identifier runtime-use audits of partially inspected large files. Completing those obligations with the available read-only operations exceeds this final bounded continuation. The cross-component task also depends on that unfinished inspection. I have therefore left both tasks pending rather than falsely marking them complete.

The twelve published findings are individually source-validated; this is not a clean assessment of the unreviewed changes. This RESET review compares the current head with the supplied dev base: importing code unchanged from main does not eliminate a regression relative to that base. The required protected-base style-guide file was unavailable. No tests were executed; acceptance checks below are proposed. Suggested code was traced against inspected callers and preservation cases but requires author verification.

F6 — Preserve positional model-constructor compatibility

[IMPORTANT Compatibility] Failure: logit_dtype was inserted before parallel_output in GPTModel and HybridModel. For example, GPTModel(config, spec, 128, 32, True, True, False, False) previously selected parallel_output=False; it now binds logit_dtype=False and leaves parallel output enabled. Every later positional argument also shifts. Compare the base GPT signature and base Hybrid signature. This is unresolved prior F6.

Correction: move the new optional parameter after every pre-existing parameter in both constructors. Apply these signature edits independently:

-        logit_dtype: Optional[torch.dtype] = None,
         parallel_output: bool = True,
@@
         vp_stage: Optional[int] = None,
+        logit_dtype: Optional[torch.dtype] = None,

Preservation and acceptance — proposed: signature-binding tests must retain every old positional index, including pg_collection and vp_stage. The GPT example must bind parallel_output=False, logit_dtype=None; test the analogous Hybrid positional sequence. Keyword logit_dtype=torch.float32 must still reach the output projection. Retain the GPT builder assertions and Hybrid builder assertions: FP32 logits, parallel_output is False, enabled embedding sharing, and unchanged process-group/virtual-stage forwarding.

F7 — Normalize string handoff prompts before hashing or tensorization

[CRITICAL Implementation] Failure: both client handoff APIs accept strings, but handoff admission tensorizes the prompt directly. With otherwise valid remote metadata, prompt="hello" raises at torch.tensor(prompt, dtype=torch.int64) instead of entering decode. The scheduler calls this without converting the exception into a terminal client response. This is unresolved prior F7.

Correction: change the handoff method's prompt annotation to str | list[int], and replace the tensorization statement with:

if isinstance(prompt, str):
    prompt = self.controller.tokenize_prompt(
        self.controller.tokenizer, prompt, sampling_params.add_BOS
    )
prompt_tensor = torch.tensor(prompt, dtype=torch.int64)

This follows the inspected ordinary tokenizer/BOS contract. The normalized list must supply lengths, hashes, deferred transfer state, and final request construction, without applying BOS again.

Preservation and acceptance — proposed: use a tokenizer producing [1,2,3,4] and valid matching handoff metadata. String and token-list submissions must store/hash identical IDs and complete through streaming and non-streaming clients. For strings, add_BOS=True must produce exactly one leading BOS, while false retains ordinary leading-BOS normalization. Pretokenized lists must remain unchanged even when add_BOS is set. Assert that the engine remains alive after submission.

F8 — Exclude imported decode prompts from the prefill-token limit

[CRITICAL Implementation] Failure: import finalization calls ordinary admission, whose prefill guard rejects prompts exceeding context.max_tokens without chunking. A completed 12-token import into a decode context with max_tokens=8, maximum sequence length 32, sufficient KV capacity, one resume token, and two requested generated tokens is therefore FAILED with TokenOverflowError. Expected work is a one-token decode query, not execution of the imported prompt. This is unresolved prior F8.

Correction: explicitly distinguish prefilled imports and bypass only the prefill-token guard. Suggested edits to engine admission and the linked finalizer:

@@ DynamicInferenceEngine._add_request
-        self, request: DynamicInferenceRequest
+        self, request: DynamicInferenceRequest, *, prompt_is_prefilled: bool = False
@@
-        if len(request.prompt_tokens) > self.context.max_tokens and not self.enable_chunked_prefill:
+        if (
+            not prompt_is_prefilled
+            and len(request.prompt_tokens) > self.context.max_tokens
+            and not self.enable_chunked_prefill
+        ):
@@ DynamicInferenceEngine.add_request: after precomputed_block_hashes
+        *,
+        prompt_is_prefilled: bool = False,
@@
-        return self._add_request(request)
+        return self._add_request(request, prompt_is_prefilled=prompt_is_prefilled)
@@ InferenceStateHandoffMixin._finalize_kv_handoff_import: self.add_request(...)
+            prompt_is_prefilled=True,

Update handoff test doubles to accept the optional keyword. Do not bypass maximum-sequence-length, generation-budget, or total-KV-capacity checks.

Preservation and acceptance — proposed: exercise real shared validation, not only a mocked add_request. The imported example must enter decode at KV offset 12, query length 1, with zero prefill requests. An ordinary 12-token prompt must still fail without chunking and remain eligible with chunking. Oversized imported sequences/KV footprints must still fail, and generation-length clamping must remain intact. Retain the exact SSM/MTP admission assertions: positions [3,4,5], block IDs [10,11,11], and SSM slot 10.

F10 — Include GTP in MIMO singleton optimizer writer election

[CRITICAL Correctness] Failure: the newly enabled heterogeneous layout separates GTP from DP, but MIMO's singleton replica ID includes only TP/PP/DP. In the TP2/GTP2/DP1/PP1 language layout, both TP-rank-zero GTP peers return (0,0,0). Consequently, singleton optimizer metadata extraction emits duplicate main replicas for the same param_state_sharding_type key, and for extracted parameter groups/scaler when present. Object integrity validation rejects them. This is unresolved prior F10.

Correction: replace _get_replica_id's final return with the following and update its docstring:

replica_id = (
    pg_collection.tp.rank(),
    pg_collection.pp.rank(),
    pg_collection.dp.rank(),
)
gtp_group = getattr(pg_collection, 'gtp_remat', None)
if gtp_group is not None and gtp_group.size() > 1:
    replica_id += (gtp_group.rank(),)
return replica_id

The invariant is one main writer per module-wide singleton. Do not apply singleton election to genuinely sharded optimizer tensors.

Preservation and acceptance — proposed: gather singleton metadata across the four language ranks and assert exactly one main replica per module/suffix/key; save/load optimizer state with integrity validation enabled. Absent GTP and size-one GTP must retain the old tuple and TP/PP/DP election. Preserve independent module/chained-optimizer keys, current parameter-ID restoration, and the existing TP-deduplication save assertion. This remains distinct from F11 and F12.

F11 — Retain GTP-inclusive metadata for replicated MIMO model state

[CRITICAL Correctness] Failure: MimoModel.sharded_state_dict overwrites checkpoint metadata with pg.dp_cp, which excludes GTP peers in the new topology. Corresponding replicated Mamba parameters on both GTP peers therefore receive the same main replica ID. This affects direct Mamba parameters, and the gathered in_proj checkpoint tensor. Access-integrity validation sees duplicate main writers. This is unresolved prior F11 and remains independently reproducible on replicated entries after F12's earlier assertion is repaired.

Correction: replace the linked MIMO metadata assignment with:

mod_metadata['dp_cp_group'] = (
    getattr(pg, 'dp_cp_gtp_remat', None) or pg.dp_cp
)

Keep the GTP-excluded true-replica group separately available for genuinely GTP-sharded weights, as required by F12. Do not redefine pg.dp_cp, since reductions still need its excluded domain.

Preservation and acceptance — proposed: through MIMO's own sharded_state_dict, gather metadata for A_log, convolution parameters, final norm, and gathered/split in_proj; each replicated logical shard must have exactly one main writer. Model-only save/load must round-trip under TP2/GTP2/DP1/PP1. Legacy/non-GTP partial collections must retain fallback behavior. Preserve genuine GTP shard offsets/padding and one writer per offset. Retain the Mamba replica-ID assertions and the inclusive-versus-excluded distinction in the immediately following regression.

F12 — Use explicit topology throughout GTP checkpointing

[CRITICAL Correctness] Failure: heterogeneous MIMO deliberately bootstraps without MPU groups, and this PR enables GTP in that path. Nevertheless, GTP checkpoint metadata construction falls back to global DP rank. Its sharded-tensor branch also invokes a helper that unconditionally queries global DP rank for GTP tensors. Checkpointing therefore raises the data-parallel-group-not-initialized assertion despite supplied explicit groups. A minimal isolated trigger is a GTP-enabled embedding checkpointed through its explicit-group path with MPU uninitialized. This is unresolved prior F12.

Correction: thread the module's explicit GTP-excluded true-replica group through the MIMO metadata producer, the ordinary transformer checkpoint wrapper, the optional intra_dp_cp_group path in the GTP helper, and the direct embedding/single-tensor helper. Both GTP helper branches must use supplied topology instead of querying MPU; grouped experts need their expert replica domain, not the dense one. Keep global lookup only as a fallback when explicit topology was not supplied. Exact API plumbing remains an implementation choice, so no speculative blanket patch is proposed.

Preservation: F11's replicated/gathered tensors require the inclusive domain; genuinely GTP-sharded tensors require the excluded domain and distinct offsets. Using the inclusive rank everywhere would leave nonzero GTP shards without main writers. Preserve TP×GTP offsets, alignment padding, native-FP8 dequantization/backlinks, and SSM projection split/merge behavior.

Acceptance — proposed: with MPU globals uninitialized, construct explicit TP/GTP/replica groups and checkpoint a GTP embedding and TE projection. Assert no global-group lookup and exactly one main writer per distinct GTP offset. Save/load a small heterogeneous MIMO model and optimizer; repeat legacy initialized-MPU calls to retain fallback behavior. Retain the embedding writer-election assertions, GDP padding/split regressions, and the heterogeneous test's train/save/resume assertions.

F13 — Key checkpoint distributions by the complete process group

[CRITICAL Correctness] Failure: the new cache filename uses only the minimum global rank, and the process-global memo uses that same filename before gathering. Independently selected load/save group families need not share this identity: the configuration allows DP loading and expert-DP saving with one cache directory, and both callers pass that directory to their respective wrappers.

A minimal four-rank setup uses DP [0,1,2,3], expert-DP [0,2]/[1,3], and four fully replicated tensors of distinct positive sizes whose original main replica is global rank zero. First compute the cached DP load distribution, then compute the expert-DP save distribution from fresh metadata in the same job/config. Both DP and [0,2] resolve to pg_dist_0.pkl. The DP greedy distribution elects local ranks 0, 1, 2, and 3; reusing it in a two-rank group leaves two tensors with no possible writer. The application code marks those replicas non-main, so integrity validation fails, or shards are omitted when that validation is bypassed. Expected behavior is the same election as uncached computation for the actual group. The base implementation gathered each requested group separately.

Correction: identify both disk and memory entries by complete ordered group membership, not its minimum. At the linked helper, add import hashlib and replace its final two statements with:

ranks = torch.distributed.get_process_group_ranks(parallelization_group)
group_key = hashlib.sha256(
    ','.join(map(str, ranks)).encode('ascii')
).hexdigest()
return os.path.join(cache_path, f'{PG_DIST_CACHE_FILE_PREFIX}_{group_key}.pkl')

Both cache creation and memo lookup already use this helper, so this separates the demonstrated identities without adding a collective or creating overlong filenames. Update the min-rank-identity documentation and require existing min-only caches to be regenerated rather than blindly reused. The minimum-rank writer election itself can remain unchanged.

Preservation and acceptance — proposed: add a four-rank sequential DP-load/expert-DP-save regression with the fresh replicated metadata above. Cached distributions must equal their uncached counterparts, filenames must differ for different memberships, and every logical tensor must have exactly one main writer. Exercise a real save/load round-trip with integrity validation. Clear the process memo and repeat READ against caches created separately for both group families. Retain one file writer per group, both ignore_groups variants, at-most-once gathering for repeated identical identities, trivial-group/default-uncached behavior, and the existing READ-must-not-gather assertions and cached round-trip value assertions.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex

[svcnvidia-nemo-ci]

[CRITICAL Correctness] F1 — Every PP peer must join cache-capacity negotiation

Failure: initialize PP=2 HybridModel pattern M*|**, prefix caching enabled, a positive Mamba-cache budget, and no explicit Mamba-memory ratio. HybridModel selects stage-local layers, and from_model returns no recurrent config on the attention-only stage. The allocation guard consequently calls this PP MIN collective only on the recurrent stage, which waits indefinitely. Both stages have attention, so the attention-free-layout guard does not reject the configuration. Expected initialization completes on every stage. This is unresolved prior F1, comment 4280274952 in review 5529654466.

Correction: separate all-peer capacity negotiation from local recurrent allocation. Every PP stage must join in identical collective order when the cache option is enabled. Nonrecurrent stages must contribute a non-limiting capacity, neither divide by a zero footprint nor allocate recurrent buffers. Pass the negotiated minimum into local allocation. Broadening the current allocator guard alone is unsafe; no one-line patch is suggested.

Preservation: retain the repaired scratch reservation, hard errors when a recurrent stage cannot fit one durable slot, and synchronized minimum capacity when every stage has recurrent layers. PP=1 and disabled caching must retain ordinary behavior.

Acceptance — proposed, not executed: initialize both contexts for M*|** under a bounded distributed-test timeout. Assert completion, no Mamba allocator on the attention-only stage, and budget-respecting recurrent allocation. Retain the uneven-layer test's identical-capacity assertion, local insufficient-budget errors, and the remote-limiting-stage assertions, including the message identifying room for three local durable slots.

[svcnvidia-nemo-ci]

[CRITICAL Implementation] F2 — Do not commit reasoning that subsequent parsing can retract

Failure: use StreamingChatParser(lambda t: DeepSeekR1ReasoningParser.parse(t)); parse hello</thi, then finalize hello</think>world. The first call commits reasoning hello</thi; final reasoning is hello, which no longer starts with _reasoning_sent, so the fragment cannot be corrected. Expected concatenation is reasoning hello, content world. Opening-tag splits have the same reclassification problem. See the whole-output parser. Content-marker buffering below does not protect this reasoning branch. This is unresolved prior F2, comment 4280274959 in review 5529654466.

Correction: implement parser-aware stable emission, withholding incomplete reasoning markers and configured implicit boundaries before committing deltas. Emitted text must not later change channels or be retracted, and concatenation must equal final whole-output parsing. Also account for Nemotron's empty-content override: stripping marker suffixes alone cannot prevent that channel reclassification. Exact incremental implementation remains a design choice.

Preservation: unconfigured <tool_call> remains literal reasoning; configured markers end reasoning while remaining available to the downstream tool parser. An unfinished literal hello</thi must be flushed at finalization, not discarded. Retain ordinary content and stable tool IDs/names/JSON arguments.

Acceptance — proposed, not executed: test every split and character-at-a-time delivery of opening/closing tags and configured implicit boundaries, including Nemotron's enable_thinking=False and force_nonempty_content=True. Assert final channel concatenation equals whole-output parsing. Retain the literal-marker assertion and Qwen streaming assertions: exact Checking. content, one name/ID emission at index zero, JSON {"city":"SF"}, and tool_calls finish reason.

[svcnvidia-nemo-ci]

[CRITICAL Correctness] F3 — Send terminal replies for failed, unadmitted handoffs

Failure: submit a coordinator-routed handoff, then fail transfer submission or polling. This branch completes only the engine-local future exceptionally and continues. The scheduler discards that future, and no ordinary request record exists before finalization. No ENGINE_REPLY reaches the client: its future/stream hangs while routing state and pending load remain allocated. Expected behavior is one terminal FAILED response without blocking unrelated requests. This is unresolved prior F3, comment 4280274966 in review 5529654466.

Correction: add exactly-once terminal failure reporting for unadmitted handoffs on the MP coordinator, retaining the local exception for direct-future callers. Construct a serializable request/error: event serialization requires ContextErrorFactory support; an arbitrary RuntimeError error-event payload is insufficient. Use a supported error carrying the failure or register a handoff-specific error. The normal reply handler can then release routing state and pending load. Exact record construction is an implementation choice.

Preservation: client completion must not imply destinations are reusable. Keep quarantine for potentially active operations, exactly-once safe KV/SSM release, immediate MP failure notification, and admission only after all ranks finish.

Acceptance — proposed, not executed: extend the submission-failure test through mocked coordinator/client framing. Assert one FAILED reply, client resolution/stream termination, routing-map removal, and one pending-count decrement. Retain its direct RuntimeError and release list [[10,11,12]], plus peer-failure quarantine without release. A subsequent valid request must complete.

[svcnvidia-nemo-ci]

[CRITICAL Correctness] F4 — Reject impossible imports before FIFO insertion

Failure: with pool_size=4 (three usable blocks), block size four, and otherwise valid limits/metadata, submit a 16-token/four-prompt-block handoff. It cannot fit even in an empty pool, but this branch queues it as temporary backpressure. The FIFO drain stops at that head, indefinitely stalling later fitting handoffs. Ordinary permanent KV-capacity validation runs only after import. Expected behavior is a terminal capacity rejection, not indefinite FIFO occupancy. This is unresolved prior F4, comment 4280274976 in review 5529654466.

Correction: before FIFO insertion, distinguish impossible resident footprints from temporary occupancy. Validate prompt plus required resume-continuation storage against total usable capacity; at minimum a prompt exceeding allocator.pool_size - 1 is permanently impossible. Reject through a serializable terminal capacity response, not an uncaught scheduler exception or unresolved future. No destination allocation/transfer should start for a permanently rejected import; coordinate rejection with the push path so it does not leave unmatched operations. Exact failure-reporting integration remains an implementation choice.

Preservation: fitting requests must still defer during temporary shortages and start when capacity returns, preserving NCCL receive order. Cached-prefix blocks consume resident capacity but are not new allocations; retain their references/protection and suffix-only NIXL pulls. Do not trim NCCL's matched source push as though it were a NIXL pull.

Acceptance — proposed, not executed: submit the four-block example followed by a one-block prompt needing one continuation block. Assert the first gets a terminal error, never enters the FIFO, and posts no transfer; the second starts. Retain the temporary-shortage test's exactly-one pull after capacity returns, the cached-prefix test's suffix-only pull and predecessor hash, and the neighboring per-stage/per-rank NIXL trimming and unchanged NCCL source-push assertions.

[svcnvidia-nemo-ci]

[CRITICAL Correctness] F5 — Route aborts through deferred and pending handoff state

Failure: submit a streaming handoff while capacity is unavailable or transfer is pending, then close the stream. The client sends ABORT_REQUEST, and the coordinator forwards it. This lookup finds nothing because handoffs enter self.requests only during finalization. Cancellation is silently ignored; the canceled request can later import and run generation. Expected cancellation covers the entire request lifecycle. This is unresolved prior F5, comment 4280274981 in review 5529654466.

Correction: consult deferred/pending handoff state before the active-request lookup. Deferred cancellations must never start later; pending transfers must be marked canceled and prevented from entering decode. Complete the internal future and coordinator routing lifecycle exactly once. Retain destination ownership until transport completion is safe: cancellation does not permit recycling KV/SSM storage, and posted NCCL operations must preserve matched ordering. Exact transport-aware cancellation implementation requires lifecycle changes rather than a one-line lookup patch.

Preservation: retain active-request stop behavior, successful uncanceled handoffs, exactly-once safe release, and quarantine of operations that may still access buffers.

Acceptance — proposed, not executed: test abort before capacity admission, during a nonterminal transfer, and after ordinary admission. Restoring capacity after deferred cancellation must cause no transfer or forward. Completing pending handles must add no decode row and release safe storage exactly once; coordinator routing must clear. Retain the client stream-close assertions, the unsafe-reset no-release assertion, and normal active-request completion.

[svcnemo-autobot]

Sync review dispositions (head 2e5f599)

This covers reviews 5516528726 (7b81c45), 5529654466 (a6d4c20) and 5531673257 (2e5f599). Each cited range was traced with git blame on the head and compared with main 19deef6 and the dev base c3010d6.

None of the 62 cited ranges contains a line from the merge commit 4ade8e7. Lines from the repair commit appear only in the F1 anchor (dynamic_context.py 1775-1800). There, the repair restores main's extraction-scratch reservation, which the conflict resolution had dropped. _allocate_mamba_cache on the head now matches main.

Finding	Origin	Disposition
F1 PP cache-capacity negotiation	The PP MIN all-reduce in _allocate_mamba_cache comes from main 731b791 (#6584). The stage-local allocation guard is shared code (c4bffde) and is identical on main, so main 19deef6 has the same code path.	Fix on main
F2 streaming reasoning retraction	openai_streaming.py is identical to main; the cited lines come from main ad3ecd9.	Fix on main
F3, F4, F5, F7, F8 handoff, abort and prompt handling	inference_state_handoff.py, inference_client.py and handlers.py are identical to main (cac6b7b). The dynamic_engine.py anchor is shared code plus cac6b7b.	Fix on main
F6 logit_dtype position	Main 19deef6 itself places logit_dtype after fp16_lm_cross_entropy in both GPTModel and HybridModel (897649d). The head matches main; dev has no such parameter.	Fix on main
F9 sock argument	Fixed in the repair commit.	Closed
F10-F12 GTP replica id, MIMO dp_cp group, checkpoint topology	The cited code is shared by both branches; main cfb8e46 enables GTP in the MIMO example. The dev-only lines inside these ranges (a2a1c89 metadata guard; 68447ee signature and default) do not carry the defects.	Fix on main
F13 pg_dist cache key	exchange_utils.py is identical to main (c95b0da).	Fix on main

On the RESET point: agreed. Compared with the dev base, these behaviours arrive with this sync. The sync contract is to import main's 100 commits unchanged and repair only merge or compatibility breakage. Fixing main-owned code only on dev would make the two branches diverge and conflict on every later slice. Recommended route: land the fixes on main, and they will reach dev in a later slice. If maintainers want a specific finding fixed in this PR instead, name it and it will be added to the single repair commit.

DCO: 3 of the 102 commits have no Signed-off-by. All three are imported main bot commits by github-actions[bot]: cab7116 and 15c83d2 ("chore: rotate oncall schedule") and 5f9b2bb ("Update copy-pr-bot.yaml"). The merge commit 4ade8e7 and the repair commit 2e5f599 are both signed off. Imported main commits cannot be rewritten. The merged sync PRs #8112 and #7906 show the same DCO result (DCO job SKIPPED, DCO app ACTION_REQUIRED), with Nemo_CICD_Test passing.

[FDecaYed]

LGTM for this sync. I reviewed the conflict resolutions and the latest socket integration fix, and found no remaining merge-specific blockers. The main-origin issues should be fixed on main and brought over in later syncs.

Nemo_CICD_Test passed. Please resolve the conflict with the latest dev and address the patch-coverage check before merging.

[svcnvidia-nemo-ci]

/ok to test 4a534df