Repository navigation
fix(inference): AUT-2539 fail impossible KV handoffs on arrival and honor aborts - #8267
Closed
svcnemo-autobot wants to merge 3 commits into
Closed
svcnemo-autobot wants to merge 3 commits into
svcnemo-autobot wants to merge 3 commits into
Conversation
A decode KV handoff that exceeded a fixed limit (the KV pool or the maximum sequence length) waited at the head of the capacity queue forever, because capacity that can never appear blocked every later handoff behind it. An ABORT_REQUEST for a handoff that was still queued or transferring was dropped, because such a request has no engine entry until its import is admitted, so the client never got a reply. - Check each handoff against the limits add_request applies, through a shared _request_limit_errors helper, before it reserves cache state or joins the queue. A rejected handoff replies once through _fail_submission. - Route ABORT_REQUEST for a request with no entry to _abort_kv_handoff. A queued handoff replies at once; an import still transferring is marked cancelled and retired, released or quarantined, once every model-parallel rank reports its transfer terminal, then replies once. - On the NCCL push path the prefill sends regardless, and two-sided operations pair by post order per peer, so a rejected or aborted push handoff keeps its FIFO position as a discard. begin_discard_blocks posts matching receives into scratch buffers without needing cache capacity. Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
svcnvidia-nemo-ci
marked this pull request as draft
October 15, 2026 19:30
Contributor
|
This PR has been automatically converted to draft because all PRs must start as drafts. When you are ready for review, click Ready for Review to begin the review process. This will:
See the contribution guide for more details. |
svcnvidia-nemo-ci
added
the
Run full unit tests
Contributor
|
/ok to test 5dcd028 |
…stub has_admittable_kv_import now reads PendingKvImport.cancelled. The SimpleNamespace stand-in for a pending import in test_ready_handoff_uses_safe_no_overlap_admission_point lacked that field and raised AttributeError, so give it cancelled=False to match the dataclass default. Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
Contributor
|
/ok to test e638ce6 |
Bring in NVIDIA#8263, which seeds the CPU-initialized models in test_checkpoint_load_without_microbatch_calculator. This PR's testmon selection runs those 16 cases alone. Without the seed, the data-parallel ranks build different reference weights, and the H100 unit job hangs in the NCCL barrier of the next case's fixture. Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
Contributor
|
/ok to test a5b70b0 |
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
In disaggregated inference, a decode-side KV handoff that exceeded a fixed limit (the KV pool or the maximum sequence length) waited at the head of the capacity queue forever and blocked every later handoff. An
ABORT_REQUESTfor a handoff that was still queued or transferring was dropped, so the client never got a reply.What changed
dynamic_engine.py: a shared_request_limit_errorshelper checks each handoff against theadd_requestlimits before it reserves cache state or joins the queue. A rejected handoff replies once through_fail_submission.ABORT_REQUESTwith no engine entry goes to_abort_kv_handoff:transfer_backends/nccl.py): a rejected or aborted push handoff keeps its FIFO position as a discard.begin_discard_blocksposts matching receives into scratch buffers.test_disagg_handoff_lifecycle.pyandtest_nccl_transfer_backend.py, new cases intest_dynamic_engine.py, and one updated test intest_dynamic_engine_async_sched.py.ee6baa2e(test(checkpointing): seed CPU initialization in checkpoint load test #8263). It seeds the models intest_checkpoint_load_without_microbatch_calculator, which hung the H100 unit job one638ce6ain an NCCL barrier.Details
This fixes two review findings: impossible handoffs (F4) and aborts during a handoff (F5). A rejected or aborted handoff drains its transfer instead of cancelling it mid-flight.
flowchart LR A[Handoff arrives] --> B{Within request limits?} B -- no --> R[Reply failure once; NCCL push posts a discard receive] B -- yes --> Q[Capacity queue] Q -- abort --> X[Reply aborted once] Q --> T[Transferring] T -- abort --> C[Mark cancelled; retire after all MP ranks finish; reply once] T --> D[Admitted]Tested
sha256:7b4e22a3,tests/unit_tests/**/*.pyran on this head merged with the main tip (ce8ee4cb): 3424 passed, 142 skipped, 0 failed. A second pass ran 24 more tests, all passed. The exit code was 0, andtest_checkpoint_load_without_microbatch_calculatorfinished.