cuda: keep the raw MMQ MoE tier off streamed routed experts - #747
Open
pmasala wants to merge 1 commit into
Open
Conversation
The tier resolves a pointer to the whole routed tensor of the layer, which under SSD streaming is not resident and makes the fallback cache materialise hundreds of MiB per layer, failing outright on a card without the room. Add the !g_ssd_streaming_mode guard that the aligned variant of the same tier already carries; the dense MMQ tiers keep working, and that is where its prefill win is. Closes antirez#732 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #732.
The tier needs a pointer to the whole routed tensor of a layer. Under
--ssd-streaming that tensor is not resident, so resolving it makes the fallback
cache build it: 528 MiB gate, 528 up, 1792 down, per routed layer. It tries to
cache 32.06 GiB, the alloc fails on a 12 GB card, and any real prompt then dies
with an illegal memory access.
The guard is !g_ssd_streaming_mode. The aligned IQ2 variant above already has
it, and the MXFP4 tier below also declines under streaming, so only the raw
variant was missing it. Failures here already fall back to the sorted-pairs
path.
Outside streaming nothing changes: the flag is set in one place, from the CLI
toggle, and the tier is prefill-only. The dense MMQ tiers stay on, which is
where the win is: prefill 9.14 / 16.08 / 17.61 t/s at 512 / 1024 / 2048 tokens,
and 21.40 t/s at 2048 with #734, about 2.1x the DS4_CUDA_MMQ=0 baseline.
Tested only on 12 GB: RTX 3500 Ada (sm_89), CUDA 13.0, WSL2, DeepSeek-V4-Flash
IQ2_XXS 80.8 GB. Bigger cards should show the same bug with a milder symptom.