Repository navigation
--mtp-hnorm stream beats the default on sm_89 (IQ3_XXS): draft acceptance 82%->86%, decode +3.8%/+2.3% at 6K/70K, neutral at 120K, answers byte-identical #2227
Description
Summary. On an RTX 4090D (sm_89, Windows 11, 48 GB, PCIe Gen4 x4), Strata 0.1.43.1 with Qwen3.8-Flash-Next IQ3_XXS (native pack, 24,576/24,576 experts in VRAM) was measurably faster with --mtp-hnorm stream than with the default (pooled). The draft acceptance rate rises about 4 points and the decode follows; at 120K context the two are inside the noise band. Answers are unchanged (byte-identical on our greedy set).
Rig / flags (nothing else differs between arms).
- RTX 4090D 48 GB, Windows 11 (WDDM), Gen4 x4, 128 GB RAM, NVMe; engine 0.1.43.1 release build (
strata-windows-x64.zip, archs 75/86/89/120, CUDA 13.0). - Qwen3.8-Flash-Next GSQ-RCO IQ3_XXS (
--pack+--native+--ple-gguf),--ple-io direct,--embd-ggufBF16 embeddings. - Serving:
--spec 4 --spec-min-p 0.70 --mtp <rt>(shipped draft vocab, 106,299 ids)--prefill auto:32768 --pcie-frac 0.35 --kv int8 --kv-resident 32768 --prompt-cache 12 --conversation-cache-mib 32768 --conversation-cache-slots 8 --vram-reserve-mib 700. - Each arm is a config + engine restart; arms were interleaved (default → stream → default → stream) in one session because this box drifts ±4-6% across boots.
Method. One fixed filler prompt read to 6K / 70K tokens, then 5 greedy 384-token generations per arm (bench/strata-decode-probe.py style); the engine's own per-request line supplies generated in X ms (Y tok/s), drafts accepted A of B — the table shows the median of the 5 reps. 120K: the repo's own 3-needle prompt, 2 requests per pass, greedy.
Results.
| context | default (pooled) | --mtp-hnorm stream |
delta |
|---|---|---|---|
| 6K | 219.9 / 219.4 tok/s, accept 82% / 82% | 228.0 / 228.2, accept 86% / 86% | +3.7% / +4.0% |
| 70K | 203.0 / 203.3, accept 81% / 81% | 207.6 / 208.0, accept 85% / 85% | +2.3% / +2.3% |
| 120K needles | mean 220.2 (214.7, 225.6), accept ~89.5% | mean 220.2 (220.2, 226.6, 211.7, 222.4), accept ~88.5% | neutral |
The default arm is very reproducible across the interleave (219.9/219.4 and 203.0/203.3), so the 6K/70K gap is wider than its own band; at 120K the two overlap.
Quality. The draft only proposes and every token is still verified against the main model, so nothing should change — and nothing did on our checks: six greedy answers (Cantonese/English short, code, Cantonese/English long, translation) byte-identical between arms and against 0.1.43; 32K needle 3×3 = 9/9 and 120K needle 2×3 = 6/6 on both arms; a Cantonese temp-0.3 sanity prompt non-empty 6/6.
Why (a guess). stream is llama.cpp's qwen4exp MTP graph (one RMS per hyper-connection stream); the default pools the four into one (vLLM's qwen4_exp). This model's MTP head comes from the llama.cpp-side family, so the per-stream norm may match how the draft layer was trained.
Two extra data points (may save someone a run).
STRATA_MTP_FULL_HEAD=1(draft over the full 248,320-token vocabulary): acceptance 86% → 90% at 6K, but decode falls 228.0 → 223.2 (−2.1%). The bigger head costs more than the extra acceptances earn, so the 106,299 subset stays.STRATA_MTP_TOP2=1: at rejected drafts the runner-up was the token the model picked 0 of 14 times (rejected/runner-up by depth 6/0 4/0 4/0 0/0over ~137 windows) — the remaining rejections are not near-misses.- For completeness:
--lookup-chain 3measured ±0.05% here (219.8/202.9 vs 219.9/203.0); its counters show 1-4 of ~148 windows carrying a chain and 0-3 chained drafts accepted per request.
Suggestion. If other sm_89 (or NVIDIA) cards confirm stream ≥ pooled, it may be worth a line in the MTP docs — or even flipping the default for NVIDIA. Happy to re-run this on request with different flags or a specific protocol.
Activity