[Feature]: MTP drafts in pipelined batch groups (2-GPU layer split, parallel 2)

What do you want to do?

Run two agents on a 2-GPU layer split and have each of them keep its MTP drafts.

Rig: 2x RTX 3090, layer split 0-24 / 25-47, IQ3_S, 262144 context, "parallel": 2, Strata 0.1.43.1 on Linux, i7-6850K, 64 GB.

With the default pipelined groups (2 groups of 1) both cards stay busy, but each slot gets one token per window. One request alone decodes at about 117 tok/s here, two together get 57-66 tok/s each. So windows per second roughly double and tokens per window drop from about 2.4 to 1, and the total stays flat.

I tried --batch-mtp --batch-groups 1. Two streams go to 65-67 tok/s each, but a 41K prompt arriving next to a decoding stream reads at about 1,400 tok/s instead of 3,100, so for agent traffic the pipelined default is still the better choice on this box.

Would drafts in the pipelined path be something you'd consider? From reading generate.cpp and verify.cpp, the blocker looks like batch_launch queueing the commit graph right behind the exec graph since every row is kept, which leaves no room for an accept/reject. A launch without the automatic commit, then commit_slot_prefixes per stage once the last stage reports, seems like the shape of it. The part I'm not sure about is keeping group A's window state alive on stage 0 while group B's window runs there. Maybe the two verifiers per stage that --pipeline-windows already sets up could carry that.

I've forked the repo and can try a PR if you can tell me whether that approach is sane, or test builds on this box. Numbers are from tools/strata_bench.py plus a small two-stream client, happy to share the runs.

I had a go at this in a fork and it works on my box.

I ended up not deferring the commit. The commit still goes right behind the window, but only keeps the slot's real token, and each group gets its own copy of the buffers a commit reads. An accepted proposal then gets a small one-row commit graph queued on every stage before that slot's next window. No second verifier.

2x 3090, IQ3_S, 262144, "parallel": 2, warm, median of 5 runs of 2 x 1200 tokens:

	tok/s per stream
stock pipelined groups	51
1 proposal per window	73
3 proposals, chain cut at `--spec-min-p`	87

tools/batch_test.py says IDENTICAL for both slots with 1 and 3 proposals, one request alone is unchanged, and a 41K prompt next to a decoding stream still reads at about 3,050 tok/s.

Branch: https://github.com/davidbhoward/strata-engine/tree/pipelined-slot-drafts. It is opt-in (STRATA_PIPE_MTP=1..3 with --batch-mtp --batch 2 --batch-groups 2) and only handles groups of one slot so far. I can open a PR if you want it, or rework it to fit however you'd rather structure this.

The branch also has a one-line pick_slot change for #1604. With groups of one slot the picker ignored what the slots hold, so a request coming back from the solo path was admitted over the other conversation, and that one's next turn read 243K tokens again (59 s here). With the change it reuses 243,496 of 243,519.

Same shape on a Volta pair — and the serial arm is worse here than on the
cards in docs/BATCHING.md. Happy to act as a V100 test box for your branch.

Rig: 2x Tesla V100-PCIE-32GB (sm_70), Xeon E5-2673 v3 x2 (24C/48T), 125 GiB
RAM, Linux, CUDA 12 source build. Qwen3.8-Flash-Next GSQ-RCO IQ3_XXS native
pack, 262144 context, --kv int8 --kv-resident 32768, --layer-split auto,
--spec 4 --spec-min-p 0.5, STRATA_SM70_TABLE=1 (opt-in Volta decode kernels,
+7% decode measured on this box). Expert cache 24,167/24,576 slots = 98.3%
resident in VRAM, pcie_share 0.0 on every request.

Burst of 4 concurrent streams x 800 tokens, temperature 0, quiet box (cron
paused), aggregate tok/s over the streams:

arm	aggregate	vs baseline
v0.1.41, --batch 4, groups 2 (pipelined)	139.7	—
v0.1.43.1, STRATA_BATCH_MTP_SERIAL=1 (groups 1)	103.2	-26%
v0.1.43.1, default pipelined (groups 2)	138.8	-0.6%

The serial arm's drafters genuinely ran (--batch-mtp: 4 slot drafters on CUDA1, MTP proposals accepted 71.5%), so this is not a broken build — it
is the pipeline loss on two cards, and it is larger than the -21.9% the docs
report for 2x R9700 with all experts in VRAM. It confirms your rule on Volta,
and suggests the gap may widen as the expert tier gets closer to fully
resident, which is the case on 32 GB cards.

Two asks:

Would you consider the pipelined-slot-drafts shape from the fork above as
the way to close this, rather than tuning the serial path? Our box has no
headroom for serial windows at 4 slots.
If davidbhoward's STRATA_PIPE_MTP=1..3 branch can be built for sm_70
(CUDA 12, compute capability 7.0), I will run it here against the same
baseline and report per-stream tok/s, aggregate, prompt-read rate beside a
decoding stream, and batch_test.py identity — same-day runs, same binary,
same prompts.

Not tested on this box: more than two stages, sampled decoding, unequal card
pairs, and whether the #1413 drafter-construction waste still happens on
0.1.43.1 (the engine printed the batch-mtp-off line for us, so I have not
seen the silent case).

Thanks for the update on pipelined-slot-drafts. I can help validate this on a dual Tesla V100 32GB (sm_70) system.
I ran a 14-arm A/B/A tuning sweep on our current production baseline. All tested flag-only variants were rejected. On our actual 2-GPU layer-split workload, serial batch MTP also regressed substantially:
v0.1.41 pipelined groups, 4-stream burst: 139.7 tok/s aggregate
v0.1.43.1 serial batch MTP: 103.2 tok/s aggregate (-26.1%)
v0.1.43.1 default pipelined mode: 138.8 tok/s aggregate (-0.6%)
Our expert residency is already 98.3%, with a 262K context and STRATA_SM70_TABLE=1. This makes preserving pipeline overlap particularly important for our workload.
The proposed per-slot draft/commit path looks promising because it may restore speculative decoding without giving up pipelined execution. I'd be happy to build the branch for sm_70, run correctness tests, and report comparative throughput and TTFT results.
One limitation on our side is that production currently uses 4 slots across 2 groups, while the branch currently supports one slot per group (--batch 2 --batch-groups 2). I'll start with the supported 2-slot configuration and keep the results separate from our production baseline.
Could you point me to the preferred commit or branch revision for testing, and let me know whether there are any known sm_70 limitations or specific correctness cases you would like me to cover?
I can report results for both the draft-enabled path and the stock pipelined baseline. I'll test the new top-k split kernel separately so that its effect can be isolated from the MTP changes.
Thanks for working on this. A reproducible Volta result would be useful for determining whether this approach generalizes beyond Ampere.

@davidbhoward

Another data point, from AMD: 2x Radeon AI PRO R9700 (gfx1201), Linux, ROCm 7.2.4, Flash-Next IQ3_S, --layer-split 26, all experts resident.

I built the same idea a heavier way: --lanes 2 (https://github.com/justincinmd/Strata/tree/pr/dual-lanes, 6 commits on v0.1.43). Each lane gets its own verifier on each stage and a full MTP drafter. With both lanes decoding, GPU 0 runs one lane's window while GPU 1 runs the other's window and then its draft. Each lane's output equals the solo path, greedy and seeded sampled (tools/batch_test.py). With lanes off, the tokens equal stock v0.1.43.

Through the server, 512 tokens, temperature 0.7, against --batch 2 on the same binary:

Requests	`--lanes 2` tok/s total	`--batch 2` tok/s total
1	84.7	89.1
2	113.8	87.4
4	111.9	84.7

While both lanes generate, the engine runs about 170 tok/s in total. The cost is VRAM: 1.3 GB on GPU 0 and 1.4 GB on GPU 1 for the second verifiers and drafters.

@davidbhoward's design gets drafts into the pipelined groups without a second verifier, so it costs much less VRAM and keeps the group structure. That looks like the better base for upstream. I can run pipelined-slot-drafts on the R9700s (HIP) with the same prompts and report per-stream tok/s, prompt reads beside a decoding stream and batch_test.py identity, so the approach is covered on gfx1201 as well as Ampere and Volta.

Lanes have only been run on HIP. The CUDA and SYCL builds compile with no new diagnostics, but nothing ran on NVIDIA. @llollaalloll, if your V100 pair has room after the slot-drafts tests, a --lanes 2 run there would tell us whether the two designs compare the same way on CUDA.

Correction: the slower start was my build. I configured pipelined-slot-drafts by hand without -DSTRATA_PREFILL_MMQ=ON. Rebuilt with it, rerun in one session:

	`--lanes 2`	`PIPE_MTP=3`
time to first token, 1 request	0.64 s	0.62 s
2 requests, total tok/s	114.5	117.3
4 requests, total tok/s	110.0	112.3
46K read beside a stream (`READ_SHARE=1`), longest pause	7.8 s	7.7 s

The implementation looks good and works on the R9700s (gfx1201): output stays IDENTICAL to solo, greedy and sampled.

Data point from RDNA2 with the experts not resident (RAM offload), which none of the rigs above have.

Rig / config: 2x RX 6900 XT (gfx1030, 16 GB each), Ryzen 5 5600X, 128 GB DDR4, ROCm 10.0. Flash-Next GSQ-RCO IQ3_S, 262144 context, --kv int8 --kv-resident 32768, layer split, --spec 3 --spec-min-p 0.7, "parallel": 2, --batch-groups 2 --batch-mtp. I applied the back-fill version (e7c4b03) on top of 0.1.44 with git am -3; it went in cleanly.

Correctness (tools/batch_test.py, 2 prompts, 256 new tokens, fixed expert cache): both slots IDENTICAL to the solo runs for no drafts, PIPE_MTP=1 and PIPE_MTP=2 (6/6), and again with --pipeline-windows 2 kept on (4/4). No stalls.

Speed (server, production config, 2 runs per arm, 2 x 600 tokens started together):

	2 streams, total tok/s	single stream, 32K decode
pipelined groups, no slot drafts	51.9 (median)	72.5–72.7
`PIPE_MTP=1`	55.0 (+6%)	75.9 / 76.4
`PIPE_MTP=2` (earlier run, without the PW fix below)	53.9 / 55.3	65.7

Acceptance 65–71%, 1.65–1.71 tokens per window. PIPE_MTP=1 beats =2 here.

One interaction worth knowing: with --batch-mtp, --pipeline-windows is switched off for the solo path too, so our single stream first dropped from ~76 to 65.7 (−14%). A small local patch that keeps --pipeline-windows on for the solo path beside the batch slots removes that cost (the 75.9 / 76.4 above). Also, on gfx1030 mtp_first is off by default (decode_defaults_arch_ok() is gfx12 only), so the drafter already sits on card B here, and the slot-drafter placement only matters on gfx12.

Why the gain is smaller than on the R9700s (+34%): our experts do not fit in VRAM, so every extra row in a window brings another batch of CPU-served experts. With a fixed cache, the group step went 18.2 → 34.8 → 44.4 ms for 0 / 1 / 2 draft rows. Most of the draft gain is spent there.

It has been in our production since Oct 15 with no stalls, and a 65K needle test passes 8/8. These are preliminary numbers: 2 runs per arm, not 10+ interleaved pairs.

Developed with an AI coding assistant; every number above was measured on 2x RX 6900 XT (gfx1030, PCIe 4.0 x8 each) / Ryzen 5 5600X, ROCm 10.0.

A 4-stage data point for this thread, #2973 and its PR #2985. Our fork on 0.1.44.1 (plus our other patches), not upstream code; no same-day comparison with PIPE_MTP, --lanes or #2985's STRATA_PIPED_MTP.

4x RTX 3090, CUDA 13.0, Qwen3.8-Flash-Next UD-IQ4_XS, --layer-split 12,25,37, --batch 16 (4 groups), --spec 4 --spec-min-p 0.5, int8 K/V, all experts resident. HTTP, greedy, 512 new tokens, ~170 / ~1.2K-token prompts, 2 runs per arm in ABBA order, 2 requests per user per run (4 for the one-user line).

STRATA_SPEC_LANES=2..4 (opt-in): when 2-4 requests decode, each alone on the first slot of its group, each runs as its own solo MTP stream with its own verifier per stage; the windows interleave over the cards. Admissions and prompt reads drain the lanes; more than 4 requests, or penalties / logit bias / images, stay on the batch path.

Short / long, tok/s:

users	per user p50, no drafts	per user p50, lanes	total, no drafts -> lanes
2	83.6 / 81.8	121.2 / 112.8 (+45 / +38 %)	157 -> 221 / 148 -> 196 (+41 / +32 %)
3	82.1 / 79.2	113.7 / 103.3 (+38 / +30 %)	231 -> 302 / 208 -> 265 (+31 / +28 %)
4	74.8 / 70.8	94.2 / 84.8 (+26 / +20 %)	278 -> 326 / 244 -> 275 (+17 / +13 %)
5, 6			unchanged (±0.5 %)

Row 2: one build, switch off/on, --kv-resident 32768. Rows 3-4: against our previous build (2 lanes, so the batch path at 3+ users), 28672.

One request: STRATA_SOLO_PIPELINE=1 (on by default where STRATA_SPEC_LANES=2..4 can run, =0 turns it off) runs the existing --pipeline-windows 2 path in a --batch server, like Vop3p's local patch above: total +12.8 / +10.4 % vs serial solo spec, measured with that switch alone on the build before the 4-lane one.

Each lane slot is token-identical to serial solo spec (2/3/4 lanes: 10/10, 12/12, 12/12), also across 1-2-3-4-2-1 switches, once with the shared buffers poisoned and forced rollbacks on the pipelined one-user windows; 16 users: 32/32 identical.

Cost against no lanes, the solo pipeline's second verifier included: ~450 MiB VRAM on CUDA0-2 and ~730 MiB on CUDA3, the drafter card (added up from boot logs at --kv-resident 32768 and 28672). Lane 0 reuses the main verifier's buffers, lane 1 the solo pipeline's second verifier's, lanes 2-3 own theirs; each lane drafter shares the solo drafter's weights and adds 50 MiB of its own K/V state and buffers. The room comes out of the expert cache; all experts stayed resident here, which a rig with experts in RAM would not get for free. Host RSS +593 MiB vs our 2-lane config (no-lanes RSS not measured). 4 lanes need --kv-resident 28672 on 24 GB cards; 32768 fits 3.

Not tested: 0.1.45, 2-stage splits, HIP/SYCL/Metal, sampled output, other models.

Patches, on top of our other 0.1.44.1 patches (they do not apply to a stock tree and are not rebased onto 0.1.45); after our v0.1.44.1 and v0.1.44.1-ll series, in this order: ll3/ovkv, ll3/lanes, ll3/all (the interplay fix), ll4. https://github.com/havok2-htwo/StrataQ38FN/tree/main/linux/patches/v0.1.44.1-ll3 and https://github.com/havok2-htwo/StrataQ38FN/tree/main/linux/patches/v0.1.44.1-ll4. Raw data: entries 82 and 84 in https://github.com/havok2-htwo/StrataQ38FN/blob/main/linux/TESTLOG.md, https://github.com/havok2-htwo/StrataQ38FN/tree/main/linux/results/lanes and https://github.com/havok2-htwo/StrataQ38FN/tree/main/linux/results/lanes4.

#2985 lists its cost as 25-51 MiB of saved rows per stage (4 slots, N = 1..3), --batch-mtp's slot drafters on the drafter card and the graphs per row count: by those numbers less than the lanes on the first three cards; on the drafter card I do not know, and I have not measured it here. It needs --batch-mtp, which the lanes refuse, so the two do not stack. I can run #2985 on this box at "parallel": 4 on 0.1.45, against the same build without the variable and with the prompts above, and post the numbers here. On stock 0.1.45 here STRATA_PIPED_CONCURRENT=1 turns itself off beside the in-place GGUF expert source, so that run would be without it.

Data point for the `helper expert caches` cell that the PR's "Not run" leaves empty

Rig: 2x RTX 2080Ti 22GB (SM75), 2-GPU layer split (24/24), Flash-Next IQ3_XXS, 262144 ctx, int8 KV, --kv-resident 65536, --remote-expert-opt, --spec 4 --spec-min-p 0.5, parallel 2 (--batch 2 --batch-groups auto = 2 groups of 1 slot), 96K context, greedy, 256 output tokens.

Same binary for every arm (PR head 2b7ffada, built for sm_75), only the env/flag changes. --batch-mtp is accepted on this build (no "is off" line) and simply does not draft on the pipelined path. The 2-way numbers are two concurrent streams over the same pre-filled 96K text (warm, TTFT 0.1-1.0 s), so the run measures the batch window itself; cold = two different 96K prompts.

arm	2-way decode median	2-way sum	single stream	2x96K cold TTFT (median / sum)
default (no `--batch-mtp`)	41.93	78.62	71.90	102.9 / 154.0 s
`--batch-mtp`, `PIPED_MTP` off	39.00	73.26	74.19	102.8 / 153.9 s
`STRATA_PIPED_MTP=1`	43.26	80.76 (+2.7%)	79.05	102.9 / 153.9 s
`STRATA_PIPED_MTP=2`	43.26	80.84 (+2.8%)	82.83 (+15%)	103.0 / 154.2 s

On this shape the lever is worth +2.7% aggregate (and +10-15% single stream): the "experts not resident" band (+6% above), not the +10-33% measured with experts resident. Expected, since our experts live in the other GPU's helper cache and each draft row drags more expert traffic.
--batch-mtp without PIPED_MTP costs 6.8% here; a config of ours carried it for a while for nothing.
Prefill is untouched, as designed: two cold 96K reads still take 51 s + 103 s, and the peer stream decodes at 4.4 tok/s while a read runs.
Not verified here: token-identical output across arms. My identity run used a different prompt salt per arm, so it measures nothing; the 4-slot exactness numbers in the PR cover resident experts, not this shape.

Why post it: this is the helper expert caches cell the PR leaves empty. On our shape the switch works (it is a pure gain, no single-stream penalty) but it does not move our multi-user problem, which is still "reads are serialized and a read stops the slots" (asks 2 and 3 in #2275). Anyone with experts off-card should expect the +3% band, not +30%.

Thanks for taking the time to report this.

We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to read properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.

To reopen it takes about two minutes:

Re-test on 0.2.0-beta. It may already be fixed. Write down what you see.
Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other].
Fill in the parts of the issue form: what happened, the Strata version, your graphics card, memory and system.
Comment /reopen. A check runs and either reopens the issue or lists what is still missing.

The rules are in CONTRIBUTING.md.

Navigation Menu

Repository navigation

[Feature]: MTP drafts in pipelined batch groups (2-GPU layer split, parallel 2) #2235

What do you want to do?

Data point for the `helper expert caches` cell that the PR's "Not run" leaves empty

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

[Feature]: MTP drafts in pipelined batch groups (2-GPU layer split, parallel 2) #2235

Description

What do you want to do?

Activity

davidbhoward commented on Oct 13, 2026

llollaalloll commented on Oct 13, 2026

llollaalloll commented on Oct 15, 2026

justincinmd commented on Oct 15, 2026

justincinmd commented on Oct 15, 2026

Vop3p commented on Oct 17, 2026

havok2-htwo commented on Oct 18, 2026

superniker commented on Oct 19, 2026

Data point for the helper expert caches cell that the PR's "Not run" leaves empty

Niko1221 commented on Oct 19, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Data point for the `helper expert caches` cell that the PR's "Not run" leaves empty