spec: adaptive draft sizing (--spec-draft-adaptive) - port of LaurentZuijdwijk's controller

[aic0d3r]

PR: spec: adaptive draft sizing (--spec-draft-adaptive) - port of LaurentZuijdwijk's controller

aic0d3r:adaptive-verify (25a641c, rebased on strix-halo-vulkan df1671a) -> Nathanw1014:strix-halo-vulkan
Closes #11 (validation data there). Draft for review before opening.

Controller design and constants are LaurentZuijdwijk/llama.cpp b10641 - full credit for the idea. The censoring-aware EMA (full accepts probe additively, partial accepts average the exact stopping point), the constants (alpha 0.25 / probe 1.0 / init 2.0), the n_min semantics, and the content-class framing are all from his fork. This PR is the port onto strix-halo-vulkan (rebased on current tip df1671a, one commit, ~75 lines in common/ only, no backend changes) plus fresh validation on the rebased base.

Mechanism

Per-seq EMA of accepted-tokens-per-draft, censoring-aware:

• partial accept -> EMA of the exact stopping point (backing off is averaged)
• full accept -> only proves acceptance >= n_drafted, so probe upward additively instead of averaging a ratcheting-down bound

Each draft is sized clamp(round(EMA), n_min, n_max) instead of always n_max. Defaults off; two new flags (--spec-draft-adaptive, --spec-draft-n-min), wired into the dflash and mtp draft loops.

Validation on the REBASED base (all fresh, this week)

PPL (spec-independent): wikitext-2, 580x512, Qwen3.8-27B Q5_K_XL-v2:

• fa on, b4096: 7.0801 +/- 0.04626
• fa off, b1024: 7.0838 +/- 0.04628
• delta 0.0037, inside combined error bars; matches the pre-rebase values (7.0819 / 7.0826). Clean across kernel paths.

Greedy determinism (honest revision of the earlier "bit-identical" claim):

• no-spec baseline: byte-identical across independent boots (same sha256 across two fresh servers, temp 0, seed 42, first request per boot).
• spec-on arms (fixed n4 AND adaptive n3-7): each diverges from no-spec at temp 0 (divergence starts inside reasoning; fixed differs at ~char 679, adaptive at ~char 378, n=1 sample per arm). This is the known GPU verify-batch reduction class on this backend (batch composition flips near-tie argmax) - it affects fixed n4 identically, so it is a property of speculative decoding on Vulkan/FA here, not of the controller. The pre-rebase byte-identity result (issue qwen4exp: grouped-union sparse prefill flash attention — fixes the pp depth cliff #11) is not reproducible on the rebased base (the windowed kv-cache get_prev_tokens port landed in between) - reported as superseded rather than silently dropped.

Throughput (content-class ratchet, from issue #11, same-binary A/B, every cell measured):

content class	fixed n4	adaptive n3-7
arithmetic/patterned emission	41.8	51.5	+23%
agent tool-call scan loops (JSON)	38.5	48.3 @ 93.8% acc	+25%
file edit re-emission	35.1	36.7 @ 90.3% acc	+4.5%
JSON record generation	34.2	34.3	par
novel code @ 28k fill	23.9	22.7	-5%

Not a universal win - it drafts what the acceptance histogram supports. Default recommendation stays fixed n4 for prose-heavy work; adaptive for agent/emission-heavy sessions.

In-harness (issue #11): three full game-build runs (low/med/high effort) with adaptive on, try-1 success each under a runtime smoke gate, decode aggregate 18-26 t/s.

Relation to adaptive-mtp-port

Complementary, not competing: that branch is Stew's independent controller (hysteresis climb-counter with a rise-cost table, MTP-scoped, with tests); this one is the EMA design, draft-type-agnostic (dflash path included - the Qwen agent stack here runs dflash). Both independently land on the same content-class behavior; happy to run a controller-vs-controller A/B on the same binary if useful, or to rebase the dflash wiring onto that branch instead of carrying two controllers.

Honest limits

• Loses 3-5% when acceptance genuinely decays (novel code/prose) - tracks fixed n4 minus probe overhead.
• Constants are Laurent's, not re-tuned here. Validated on one machine, one model family, dflash+mtp paths.
• Temp-0 reproducibility: see determinism section - spec decode on this backend diverges from no-spec at temp 0 regardless of controller; adaptive adds no NEW divergence mechanism (draft sizing only changes verify batch composition, same as changing n_max).

[aic0d3r]

First end-to-end agent-build runs on this branch — receipts from a 4-cell N-test (Qwen3.8-27B UD-Q4_K_XL + DFlash2 Q4_K_M, --spec-draft-adaptive --spec-draft-n-min 3 --spec-draft-n-max 7, Sharp v22.4.0, medium effort, public game-contract harness neon-ladder):

• 4/4 try-1 server-stable runs (files + runtime-soak clean every time) — zero spec-decode anomalies across ~165k generated tokens
• Decode 18.0 t/s aggregate on mixed agent traffic (peaks 24.4), prefill 234-245 t/s — parity-or-slightly-ahead of fixed n4 cells on the same engine (17.5-17.6 t/s decode)
• Acceptance 65.3% on agent content, in line with the content-class table in qwen4exp: grouped-union sparse prefill flash attention — fixes the pp depth cliff #11
• Build quality 3/4 clean on human playtest (the one failure was ball-physics bugs in the generated game — model roll, not the PR: the controller is greedy-lossless by construction and validated bit-identical in qwen4exp: grouped-union sparse prefill flash attention — fixes the pp depth cliff #11's data)

Happy to run any cell you'd want to see before merge.

[Nathanw1014]

Thanks for the port and the careful validation. I'm not going to merge this here, for two reasons. First, development is moving to the halo-box community fork (https://github.com/halo-box/strix-llama.cpp), which has already merged this controller. Second, when we tested it there we found a problem with several requests in flight (-np 2): the per-sequence draft widths drift apart, the batch gets split, and acceptance drops from about 0.99 to 0.56-0.66 with no error. There's also a smaller issue: a draft cut short by p_min counts as a partial accept and pulls the EMA down. A fix for the first one is going to halo-box. Closing here, thanks again.