tests: measure batched-verify greedy divergence rate, not just worst gap

[Flor1an-B]

Motivation

ds4.c documents that the non-quality Metal speculative path "may pick a
different greedy token when batched reductions perturb nearly tied logits".
The --mtp-verify-depth / --dspark-verify-depth regressions already
teacher-force every committed token through ordinary decode, but they keep
only the worst argmax gap and gate on gap <= 2.0 (a catastrophic
mis-commit). They never report how often the batched verifier commits a
non-argmax token, nor whether those divergences are harmless near-ties or
material greedy changes — the characterization that decides whether that
compromise is worth closing with an exact verifier.

Change (test-only, no release-path change)

• Extend the shared teacher-forcing helper to accumulate a divergence rate
  (committed token != autoregressive argmax) and a severity histogram
  alongside the existing worst_gap. Both verify-depth tests print it; the
  gated worst_gap <= 2.0 invariant is unchanged.
• Add a diagnostic --verify-divergence mode (no pass/fail gate) that
  points the same instrument at an open-ended, higher-entropy prompt where
  nearly tied logits actually occur — the confident copy fixture the
  regressions use structurally yields ~0 divergence. Prompt and sample size
  are overridable via DS4_TEST_DIVERGENCE_PROMPT / DS4_TEST_DIVERGENCE_MAXGEN.

Testing

Apple M5 Max, Metal backend, DeepSeek-V4-Flash-DSpark-support.gguf (stages=3):

make ds4_test          # -Wall -Wextra, 0 warnings
./ds4_test --server    # OK

DS4_TEST_DSPARK=... ./ds4_test --dspark-verify-depth
  nspec=128  max_chunk=6  divergence=0/128  (0.00%)  worst_gap=0.000

DS4_TEST_DSPARK=... ./ds4_test --verify-divergence
  open-ended  256 tok  max_chunk=3  divergence=0/256  (0.00%)  worst_gap=0.000
  long-form  1024 tok  max_chunk=4  divergence=0/1024 (0.00%)  worst_gap=0.000

Measured result: no batched-verify greedy divergence over 1408
teacher-forced positions on this config, across confident and higher-entropy
generation, with the multi-row verify path exercised (max_chunk up to 6).
On this hardware/model the documented divergence appears latent — a useful
"measure before you build" data point for anyone weighing an exact Metal
verifier. The --verify-divergence diagnostic is the tool to extend that
sample to other models, prompts, and lengths.

🤖 Generated with Claude Code

[Flor1an-B]

Ran the same instrument on the legacy MTP path (DeepSeek-V4-Flash-MTP-Q4K-Q8_0-F32.gguf, draft=4), Apple M5 Max, Metal. This is the interesting case — the two speculative paths behave differently:

Path	copy fixture	open-ended 1024 tok
DSpark	0/128	0/1024
MTP	0/206	24/1024 (2.34%)

MTP, open-ended (--verify-divergence, DS4_TEST_DIVERGENCE_MAXGEN=1024):

verify-divergence backend=mtp nspec=1024 max_chunk=5
divergence=24/1024 (2.34%) mean_gap=0.179 worst_gap=1.010 at=24
gap_hist[<=0.1:10 <=0.5:12 <=2.0:2 >2.0:0]

Two takeaways:

1. The instrument detects real divergence, not just reports zero. On the MTP
   batched multi-row verify, 2.34% of committed tokens differ from the exact
   autoregressive argmax under higher-entropy generation — 10 near-ties (≤0.1),
   12 small (0.1–0.5), 2 moderate (0.5–1.0). So the confident-fixture 0s are a
   property of the stimulus, not the instrument.

2. The existing worst_gap ≤ 2.0 gate misses all of it (worst here is 1.010).
   That is exactly the gap this PR fills: turning "may diverge" into a tracked
   rate + severity histogram surfaces divergence the worst-gap assertion cannot.

So the compromise ds4.c documents is benign on DSpark but genuinely present on
MTP at ~2.3% of tokens (all sub-catastrophic). Whether that warrants an exact
Metal verifier is now a measured question per path rather than an assumption.

[Flor1an-B]

Re-ran on current main (84cc882) after 0e89a0e "dspark: commit accepted verifier state directly" landed. The picture moved — and in a way that's directly relevant to that change:

path	before 0e89a0e (b030961)	current main (84cc882)
MTP	24/1024 (2.34%)	24/1024 (2.34%) — unchanged
DSpark	0/1024	31/1024 (3.03%)

DSpark, current main (--verify-divergence, 1024 tok, same prompt):

divergence=31/1024 (3.03%) mean_gap=0.238 worst_gap=1.363 at=129
gap_hist[<=0.1:12 <=0.5:15 <=2.0:4 >2.0:0]

DSpark was byte-exact-argmax before because it replayed accepted tokens through ordinary decode; committing the verified accept state directly (the speed win) now lets the batched verifier's nearly-tied-logit picks reach the output. So the ~3% is the measured greedy-divergence cost of the direct-commit default — all sub-catastrophic (worst 1.363), and none of it trips the existing worst_gap ≤ 2.0 acceptance-depth check. That is the gap this instrument fills: the rate + histogram surface exactly this, whereas the worst-gap gate reports 0.

Not arguing the tradeoff is wrong — direct-commit is a real speedup — just that it's now a measurable quality delta rather than an invisible one.