inureyes · Sep 20, 2026 · Sep 20, 2026 · Sep 20, 2026 · Sep 20, 2026 · Sep 20, 2026
diff --git a/docs/CONTINUOUS_BATCHING.md b/docs/CONTINUOUS_BATCHING.md
@@ -374,7 +374,13 @@ Supplying a drafter checkpoint (`--draft-model` on `mlxcel serve`,
 `--model-draft` on `mlxcel-server`; the spellings are cross-aliased) turns on
 server-side speculative decoding. The kind is auto-detected from the drafter's
 `config.json` (`--draft-kind mtp|dflash` overrides), and the draft block size
-K defaults per kind (4 for MTP, 16 for DFlash; `--draft-block-size`).
+K is resolved at startup (`--draft-block-size` and its two environment
+spellings override it): a width the drafter checkpoint declares for itself
+wins first, then a measured default for the running compute capability and the
+target's quantization mode (issue #1797; a DFlash drafter against an
+affine-quantized target on GB10 resolves to 4), then the flat per-kind
+constant, 4 for MTP and 16 for DFlash. The resolved width and which of those
+produced it are logged at startup on the `speculative=` line.
 
 Whether the singleton (B=1) speculative burst actually runs is decided by an
 adaptive policy (issues #333/#736), not by the flag alone. The first 4

diff --git a/docs/benchmark_results/data/draft-block-width-gb10-2026-09-20/affine-tf32-off.jsonl b/docs/benchmark_results/data/draft-block-width-gb10-2026-09-20/affine-tf32-off.jsonl
diff --git a/docs/benchmark_results/data/draft-block-width-gb10-2026-09-20/affine.jsonl b/docs/benchmark_results/data/draft-block-width-gb10-2026-09-20/affine.jsonl
diff --git a/docs/benchmark_results/data/draft-block-width-gb10-2026-09-20/harness/README.md b/docs/benchmark_results/data/draft-block-width-gb10-2026-09-20/harness/README.md
@@ -0,0 +1,32 @@
+# #1797 draft block width harness
+
+Reproduces `docs/benchmark_results/draft-block-width-default-gb10-2026-09-20.md`.
+
+```
+docs/benchmark_results/data/draft-block-width-gb10-2026-09-20/harness/run_session.sh \
+  /path/to/mlxcel-server
+```
+
+## Files
+
+| file | what it is |
+|---|---|
+| `run_session.sh` | the whole session: identity, both families, both summaries |
+| `identity.sh` | environment identity for one session |
+| `sweep_server_widths.py` | served sweep, one `mlxcel-server` per arm |
+| `summarize.py` | JSONL to the record's table, including the drift check |
+| `prompt_retry.txt` | the fixed prompt every arm sends |
+
+## Four things that are not stylistic
+
+**The binary is renamed before it runs.** `run_session.sh` copies it to `mlxcel1797-server`. The imported host gate blocks on a foreign inference process by `/proc/<pid>/comm`, matching `mlxcel` and `mlxcel-server`, which is what keeps a peer session's model off the GPU during a timed run. A server started under the stock name matches that list and gates itself forever.
+
+**`MLX_CUDA_ARCHITECTURES=121` is pinned for the build and for every server.** `build.rs` auto-detects `121a` on this host, and every earlier GB10 record here pins plain `121`. The suffix does not change the decode kernels: in the pinned MLX tree only the NVFP4 weight-quantization converter keys on `__CUDA_ARCH_SPECIFIC__`. The pin is for comparability with those records, and it happens to match the release configuration as of 2026-09-20.
+
+**The host gate is imported from `../../sdpa-plan-bucket-gb10-2026-09-12/harness/hostgate.py`, not copied.** It encodes three predicates that were each paid for with a lost measurement: contention means processes that can consume CPU now (match `/proc/<pid>/comm` exactly, drop state `T`, exclude the CI runner's permanent `RunnerService.js` and `Runner.Listener` daemons, never a `%cpu` threshold because `ps` reports a lifetime average); a foreign model process is a reason to wait; and the cumulative `NV_ERR_NO_MEMORY` count is the freeze precursor on this host and does not decay. A second copy would drift from the original.
+
+**Classic runs first and last.** A served width cannot be interleaved the way `scripts/bench_block_width.sh` interleaves the offline path, because `draft_block_size` is worker-owned and changing it means restarting the server. The two classic arms bracket the widths instead: ranges that overlap say the session did not drift under them, ranges that separate say every arm in between is suspect. `summarize.py` prints that verdict before the table.
+
+## Driver budget
+
+`sweep_server_widths.py` records the `NV_ERR_NO_MEMORY` delta per arm and refuses to start an arm whose projected cumulative count would pass `--nvrm-budget` (400 by default), projecting from the worst arm measured so far. The window and cumulative ceilings themselves live in `hostgate.py` (`NVRM_WINDOW_MAX`, `NVRM_TOTAL_MAX`). Runs completing is not evidence the errors are benign; the two hard freezes this host took in July 2026 had exactly that shape.
diff --git a/docs/benchmark_results/data/draft-block-width-gb10-2026-09-20/harness/identity.sh b/docs/benchmark_results/data/draft-block-width-gb10-2026-09-20/harness/identity.sh
@@ -0,0 +1,88 @@
+#!/usr/bin/env bash
+# Environment identity for one #1797 sweep session.
+#
+# Records what the previous GB10 records could not. Until 2026-09-20 no harness
+# in this tree wrote the kernel release or the NVIDIA driver version anywhere,
+# and on that day a kernel upgrade (6.17.0-1029-nvidia -> 7.0.0-1019-nvidia)
+# landed without its matching NVIDIA module. The GPU was dead on first boot and
+# the driver was reinstalled at 580.178.04 (from 580.173.02). Nothing in the
+# #1820 record or its harness could express that, which is why it had to be
+# passed along by hand. Both values are captured here so the next person
+# comparing across sessions has something to key on.
+#
+# The MLX pin, the toolchain, the effective CUDA architecture list and the
+# binary hash are here for the same reason: a record that says only "GB10" does
+# not say enough to know whether two numbers are comparable.
+#
+#   identity.sh <server-binary> [model dirs...]
+set -uo pipefail
+BIN=${1:-}
+shift || true
+REPO=$(git rev-parse --show-toplevel 2>/dev/null || pwd)
+
+printf 'date: %s\n' "$(date -Is)"
+printf 'host: %s\n' "$(uname -n)"
+printf 'kernel: %s\n' "$(uname -r)"
+printf 'arch: %s\n' "$(uname -m)"
+printf 'nvidia_driver: %s\n' \
+  "$(nvidia-smi --query-gpu=driver_version --format=csv,noheader 2>/dev/null | head -1)"
+printf 'gpu: %s\n' \
+  "$(nvidia-smi --query-gpu=name,compute_cap --format=csv,noheader 2>/dev/null | head -1)"
+# The architecture list the build was PINNED to, which is not what a default
+# build produces here: build.rs auto-detects and yields `121a`, while every
+# prior GB10 record pins plain `121`. The suffix does not change the decode
+# kernels (only the NVFP4 weight-quantization converter keys on
+# __CUDA_ARCH_SPECIFIC__); it is recorded so two sessions can be told apart.
+printf 'MLX_CUDA_ARCHITECTURES: %s\n' "${MLX_CUDA_ARCHITECTURES:-unset (auto-detected)}"
+# nvcc is not on the default PATH on this host; the CUDA install root is.
+NVCC=$(command -v nvcc || echo /usr/local/cuda/bin/nvcc)
+printf 'cuda_toolkit: %s\n' \
+  "$("$NVCC" --version 2>/dev/null | sed -n 's/.*release \([0-9.]*\).*/\1/p' | head -1)"
+printf 'mlx_pin: %s\n' \
+  "$(sed -n 's/^ *GIT_TAG \([0-9a-f]\{40\}\).*/\1/p' "$REPO/src/lib/mlx-cpp/CMakeLists.txt" | head -1)"
+# The rustc that BUILT the binary, which is not necessarily the one first on
+# PATH here: this repo pins a toolchain and the build exports it ahead of the
+# default. `BUILD_RUSTC` lets the caller state the one it used; both are
+# printed so a mismatch is visible rather than silently recorded as the wrong
+# one.
+printf 'rustc_on_path: %s\n' "$(rustc --version 2>/dev/null)"
+printf 'rustc_used_for_build: %s\n' \
+  "$(${BUILD_RUSTC:-rustc} --version 2>/dev/null)"
+printf 'git_commit: %s\n' "$(git -C "$REPO" rev-parse HEAD 2>/dev/null)"
+printf 'git_dirty: %s\n' \
+  "$([ -n "$(git -C "$REPO" status --porcelain 2>/dev/null)" ] && echo yes || echo no)"
+printf 'uptime: %s\n' "$(uptime -p 2>/dev/null)"
+printf 'mem_total_gib: %.1f\n' \
+  "$(awk '/MemTotal/ {print $2 / 1048576}' /proc/meminfo)"
+
+if [ -n "$BIN" ] && [ -x "$BIN" ]; then
+  printf 'binary: %s\n' "$BIN"
+  printf 'binary_sha256: %s\n' "$(sha256sum "$BIN" | cut -d' ' -f1)"
+  printf 'binary_mtime: %s\n' "$(date -Is -r "$BIN")"
+  # What the binary was actually compiled for is not readable from the file:
+  # the first bare numeric string in a 100 MB binary is not the architecture
+  # list. The binary prints it itself at startup, as
+  # `compiled for [121] (cubin)` on the arch summary line, and that log line is
+  # the evidence the record cites. `MLX_CUDA_ARCHITECTURES` above records what
+  # the build was asked for; the two are compared in the record.
+fi
+
+for d in "$@"; do
+  [ -d "$d" ] || continue
+  printf 'model %s: bytes=%s model_type=%s quantization=%s\n' \
+    "$d" \
+    "$(du -sb "$d" 2>/dev/null | cut -f1)" \
+    "$(python3 -c "import json,sys;c=json.load(open('$d/config.json'));print(c.get('model_type','?'))" 2>/dev/null)" \
+    "$(python3 -c "
+import json
+c = json.load(open('$d/config.json'))
+q = c.get('quantization')
+if q:
+    print(q.get('mode', 'affine (implicit)'))
+else:
+    qc = c.get('quantization_config') or {}
+    groups = qc.get('config_groups') or {}
+    fmts = sorted({g.get('format') for g in groups.values() if g.get('format')})
+    print(','.join(fmts) or qc.get('quant_method') or 'none')
+" 2>/dev/null)"
+done
diff --git a/docs/benchmark_results/data/draft-block-width-gb10-2026-09-20/harness/prompt_retry.txt b/docs/benchmark_results/data/draft-block-width-gb10-2026-09-20/harness/prompt_retry.txt
@@ -0,0 +1,28 @@
+"""Retry helpers with exponential backoff and jitter.
+
+Dependency-free so it can be vendored into small services that must not
+grow a third-party requirement just to retry an HTTP call.
+"""
+
+import functools
+import itertools
+import logging
+import random
+import time
+from dataclasses import dataclass
+
+logger = logging.getLogger(__name__)
+
+DEFAULT_BASE_DELAY = 0.1
+DEFAULT_MAX_DELAY = 30.0
+DEFAULT_MAX_ATTEMPTS = 5
+
+
+@dataclass(frozen=True)
+class BackoffPolicy:
+    base_delay: float = DEFAULT_BASE_DELAY
+    max_delay: float = DEFAULT_MAX_DELAY
+    max_attempts: int = DEFAULT_MAX_ATTEMPTS
+    jitter: bool = True
+
+    def delays(self):
diff --git a/docs/benchmark_results/data/draft-block-width-gb10-2026-09-20/harness/run_session.sh b/docs/benchmark_results/data/draft-block-width-gb10-2026-09-20/harness/run_session.sh
@@ -0,0 +1,64 @@
+#!/usr/bin/env bash
+# One #1797 measurement session: both families, both arms of the acceptance
+# criteria, one binary, one host identity.
+#
+#   run_session.sh <mlxcel-server binary> [outdir]
+#
+# Two things here are deliberate and easy to get wrong if this is rewritten.
+#
+# 1. The binary is COPIED to `mlxcel1797-server` before anything runs. The
+#    #1820 host gate blocks on a foreign inference process by `/proc/<pid>/comm`
+#    (`mlxcel`, `mlxcel-server`), which is what stops a peer session's model
+#    from sharing the GPU with a timed run. A server started under the stock
+#    name would match that list and gate itself forever. `/proc/comm` truncates
+#    at 15 characters, so `mlxcel1797-serv` is what the gate sees and it is
+#    unambiguously not the name a peer's server carries.
+#
+# 2. `MLX_CUDA_ARCHITECTURES=121` is exported for the build AND for every
+#    server process. build.rs auto-detects `121a` on this host, which is not
+#    what any earlier GB10 record used. The suffix does not change the decode
+#    kernels this harness measures (only the NVFP4 weight-quantization
+#    converter keys on __CUDA_ARCH_SPECIFIC__); it is pinned so these numbers
+#    sit alongside every earlier GB10 record's.
+set -uo pipefail
+BIN=${1:?usage: run_session.sh <mlxcel-server binary> [outdir]}
+D="$(cd "$(dirname "$0")" && pwd)"
+OUT=${2:-$(cd "$D/.." && pwd)}
+export MLX_CUDA_ARCHITECTURES=${MLX_CUDA_ARCHITECTURES:-121}
+# The pinned toolchain this repo builds with, so the identity records the rustc
+# that produced the binary rather than whichever one is first on PATH.
+export BUILD_RUSTC=${BUILD_RUSTC:-/home/inureyes/.rustup/toolchains/1.93.1-aarch64-unknown-linux-gnu/bin/rustc}
+
+# The renamed copy lives outside the repo: it is a 100 MB binary and the data
+# directory is committed.
+RUN_DIR=${RUN_BIN_DIR:-$(mktemp -d -t mlxcel1797.XXXXXX)}
+RUN_BIN="$RUN_DIR/mlxcel1797-server"
+/bin/cp -f "$BIN" "$RUN_BIN"
+chmod +x "$RUN_BIN"
+
+REPO=$(git -C "$D" rev-parse --show-toplevel)
+"$D/identity.sh" "$RUN_BIN" \
+  "$REPO/models/mlx/qwen3.5-4b-4bit" "$REPO/models/mlx/qwen3.5-4b-dflash" \
+  "$REPO/models/mlx/laguna-xs-2.1-nvfp4" "$REPO/models/mlx/laguna-xs-2.1-dflash" \
+  | tee "$OUT/identity.txt"
+
+# The affine family first: it is the one with a prior record to agree with, so
+# a disagreement there is a signal about the session rather than about NVFP4.
+# `default` is the arm the issue exists for (no flag at all); `16` is the
+# explicit-override control and `env16` the environment-override control.
+python3 "$D/sweep_server_widths.py" \
+  --server "$RUN_BIN" \
+  --target "$REPO/models/mlx/qwen3.5-4b-4bit" \
+  --drafter "$REPO/models/mlx/qwen3.5-4b-dflash" \
+  --widths default,2,3,4,5,6,7,8,16,env16 \
+  --n 3 --out "$OUT/affine.jsonl" --logdir "$OUT/logs-affine" --port 18797
+
+python3 "$D/sweep_server_widths.py" \
+  --server "$RUN_BIN" \
+  --target "$REPO/models/mlx/laguna-xs-2.1-nvfp4" \
+  --drafter "$REPO/models/mlx/laguna-xs-2.1-dflash" \
+  --widths default,2,3,4,5,6,7,8,16 \
+  --n 3 --out "$OUT/nvfp4.jsonl" --logdir "$OUT/logs-nvfp4" --port 18798
+
+python3 "$D/summarize.py" "$OUT/affine.jsonl" --label "affine (qwen3.5-4b-4bit + qwen3.5-4b-dflash)"
+python3 "$D/summarize.py" "$OUT/nvfp4.jsonl" --label "NVFP4 (laguna-xs-2.1-nvfp4 + laguna-xs-2.1-dflash)"