Skip to content

docs(bench): clean scrub artifacts in benchmark_results #1584

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
inureyes merged 1 commit into from
Sep 2, 2026
Merged

docs(bench): clean scrub artifacts in benchmark_results #1584

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/benchmark_results/model_tests_m1ultra.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -614,7 +614,7 @@ Ran all 80 local models (45GB threshold) to verify text/image generation.

## TurboQuant KV cache — M1 Ultra speed gate readings

First dedicated M1 Ultra reading of the epic- KV speed gate matrix.
First dedicated M1 Ultra reading of the TurboQuant KV speed gate matrix.
Hardware: Apple M1 Ultra, 128 GB unified memory. Model:
`Meta-Llama-3.1-8B-Instruct-4bit`. Date: 2026-04-29. Reproducer:
`./scripts/bench_kv_cache.sh --modes fp16,int8,turbo4-asym,turbo4,turbo4-delegated --contexts 4096 --prefill-contexts 8192`.
Expand Down
28 changes: 14 additions & 14 deletions docs/benchmark_results/model_tests_m5max.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -699,14 +699,14 @@ on the MLX graph execution path, not peak generation throughput.
| Qwen2.5-1.5B-Instruct-4bit (superseded) | fp16 | 3205.54 | 25,550 | superseded — |
| Qwen2.5-1.5B-Instruct-4bit (superseded) | turbo4asym | 2227.09 | 36,775 | superseded — |

The Qwen2.5-1.5B-Instruct-4bit rows above are retained for historical reference. found that fixture collapses on raw wikitext without a chat template; the B3 gate now uses the base variant `Qwen2.5-1.5B-4bit`. Re-run pending.
The Qwen2.5-1.5B-Instruct-4bit rows above are retained for historical reference. A later run found that the fixture collapses on raw wikitext without a chat template; the B3 gate now uses the base variant `Qwen2.5-1.5B-4bit`. Re-run pending.

For the full interpretation and per-model recommendations see
[`docs/turbo-kv-cache.md`](turbo-kv-cache.md).
[`docs/turbo-kv-cache.md`](../turbo-kv-cache.md).

## TurboQuant KV cache — M5 Max speed gate readings

First dedicated M5 Max reading of the epic- KV speed gate matrix.
First dedicated M5 Max reading of the TurboQuant KV speed gate matrix.
Hardware: Apple M5 Max, 128 GB unified memory, macOS 26.4.1 (build 25E253).
Model: `mlx-community/Meta-Llama-3.1-8B-Instruct-4bit` (local dir
`models/llama-3.1-8b-4bit`). Date: 2026-05-03. Binary: mlxcel 0.0.25
Expand Down Expand Up @@ -745,7 +745,7 @@ informal in-tree A/B (100-token decode at the same 4K prompt) measures
`turbo4-delegated` at ~41 tok/s post-fix vs ~27 tok/s on v0.0.25, a ~1.5×
decode speedup that scales sharply at longer contexts.

** Phase-1b (K-side unification):** Removes `cold_keys` and
**Phase-1b (K-side unification):** Removes `cold_keys` and
the per-step `concat(cold_k, hot_k)` graph node. Informal A/B on M5 Max
(3 warm runs each, `llama-3.1-8b-4bit`, 4109-token prompt, 100 generated
tokens): fp16 baseline 101.5–102.7 tok/s; turbo4-delegated post-
Expand All @@ -756,7 +756,7 @@ approximately conserved between the old concat layout and the new slice-update
layout. The remaining cost is the V-side `concat(cold_v_dequant, hot_v)`
graph node (Phase 2).

** Phase-2 (fused dequant + SDPA kernel):** Adds a Metal kernel
**Phase-2 (fused dequant + SDPA kernel):** Adds a Metal kernel
that reads the packed cold V indices directly inside the kernel, removing
the earlier `cold_v_dequant_cache` memo and the per-step
`concat(cold_v_dequant, hot_v)` graph node. The dequantised cold V never
Expand All @@ -780,8 +780,8 @@ the dequantised cold V into a single steel-attention SDPA call. The
0.97× M5 Max decode gate is **not cleared**. Bringing the
kernel inside the steel-attention envelope is left to follow-up work.

** Phase-3 (steel-attention-envelope kernel, M5 Max measurement).**
lands `turbo4_delegated_steel_sdpa` — a JIT-compiled Metal kernel
**Phase-3 (steel-attention-envelope kernel, M5 Max measurement).**
Phase 3 lands `turbo4_delegated_steel_sdpa` — a JIT-compiled Metal kernel
that runs the entire post-Q·K SDPA inline (per-Q numerically stable
softmax, cold-V dequant + weighted sum, hot-V FP16 weighted sum, all
normalised against the same softmax denominator). Bit-parity is gate-1
Expand All @@ -798,7 +798,7 @@ Measured on `llama-3.1-8b-4bit`, 4109-token prompt, 100 generated tokens
| `fp16` | 102.97 | 1.000× | baseline |
| `turbo4-delegated` legacy `update_and_fetch + attention()` (env unset) | 29.60* | 0.291× | ≥0.97× — **fail** (reading) |
| `turbo4-delegated` cold-only fused kernel (`MLXCEL_TURBO4_DELEGATED_FUSED=1`, earlier) | 18.90* | 0.186× | ≥0.97× — **fail** (reading) |
| `turbo4-delegated` steel envelope (`MLXCEL_TURBO4_DELEGATED_FUSED=1`, post-) | 16.23 | 0.158× | ≥0.97× — **fail** |
| `turbo4-delegated` steel envelope (`MLXCEL_TURBO4_DELEGATED_FUSED=1`, post-Phase 3) | 16.23 | 0.158× | ≥0.97× — **fail** |

`*` cross-referenced from the 2026-05-04 CSV.

Expand All @@ -812,12 +812,12 @@ overhead, avoids threadgroup tree-reduction barriers") — the assumption
held on M1 Ultra at parity contexts but breaks on M5 Max where the
threadgroup tree-reduction would actually be faster than the serial scan.

** readings (Pass 1 parallelization + cold-loop sparse cutoff on top).** splits the kernel's Pass 1 (per-Q max + sum_exp) across
all D threads of each threadgroup with a tree reduction. The follow-up in this
PR also precomputes a score-space sparse-V cutoff
**Follow-up readings (Pass 1 parallelization + cold-loop sparse cutoff on top).** The follow-up splits the kernel's Pass 1 (per-Q max + sum_exp) across
all D threads of each threadgroup with a tree reduction. The same follow-up
also precomputes a score-space sparse-V cutoff
`max + log(threshold * sum_exp)`, letting the cold loop reject fully-dead tokens
before paying the exp + dequant cost. Pass 2's weighted-sum remains
D-parallelized exactly as. Measured on `llama-3.1-8b-4bit`,
D-parallelized exactly as in Phase 3. Measured on `llama-3.1-8b-4bit`,
4109-token prompt, 100 generated tokens
(`benchmarks/turbo_kv/2026-05-06_Apple_M5_Max_issue_534_post_fix.csv`):

Expand Down Expand Up @@ -929,10 +929,10 @@ fair). Same CSV as the 4K reading
| Path | tok/s | Generated | × FP16 | gate |
|---|---:|---:|---:|---:|
| `fp16` | 63.94 | 19 | 1.000× | baseline |
| `turbo4-delegated` steel envelope (post-) | 2.39 | 19 | 0.037× | ≥0.95× — **fail** |
| `turbo4-delegated` steel envelope (post-Phase 3) | 2.39 | 19 | 0.037× | ≥0.95× — **fail** |

The 16K decode ratio (3.7% of FP16) is the gate's worst-case shortfall in
the epic- matrix to date, ~1.4× worse than the cold-only kernel
the TurboQuant KV speed gate matrix to date, ~1.4× worse than the cold-only kernel
reading (5.12 tok/s, 0.069× FP16). At 16K the per-token
serial scan over T_total is ~16K reads × 8 threads, completely dwarfing
the tens of milliseconds the FP16 attention path needs for the same step.
Expand Down