[inureyes]

Summary

Companion to the M1 Ultra Gemma3n decode profile from #329 (PR #345),
measured on Apple M5 Max (40-core GPU, 128 GB, macOS 26.5.1), current
HEAD, MLX 0.31.2, mlx-vlm 0.4.4. M5 Max is detected as Neural-Accelerator
hardware, so use_fused_decode_path() is false and decode runs the
per-op split path, a different code path than M1 Ultra. That is why the
profile is worth repeating here rather than assuming the M1 Ultra result
carries over.

The #329 conclusion (no compiled fusion is justified for Gemma3n decode)
holds on M5 Max, by a partly different route.

Findings (Apple M5 Max)

• M5 Max runs these 4-bit checkpoints at 1.8 to 1.9x M1 Ultra: e2b-4bit
  161.5, e4b-4bit 112.2 (text) / 107.0 (image) tok/s, e4b-bf16 40.4.
• Decode is 92 to 96% GPU-bound. The Rust graph-build step (forward,
  0.47 to 0.66 ms/token), where an FFI-collapsing fusion would land,
  stays hidden behind async_eval.
• MLX_MAX_OPS_PER_BUFFER is flat on M5 Max (at most +1.7% across the
  sweep, within best-of-3 noise). The same sweep gains +11 to 13% on M1
  Ultra, so that lever must stay hardware-gated and there is no free win
  for a fusion to displace.
• 4-bit decode sits at 46 to 59% of the 478 GB/s GEMV bandwidth wall, bf16
  at 76%. The 4-bit idle is diffuse dispatch/structure overhead, not a
  fusable kernel hotspot.
• mlxcel leads mlx-vlm 1.14 to 1.17x (1.20 to 1.23x on M1 Ultra); no
  parity gap pulling for a kernel.

Stale comment corrected

src/models/gemma3n.rs recorded the #60 fused decode path as a -6.3%
regression on M5-class hardware. Re-measured on this M5 Max with MLX
0.31.2 (forced on/off through a temporary override, reverted after) it is
neutral: e4b-bf16 +0.84%, e4b-4bit +0.34%, both within noise, with no
bf16 JIT crash. The gate stays off NA hardware because the fused path has
no upside on M5 and the per-op split path is the validated default; only
the figure was out of date. Gating logic is unchanged.

What changed

• Add docs/benchmark_results/gemma3n-decode-profile-m5max.md with the
  full numbers, pipeline split, bandwidth floor, command-buffer sweep,
  fused-vs-split re-measurement, text-vs-VLM, mlx-vlm parity, and
  reproduce commands.
• Link it from docs/benchmarks.md.
• Update the two stale fused-path comments in src/models/gemma3n.rs.

Test plan

• Decode best-of-3 (caffeinate, cool) on gemma3n-e2b/e4b-4bit and
  e4b-bf16 via mlxcel-bench-decode, text and image.
• Pipeline split via MLXCEL_PROFILE_PIPELINE_DETAIL; op histograms
  via MLXCEL_EXPORT_DECODE_DOT.
• MLX_MAX_OPS_PER_BUFFER sweep {default,100,200,1000,2000} on all
  three checkpoints.
• Fused-vs-split via a temporary MLXCEL_FORCE_*_DECODE override,
  reverted; tree clean and binary rebuilt from HEAD afterward.
• mlx-vlm wall-clock parity (chat template both sides).
• GEMV peak bandwidth measured at 478 GB/s (f16/bf16).
• cargo fmt --check passes. Docs and comments only; no inference
  code path changed.