[inureyes]

Problem

Follow-up to #329/#345 (the Gemma3n decode profile). That profile found raising MLX_MAX_OPS_PER_BUFFER from its default to ~1000 recovers +11 to 13% decode throughput on M1 Ultra (gemma3n-e2b 82.7 to 93.2, e4b 63.0 to 70.0) by closing command-buffer dispatch-gap idle on the high-op-density decode stack, with a plateau near 1000 and a slight regression at 2000. It deferred the change because the lever is hardware-specific: M5 must stay on its default (the earlier M5 study found larger buffers slower, and #358 later measured the M5 Max sweep as flat), so a single global default is unsafe.

Change

apply_metal_ops_per_buffer_default (src/lib/mlxcel-core/src/hardware.rs) sets MLX_MAX_OPS_PER_BUFFER=1000 at process start on pre-M5 Apple Silicon (AppleSiliconGen M1 through M4, no Neural Accelerator) when the variable is unset. M5+ (Neural Accelerator) and non-Apple (CUDA, e.g. GB10) are left on MLX's default. An operator-set MLX_MAX_OPS_PER_BUFFER (any value) always wins. The pure gate decision is metal_ops_per_buffer_default(gen, has_neural_accelerator) and is unit-tested. The applier is called at the top of main() in the CLI (src/main.rs), the server (src/bin/mlx_server.rs), and the decode bench (src/bin/bench_decode.rs), before any MLX op runs. The knob is documented in the CLI --help environment-variable list. mlxcel-core is edition 2021, so set_var is sound there given the documented early-call contract.

Validation (M1 Ultra, 2026-06-19, best-of-3, raw prompt, 80 tokens after a 20-token warmup, caffeinate)

The gate fires on this M1 Ultra (detected as M1) and reproduces the section-4 1000-row throughput from the profile:

Checkpoint	gated default (1000)	recorded #329 default	gain
gemma3n-e2b-4bit	92.5	82.7	+11.9%
gemma3n-e4b-4bit	68.4	63.0	+8.6%

An explicit low override is respected and collapses throughput to 67.7 (e2b) and 51.8 (e4b) at MLX_MAX_OPS_PER_BUFFER=10, confirming both the lever direction and that operator settings win.

Generalization (no regression on other families): dense qwen3-8b-4bit (86.1) and MoE qwen3-30b-a3b-4bit (89.0) also run faster at 1000 than at a starved cap (75.9 and 79.3 at ops=10), so the generation-wide pre-M5 gate does not regress them. The MoE decode-gap sweep (#268) was flat, consistent with the dispatch-gap binding being strongest where per-layer op density is highest.

Acceptance criteria

• A hardware-gated default that improves M1 Ultra decode with no M5 regression. M1-M4 gets 1000; M5+ is untouched, and docs(benchmarks): Gemma3n decode profile on M5 Max #358 shows the M5 Max sweep flat.
• [~] Cross-hardware bench recorded. M1 Ultra (this re-bench) and M5 Max (docs(benchmarks): Gemma3n decode profile on M5 Max #358 / gemma3n-decode-profile-m5max.md) are recorded in the profile docs. GB10 is not benched in this environment; the gate leaves GB10 (non-Apple, CUDA) on MLX's default, so it is unaffected until a follow-up bench.
• The MLX_MAX_OPS_PER_BUFFER env var preserved as a manual override (respected when set, documented in CLI help).
• A note on whether the gain is Gemma3n-specific or generalizes (added to the profile doc and summarized above).

Validated under the pinned toolchain (rust 1.93.1, matching CI): cargo clippy --all-targets --features metal,accelerate -- -D warnings clean for the root and mlxcel-core, 25 hardware unit tests pass, fmt clean.

Closes #353.