[inureyes]

Summary

mlxcel_core::gelu_approx is the exact erf GELU, while MLX's nn.gelu_approx / approx="precise" is the tanh form. This PR adds an explicit GELU selector, folds the private tanh copies into it, corrects the comments that claimed erf matched approx="precise", and switches the vision families whose checkpoint and mlx-vlm port both use tanh, evaluated in f32 through a compiled kernel, which keeps the prefill cost small: Gemma 3 +2.8% pooled (same sign in both blocks, so a small real cost), Gemma 4 +2.0%, MiniCPM-V 4.6 and PaliGemma within the null arm (the eager form cost Gemma 3 +17%).

• GeluActivation { Erf, Tanh, TanhF32, TanhF32Compiled, Fast } plus from_hf_name in mlxcel_core::utils, backed by gelu_tanh / gelu_tanh_f32 / compiled_gelu_tanh_f32 on the C++ gelu_tanh_approx helper. gelu_approx keeps erf.
• Folded copies (BERT, Kokoro, Molmo2, SigLIP with a declared gelu_pytorch_tanh, Kimi-VL/MoonViT, Qwen3-VL) keep their eager formula and precision; tests prove the folds bit-equal for f32/bf16/f16 (BERT now casts a half result back, bit-equal for f32).
• Behavior change, tanh in f32 through TanhF32Compiled: MiniCPM-V 4.6 encoder MLP (from hidden_act) and window merger; Gemma 3 VLM and PaliGemma SigLIP (missing hidden_act resolved to the SigLIP default as VisionHiddenActivation::SiglipDefaultTanh, mirrored in the XLA Gemma 3 vision emitter); Gemma 4 vision GeGLU (from hidden_activation, also reaching DiffusionGemma and EmbeddingGemma2).
• A hidden_act / hidden_activation that is not a GELU name now fails the load for the MiniCPM-V 4.6 encoder and the Gemma 4 vision tower instead of silently running erf; every local config declares gelu_pytorch_tanh or omits the key.
• Kept erf deliberately: InternViT and the MiniCPM-V 4.6 downsample merger (Hugging Face evaluates erf there); Gemma3n sparse layers (kernel, fused path and XLA oracle must move together).
• CUDA PTX cache fix (outside GELU, affects any kernel): since fix(cuda): cache custom kernels on disk to drop a per-process compile #2538 the jit_module.cpp overlay wrote each cache file through <module name>.txt.tmp<pid>_<tick>, which passes NAME_MAX for module names of about 229-245 characters, so those kernels were never cached and every process NVRTC-compiled them again. The compiled tanh kernel (234 characters) hit it: Gemma 3 prefill 2505 ms uncached against 869 ms cached. The temporary is now named from the suffix and the extension only; tests/cuda_custom_kernel_disk_cache.rs runs this kernel in a second process, fails on the old overlay and passes now.

Evidence

Inventory: docs/gelu-call-sites.md. Measurements: docs/benchmark_results/gelu-erf-vs-tanh-2026-10-23.md.

• bf16 re-test on gemma-3-4b-it-4bit's 4096-patch bf16 SigLIP tower, MiniCPM-V 4.6 and PaliGemma2: zero NaN, zero Inf for tanh in input dtype and in f32.
• Logit traces: the decided-position gate cannot distinguish erf from tanh for any measured family, so decisions follow the checkpoint's declared activation; the perplexity movements are bf16 noise and support neither arm.
• Compiled vs eager f32 tanh: bit-equal on a tower-shaped bf16 [64, 4304] input with outliers to +-170 (0 of 275456 lanes differ); within a 4-ulp f32 bound and one bf16/f16 ulp on the sample sweep (exact f32 max diff not printed); greedy 32-token replies identical between the eager and compiled builds on all five smoke checkpoints.

Image prefill, mlxcel generate --image tests/fixtures/test_image_shapes.png -p "Describe the image." -n 8 --temp 0 --profile, one binary, erf and null arms through symlinked checkpoint copies declaring gelu, arm order rotated, MLX_ENABLE_TF32=0, one gpu-lock hold per block, no CI/cargo/nvcc process during any run. GB10, kernel 7.0.0-1019-nvidia, driver 580.178.04, MLX_CUDA_ARCHITECTURES=121.

Family	n	erf median	compiled tanh median	Paired tanh - erf (range)	Null - erf (range)	Outcome
Gemma 3 SigLIP	16	859.5 ms	878.7 ms	+23.8 ms, +2.8 % (-11.8..+53.8); blocks +2.4 % and +3.7 %	-0.5 ms (-33.8..+34.8)	tanh kept, residual unresolved
Gemma 3, eager TanhF32 (previous state)	8	869.3 ms	1013.4 ms	+148.0 ms, +17.0 % (+116.5..+174.7)		reference
MiniCPM-V 4.6	6	910.8 ms	897.1 ms	-14.0 ms, -1.5 % (-36.1..+42.4)	+2.4 ms (-34.5..+31.6)	tanh kept, within null floor
Gemma 4 vision	12	661.7 ms	677.9 ms	+13.2 ms, +2.0 % (-6.0..+36.3)	+1.0 ms (-27.9..+24.8)	tanh kept, residual unresolved
PaliGemma SigLIP	6	995.5 ms	987.6 ms	-15.1 ms, -1.5 % (-37.5..+12.8)	+8.1 ms (-41.6..+20.3)	tanh kept, within null floor

The Gemma 3 and Gemma 4 residual has the same sign in every block while the null arm is centered on zero, so it is a small real cost; pooled it is under the 3 % acceptance line, though one Gemma 3 block alone (+3.7 %) was over it. On CUDA the compiled region still runs as more than one fused kernel; a fully fused kernel is the #2548 item. The MiniCPM-V erf arm keeps the hard-coded tanh window merger, so that row slightly understates the difference.

# prompt_tokens disagree with mlx-vlm for InternVL, MiniCPM-V 4.6 and Gemma 3, invalidating those cross-runtime comparisons (#2547).

Test plan

• -p mlxcel-core --lib gelu (24 passed, including the compiled-vs-eager bounds) and activation_helpers_return_the_input_dtype, --test-threads=1, MLX_ENABLE_TF32=0
• ffi_tests::shapeless_compile_audit_harness with MLXCEL_SHAPELESS_COMPILE_AUDIT=1 (compiled_gelu_tanh_f32 PASS, 3 signatures warmed)
• --test cuda_custom_kernel_disk_cache fails on the previous overlay and passes with the fix
• -p mlxcel --lib filtered to siglip (39), minicpmv4_6 (14), gemma4::activation (2), vision::config (2); earlier on this branch also internvl, molmo2, qwen3_vl, bert, kimi, kokoro, moonvit, and -p mlxcel-xla --lib vision_config emitter::vision
• cargo clippy --profile test-fast --features cuda -p mlxcel-core --lib --tests and -p mlxcel --lib --tests with -D warnings; cargo fmt --all --check
• CLI --image smoke on minicpm-v-4.6-bf16, gemma-3-4b-it-4bit, paligemma2-3b-6bit, gemma-4-e2b-it-4bit, internvl3-1b (compiled and eager builds, identical text); server request and DiffusionGemma CLI on the earlier commit

Unverified: Metal (compiled path and overlay untouched there, not run); EmbeddingGemma2 (no local checkpoint); DiffusionGemma through the server (rejects image input, pre-existing); whole-workspace test and clippy runs.

Remaining families are tracked in #2548. Criteria 3, 4 and 6 are met only for the measured families, so this PR does not close the issue.

Refs #2423

[inureyes]

Implementation Review Summary

Intent

Add an explicit erf/tanh/fast GELU selector, fold the private tanh copies into it, and switch only the vision families whose checkpoint declares tanh (Refs #2423; the rest is #2548).

Findings Addressed

• The switch to TanhF32 costs image prefill time and the PR did not say so (HIGH). Measured on gemma-3-4b-it-4bit, one binary, interleaved: erf 872 ms median, TanhF32 1025 ms (paired +151 to +172 ms), input-dtype Tanh 916 ms (+42 ms). f32 kept on the evidence already in the record; cost section added to docs/benchmark_results/gelu-erf-vs-tanh-2026-10-23.md and the PR body (715ae4a), fused-kernel remedy added to fix(models): decide the GELU formula for the families #2423 left open #2548.
• PR body did not mention that a non-GELU hidden_act / hidden_activation now fails the load for MiniCPM-V 4.6 and the Gemma 4 vision tower (MEDIUM). Body updated.
• gemma4_activation_tests.rs named gemma-4-12b-it-4bit as a checkpoint that omits the key, but gemma4_unified is encoder-free and never parses Gemma4VisionConfig (LOW). Fixed in 715ae4a.

Remaining Items

• BERT's Activation parses its own GELU names (bert.rs:280) instead of GeluActivation::from_hf_name (LOW), left as is because it also maps relu.
• TanhF32 cost on PaliGemma, MiniCPM-V 4.6 and Gemma 4 vision not timed (expected to scale with tower size).

Checked, no finding

• Missing hidden_act resolves to tanh only through default_missing_hidden_act_to_siglip, called by exactly the Gemma 3 VLM (gemma3 / gemma3_text with vision weights) and PaliGemma loaders. CLIP towers (LLaVA 1.5, LLaVA-NeXT) and the other SigLIP omitters (Aya Vision, Granite Vision 3.x) keep the serde default erf. The XLA emitter flips only model_type == "gemma3" and lowers tanh in f32, matching eager TanhF32; covered by the extended gemma3_avg_pool_projector_binds_pool_and_prompt_contract test.
• from_hf_name has two callers (MiniCPM-V 4.6, Gemma 4 vision). Every local config under models/mlx for those families declares gelu_pytorch_tanh or omits the key.
• Fold tests reproduce the SigLIP/Molmo2/Qwen3-VL and Kimi copies op for op and assert bit equality for f32, bf16 and f16. BERT's copy equals the f32 oracle for f32 input; its half-input dtype change is stated. No local BERT config takes the tanh branch.

Verification

• All stated requirements implemented (for the measured families; Refs, not Closes, is correct)
• No placeholder/mock code remaining
• Integrated into project code flow
• Project conventions followed
• Existing modules reused where applicable
• No unintended structural changes
• Tests pass: mlxcel-core gelu 22, mlxcel lib filters vision::config 2, minicpmv4_6 14, gemma4::activation 2, siglip::tests 15, molmo2 7, qwen3_vl 43, kimi 169, kokoro 14, bert 64, mlxcel-xla vision_config emitter::vision numeric_ops 23; clippy -D warnings on mlxcel-core and mlxcel lib+tests clean; fmt clean

[inureyes]

Security and Performance Review

Scope: the PTX cache temp naming (a78ffcf), config-driven GELU selection, the compiled f32 tanh GELU, and the recorded prefill cost. No CRITICAL, HIGH or MEDIUM findings, so there are no fix commits.

Checked, no finding

• Temp collisions: one suffix per write_cached_ptx call with three distinct extensions. Writers in one process are serialized because write_cached_ptx runs only inside the JitModule constructor, which get_jit_module builds under its process-wide unique_lock. Across processes, a collision needs the same pid and the same nanosecond tick.
• Rename: the temp sits in the final file's directory, including get_ptx_path's nested directories, so rename(2) stays atomic and replaces a symlink at the final path instead of following it. The default cache dir is user-owned (ensure_persistent_ptx_cache).
• Config strings: from_hf_name returns Option, and every caller of the two mlp_activation() helpers propagates the error. Nothing panics.
• Compile cache: shapeless compile retraces when the input dtype changes, so the x.dtype() cast cannot go stale. Entries are bounded by the dtype/ndim combinations in use.
• Prefill: Gemma 3 +2.8 % pooled with a null arm, recorded in docs/benchmark_results/gelu-erf-vs-tanh-2026-10-23.md.

LOW (recorded, not changed)

• A crash orphan is now a dotfile without the module name. Nothing sweeps the cache, before or after this PR.
• std::ofstream follows a symlink at the temp name. Upstream already did this; O_CREAT | O_EXCL | O_NOFOLLOW would close it.
• A null hidden_act (MiniCPM-V 4.6) or hidden_activation (Gemma 4 vision) fails the load instead of falling back to the default, unlike VisionConfig. No local checkpoint has this.
• Gemma 3's +2.8 % residual (one block +3.7 %) has not been investigated, and fix(models): decide the GELU formula for the families #2423 left open #2548's fused-GELU item predates the compiled kernel.

Tests

GB10, 316fea1, test-fast + cuda, MLX_ENABLE_TF32=0, gpu-lock, one thread: mlxcel-core gelu 24 passed; cuda_custom_kernel_disk_cache 1 passed (the child compiled nothing, no .tmp leftovers); mlxcel lib siglip 39, minicpmv4_6 14, gemma4::activation 2, vision::config 2 passed.