docs(fold): CUDA's flash-attention-off KV cells ran at batch 512; the f32 cast is llama.cpp's graph

[glennneuber]

This follows up Metal's batch finding on #387 (comment 5863032494) for CUDA, corrected after an adversarial review. It changes the fold record's CUDA KV section, and one line of kv-cache-type-per-model.md.

• The CUDA flash-attention-off cells ran at batch 512. In kv: production runs an f16 KV cache on every platform, and a cross-host loop test #387's CUDA KV × flash-attention matrix, every flash-attention-off launch ran at -b/-ub 512 with images decoded in pieces. The flash-attention-on arms ran at 2048.
  • The cause is upstream's rule in automaticGenerationBatch (llama-server followups ollama/ollama#16353), not a memory fit. On CUDA, flash attention off never promotes the automatic batch past 512. It drops to 256 when a CUDA device has 8 GiB or less free and the context is above 4096. It returns before gemma4's 2048 image-chunk floor and the fit check (generationBatchFits) apply. So a quiet GPU still gives 512, and only a pinned num_batch compares the two paths at one batch.
  • What follows. Those cells change the batch along with the path, so they are not flash attention's effect alone. The fold and 908 stay byte-identical with flash attention off, since both ran at 512.
  • What else ran at 2048. The loop-rate run and item 8's two CUDA GGUF legs ran at 2048 throughout.
• The f32 cast is llama.cpp's graph. f32 equals f16 under flash attention because llama.cpp's graph casts K and V (build_attn_mha) on every backend, as Metal found.
  • CUDA's own graphs show the cast. With flash attention on, f32 builds 2,764 nodes against f16's 2,704: one K and one V cast in each of 30 layers. With flash attention off, both build 2,829.
  • kv-cache-type-per-model.md said CUDA's and HIP's kernels did the conversion. It now says the graph does, and cites kv: production runs an f16 KV cache on every platform, and a cross-host loop test #387 for Metal's measurement.

#400 carries the related corrections into the KV task record, the skill and ADR 0044.

ai-server/mlx-cuda

🤖 Generated with Claude Code