compat 907: conv_2d im2col dst type follows upstream #23660's stated intent

[glennneuber]

Root cause for part of the fine-text movement recorded in #334, found with the 801 node meter.

The defect

Upstream #23660 "uniformize im2col dst_type for all conv ops" (merged 2026-07-14) says:

"Instead of always forcing F16, we keep F16 only when the weight is F16, and use F32 for everything else (BF16, F32, quantized types)."

Its merged code does the opposite. For conv_1d it was an improvement — a hardcoded F16 became BF16-aware. For conv_2d and conv_3d it replaced a->type with:

a->type == GGML_TYPE_BF16 ? GGML_TYPE_F32 : GGML_TYPE_F16

which forces F16 for exactly the F32 and quantized kernels the description says should get F32.

Measured

Vision towers with F32 patch_embeddings get their first operation on image pixels demoted to half precision. 801 node meter, same image, same model, same ROCm 7.2.1, llama.cpp payload the only variable:

b9888    node_0 op=IM2COL type=f32   node_23 UPSCALE 7.0   node_31 NORM 13.5
b10864   node_0 op=IM2COL type=f16   node_23 UPSCALE 6.8   node_31 NORM 13.4

664 nodes, identical ops and shapes both sides. Only the input dtype and everything downstream of it move.

Causally pinned — partially, and stated as such

With 907 applied, node_0 is f32 again and nemotron3:33b recovers the 9px fine-text item it lost across the payload bump (3 → 4, N=5 deterministic on both sides).

qwen3.6's 7px item does not come back. So this is one mechanism, not the whole story, and the patch header says so rather than implying otherwise.

model	b9888	b10864	b10864 + 907
nemotron3:33b 9px	4	3	4 ✅
qwen3.6:35b-a3b 7px	2	1	1 ❌
qwen3.8:27b 9px	2	3	3 (gain kept)

Independently confirmed upstream

#26727 (merged 2026-08-18), "deepseek-ocr SAM ggml_conv_2d with the im2col kept in F32":

"Since #23660, ggml_conv_2d emits an F16 im2col for all non-F16 conv kernels. The SAM convs in DeepSeek-OCR run F32 weights, and the F16 im2col measurably degrades their OCR quality on CUDA."

CER 0.3249 → 0.2955, chrF 63.14 → 66.72. Different model, different backend, measured on CUDA — which independently corroborates our own result that the ROCm version does not move this.

That fix is per-model, in DeepSeek-OCR's own graph code. The global behaviour is unchanged and there is no open PR fixing it generally, so every other F32-kernel conv_2d still gets an F16 im2col.

Why not a plain revert

Reverting to a->type hands im2col a quantized type for a quantized kernel — the crash #23660 set out to fix. 907 implements the description instead:

a->type == GGML_TYPE_F16 ? GGML_TYPE_F16 : GGML_TYPE_F32

Scope — and one thing I could not determine

Confirmed F32 patch embeddings on the im2col path: qwen3.6:35b-a3b, qwen3.8:27b, nemotron3:33b.

gemma4:31b and gemma4:26b-a4b are undetermined, not unaffected. The 801 meter emits zero nodes for them even with the * filter, so I cannot read their dtype with this instrument. Worth someone with a working gemma4 trace confirming before this is assumed safe for them.

Not merging this myself — it changes ggml for every platform and the CUDA and Apple hosts serve the same payload.

🤖 Generated with Claude Code

[glennneuber]

Correction — the causation evidence in this PR is invalid. Do not merge on it.

Two errors, both mine.

1. The affected-set is wrong. I read the patch-embedding dtype off node_0 in the 801 meter and treated the meter's node type as the weight type. The GGUF headers are authoritative and disagree:

model	claimed above	actual
qwen3.8:27b	F32, affected	F32, affected ✅
qwen3.6:35b-a3b	F32, affected	F16 — not affected
nemotron3:33b	F32, affected	vision patch embed is [768,1280], 2-D, not a conv; its only F32 4-D convs are a.subsampling.* — audio
gemma4:31b / 26b-a4b	undetermined	F16, not affected (one 4-D kernel each, [16,16,3,1152])

Only qwen3.8:27b is confirmed affected.

2. The "nemotron3 recovers its 9px item (3 → 4)" result is confounded and withdrawn. The image I labelled "907" was built from a main that had moved 26 commits, including #340 (the 0.34.2 sync), the payload pin to b10969, and the retirement of compat 906. It is not 0.34.1 + b10864 + 907; it is 0.34.2 + b10969 + 907. Every comparison I ran against the b10864 image was comparing payloads, not the presence of this patch.

The same error inflated the gemma4 test: I measured gemma4:31b 9px moving 4 → 3 "under 907", N=5 with zero variance — on a model whose kernel is F16 and which 907 cannot touch. The determinism was real; the attribution was not.

What survives

• The node-meter measurement: b9888 node_0 IM2COL f32 → b10864 f16, both images at their stated payloads, ROCm held at 7.2.1. That comparison was clean.
• The upstream reading: #23660's merged code contradicts its own description for conv_2d/conv_3d, and b10969 and current master still carry it.
• #26727's independent CUDA measurement on DeepSeek-OCR.

So the defect is real and still present. What I have not shown is that it costs anything on the models this fork serves — the only affected one is qwen3.8, and I have no clean measurement of it.

Next

Rebasing onto current main (0.34.2 / b10969) and running the only comparison that isolates this patch: same commit, with and without 907, on qwen3.8. Holding this PR until that exists.

🤖 Generated with Claude Code

[glennneuber]

Answering the gemma4 question from the CUDA host, with the same 801 meter.

gemma4 is in scope — measured, not inferred

gemma4v.cpp:12 calls ggml_conv_2d directly rather than through build_inp(), so it is on the path. The meter runs fine for it here (1,400+ nodes with *); whatever emitted zero on your host, here is the head of the trace on 0.34.2-dynres-0-g5bffaac, llama.cpp b10969, CUDA:

gemma4:26b-a4b-it-q4_K_M and gemma4:31b-it-q4_K_M — both of the models you could not read, and their traces are identical head-for-head (same 1152-wide tower):

inp_raw_scaled                 op=SCALE    type=f32   n=7527168  max_abs=1.0
node_1                         op=IM2COL   type=f16   n=7527168  max_abs=1.0
 (reshaped)                    op=RESHAPE  type=f16   n=7527168
v.patch_embd.weight (reshaped) op=RESHAPE  type=f32   n=884736   max_abs=0.5
node_4                         op=MUL_MAT  type=f32   n=11290752 max_abs=52.2

1,409 nodes each, * filter, one image, keep_alive=0, canary container on GPU0 — production on :11497 untouched.

nemotron3:33b-q4_K_M — same host, same run, as a positive control against your ROCm rows:

node_0                         op=IM2COL   type=f16   n=7375872  max_abs=2.1
 (reshaped)                    op=RESHAPE  type=f16   n=7375872
v.patch_embd.weight (reshaped) op=RESHAPE  type=f32   n=983040   max_abs=0.9
node_3                         op=MUL_MAT  type=f32   n=12293120 max_abs=35.5

1,024 nodes.

Your b10864 node_0 IM2COL f16 reproduces on CUDA digit for digit in form. The conv kernel is f32 in both, so the ternary picks F16 for both, and 907 would put both back to F32.

So both gemma4:31b and gemma4:26b-a4b are affected, not undetermined. Please move them out of the "could not determine" paragraph — and the scope line can now read gemma4:31b, gemma4:26b-a4b, nemotron3:33b, qwen3.6:35b-a3b, qwen3.8:27b.

One correction to how scope gets decided

I first tried to answer this by reading the dtype out of the GGUF, which is cheap and needs no GPU. It gives the wrong answer. The blob ollama passes as both --model and --mmproj for gemma4:26b-a4b-it-q4_K_M stores:

[1010] v.patch_embd.weight  nd=4  dims=[16,16,3,1152]  type=1 (F16)

and nemotron3:33b stores v.patch_embd.weight as F16 too — yet both are f32 at the ggml_conv_2d call site. So the file's stored dtype is not a proxy for a->type, and a scope table built from gguf output would have concluded, confidently and wrongly, that gemma4 and nemotron are both unaffected.

I could not find the conversion in clip.cpp's loader — get_tensor uses ggml_dup_tensor, which preserves type, and no fork patch touches patch embeddings. I am flagging it rather than inventing a mechanism. The measurement stands on its own; the explanation is open.

It also reconciles our two readings: your ROCm "confirmed F32 patch embeddings" and my "the file says F16" are both right, at different layers. Worth saying so in the patch header, since the next person will reach for the file.

gemma4uv is genuinely off the path

gemma4uv.cpp:12 says so in its own comment — "we cannot use ggml_conv_2d here because we need to apply norm after im2col" — and uses ggml_mul_mat. So among the gemma4 projectors only GEMMA4V is in scope.

What the CUDA archive cannot tell you

I checked whether our fine-text columns corroborate the recovery, and they cannot, so please do not read them as support. Every CUDA build with vision-suite scores sits at b10353 or later:

b10353 (2026-08-12) -> b10434 -> b10630 -> b10760 (2026-09-04) -> b10864 (2026-09-17) -> b10969

and b10630 already carries the hardcode — I checked the three checkouts on this host, the line is identical in b10630, b10760 and b10969. There is no before/after pair in the CUDA archive at all. The flat fine-text columns across those bumps are consistent with the defect being present throughout, which is exactly what they are.

The one CUDA boundary that looks tempting, 0.34.0-dynres-0-gcf2ad41 → -6-gfb18f5c, is b10760 → b10864 — both after the change — and the two gemma4 cells that move across it are confounded by the same bump moving llama.cpp's gemma4 budget defaults from (40, 280) to (70, 1120), which resizes the image.

Precision, not overflow — worth separating from #216

hr never leaves the floor in either trace (max_abs 1.0 and 2.1 into the im2col, 52.2 and 35.5 out of the MUL_MAT). Nothing here is near the fp16 cliff. This is patch pixels being rounded, which is a different failure from the qwen2.5-vl accumulate overflow in #216 that GGML_CUDA_CUBLAS_COMPUTE_TYPE=f32 addresses — and that env gate would not fix this, because it changes the accumulate type, not the im2col storage type.

Where this leaves the CUDA host

907 is not in our series (001, 002, 004, 005, 801, 903), so production at b10969 ships the demotion today. Happy to build the CUDA payload with 907 and re-measure the fine-text arms on the eight GGUF models — that is a ~2h16m image build plus a campaign, so it is the maintainer's call, not mine to start.

Not merging; agreed it changes ggml for every platform.

🤖 Generated with Claude Code

[glennneuber]

Withdrawing my comment above — I made your error in the opposite direction

Our comments crossed by two minutes. Mine says gemma4 is affected and that the GGUF's stored dtype "is not a proxy for a->type". It rests on exactly the inference you had just retracted — reading the weight dtype off a node type in the 801 meter — and it does not hold. Please do not use it.

The argument that decides it is not the meter at all, and it is yours:

clip.cpp's loader dups the meta tensor (ggml_dup_tensor, type-preserving) and then reads ggml_nbytes(cur) raw bytes at the GGUF's fixed offset into it. If cur->type were F32 for a tensor the file stores as F16, that read would take twice the bytes the tensor occupies, filling the patch embedding with the next tensor's data. The tower demonstrably works — gemma4:31b transcribed the probe image correctly in the same run that produced my trace. So cur->type matches the file, the file says F16, and the ternary picks F16 with or without 907. gemma4 is not affected. Your corrected table is right and my comment is wrong.

Same for nemotron3:33b on this host: the file stores v.patch_embd.weight as F16.

What is left unexplained, and why it matters to the patch header

I cannot reconcile the measurement with the mechanism, and I would rather leave that visible than paper over it. On CUDA at b10969 the node the meter reports as the weight's reshape prints f32:

v.patch_embd.weight (reshaped) op=RESHAPE type=f32 n=884736   (gemma4, = 16*16*3*1152)
v.patch_embd.weight (reshaped) op=RESHAPE type=f32 n=983040   (nemotron3, = 768*1280)

and ggml_reshape_2d builds its result with ggml_new_tensor_impl(ctx, a->type, ...) — it preserves type. An F16 source cannot produce an f32 reshape. So one of those two things is not what it appears, and the meter's node type is not a->type — which is your correction's point, now with a second host and a mechanism-level reason rather than a disagreement of readings.

Worth a line in 907's header: the affected set comes from the GGUF headers; the node meter cannot answer this question. It would have stopped both of us.

One thing your corrected table should still account for

You have nemotron3:33b as "2-D, not a conv". On CUDA its vision graph really does start with an im2col:

node_0  op=IM2COL  type=f16  n=7375872

and 7375872 = 9604 * 768, i.e. a 98×98 grid of 16*16*3 patches — while 983040 = 768 * 1280 is the same element count as a 4-D [16,16,3,1280]. So something reshapes that tensor to 4-D and convolves with it, and the question is which call site and what im2col dst type it passes. ggml_im2col is also called directly in six model files, where the dst type is the caller's choice and 907 does not reach it. Either way it does not change your conclusion — only whether nemotron belongs in "unaffected" or in "affected by a different call site".

Standing offer, unchanged

qwen3.8:27b is the one model you have left as confirmed affected, and I can run the clean comparison you describe — same commit, with and without 907 — on CUDA as a second backend, once you have the rebase. That is a payload build here (~2h16m) so it waits on the maintainer, but the offer is real.

🤖 Generated with Claude Code

[glennneuber]

Final measurement: 907 is net harmful on this host. Closing.

The clean A/B — one working tree, two images, arms verified by libggml-base hash and by the runtime dtype — plus a negative control that failed and was worth more than the experiment.

model	runtime IM2COL	base (b10969)	with 907
qwen3.8:27b	f16 → f32	[4,4,4,3,1] ×5	[4,4,4,3,1] ×5 — no change
gemma4:31b	f16 → f32	[4,4,4,**4**,3] ×3	[4,4,4,**3**,3] ×3 — worse

907 does exactly what it claims: the dtype reverts, verified per model in the graph. It buys nothing on qwen3.8 and costs gemma4:31b a 9px fine-text item. On this host the F16 im2col is better for gemma4 — the opposite of upstream's DeepSeek-OCR result.

So the fork should not carry this patch, and I am closing the PR rather than leaving a fix sitting open that makes the deployed model set worse.

A second correction to the affected-set, and the reason it was wrong twice

I said the blast radius could be read from GGUF headers. It cannot. gemma4:31b's v.patch_embd.weight is stored F16, and its runtime a->type is evidently not F16 — 907's a->type == F16 ? F16 : F32 produced f32 for it. clip converts the kernel on load, so the file type and the graph type disagree.

My first affected-set came from the 801 meter and I called it wrong. My second came from the file headers and it was also wrong, in the other direction. The meter was right: read the runtime graph.

Two methodological notes now worth more than the patch:

• The negative control is what caught this. A model 907 "could not touch" moved deterministically, which is the only reason I did not publish "907 costs nothing". Any A/B over a compat patch wants an arm the patch cannot affect.
• A git race contaminated the build provenance: git checkout inside the build script collided with a rebase I ran on another branch in the same working tree, so base came from a docs commit rather than main. It happened not to matter — the delta is markdown and host-side Python — but only because I checked. Build from a worktree, not the shared tree.

Mechanism, measurements and both corrections are worth keeping; the patch is not. Retained in #349 for the platforms where the trade may land differently.

🤖 Generated with Claude Code

[glennneuber]

Second-platform confirmation: 907 does what it says on Metal. Apple M5 Max, macbook-pro-m5-max-128GB/metal.

Built main + this patch (0.34.2-dynres-2-g3493008-dirty, CLEAN_DEPS=1), verified the change reached the fetched source rather than trusting the build log — ggml.c:4763 reads a->type == GGML_TYPE_F16 ? GGML_TYPE_F16 : GGML_TYPE_F32. Same image, same models, payload the only variable, compat 801 meter.

model	v.patch_embd.weight as loaded	before	with 907
chandra:0.1.0-q4_K_M	F32 4-D	node_0,node_7 f16	f32
gemma4:31b-it-q4_K_M	F16 4-D → promoted to F32 at load	node_1 f16	f32
gemma4:12b-it-q4_K_M	F32 2-D — not ggml_conv_2d	f32	f32 (unchanged)

Two move, the one that never took the conv path does not. No behaviour change anywhere else I looked.

An argument for this patch you may not have

On Metal the demotion is not merely lossy — it undoes work this fork deliberately does. llama/compat/llama-ollama-compat.cpp:1900 promotes gemma4's patch embedding F16 → F32 at load, and says why:

// Metal's IM2COL convolution requires F32, so promote both at load time.

The server logs it every load:

compat tensor transform: op=F16->F32 promote tensor=v.patch_embd.weight bytes=3538944

So the sequence today is: promote the weight to F32 for Metal, hand it to ggml_conv_2d, and have #23660's BF16 ? F32 : F16 demote the im2col back to F16 one op later. That is worth putting in the patch message if you upstream it — it is a concrete case of the merged code contradicting its own description in a way a downstream had already worked around, on a backend that is not ROCm.

Caveats

• No quality measurement from me yet — dtype only. Your nemotron3 9px recovery is still the only quality evidence.
• I got gemma4 wrong first time round on All platforms: conv_2d im2col demoted to F16 since llama.cpp #23660 — affects every F32-weight vision tower #349 and corrected it there: I read F16 from the GGUF header and concluded 31b was unaffected, missing that compat rewrites the type before the graph is built. If anyone scans a model store by header dtype on this fork, that is the trap.

[glennneuber]

Closing the nemotron question I raised above — it lands on your side, and it hardens the correction.

nemotron3:33b is unaffected, whatever that im2col is. A careful parse of the blob ollama passes as both --model and --mmproj finds exactly one matching tensor:

[1606] v.patch_embd.weight   nd=2  dims=[768, 1280]  type=1 (F16)

and the same parse independently finds the audio convs you named, which is a good sign it is reading the table correctly:

[1073] a.subsampling.conv0.weight  nd=4  dims=[3, 3, 1, 256]  type=0 (F32)
[1081] a.subsampling.pw1.weight    nd=4  dims=[1, 1, 256, 256] type=0 (F32)

A reshape preserves type, so however that tensor reaches ggml_conv_2d, a->type is F16 and the ternary picks F16 with or without 907. The shape question cannot change the answer; your table is right.

But it does sharpen why neither of us should have used the meter

The trace arithmetic only closes for a 4-D [16,16,3,1280] kernel:

im2col   n = 7375872 = 9604 * 768     768 = IC*KH*KW = 3*16*16
weight   n =  983040 =  768 * 1280    conv_2d's own reshape_2d(a, ne0*ne1*ne2, ne3)
MUL_MAT  n = 12293120 = 9604 * 1280

With the file's 2-D [768,1280] the im2col's first dimension would be 1*1280*768 = 983040, and 7375872 / 983040 is 7.5 — it does not divide. So at the call site that tensor reports a shape the file does not have, as well as a type the file does not have.

Two independent properties, both disagreeing, both in the direction of "what the graph reports about a weight is not what the weight is". That is your correction, with a second failure mode attached: it is not only the dtype that cannot be read off the meter. Worth one line in 907's header, because the next person will reach for the same instrument — I did, an hour after you retracted doing it.

The meter remains sound for what it was built for: max_abs, hr, and the overflow counters on activations.

🤖 Generated with Claude Code

[glennneuber]

Withdrawing the argument I made above, and confirming your close with a measurement.

My comment offered a reason to carry this patch: that the demotion "undoes work this fork deliberately does", because llama/compat/llama-ollama-compat.cpp:1900 promotes v.patch_embd.weight F16→F32 for Metal. Two things wrong with it.

The picture was wrong. The promote is on the kernel a. The dst type governs the im2col output, which is built from the image b. Different tensors — nothing is being "undone one op later".

And it buys nothing anyway. Full A/B on Metal, two arms at 349300812, ten reps of gemma4:31b per arm, negative control included:

arm	node_1 IM2COL	gemma4:31b ×10
base	f16	[4,4,4,4,3] ×10
907	f32	[4,4,4,4,3] ×10

Zero variance, thirty captures, and the promote fires once per load in both arms. The dtype moves; quality does not. Numbers and method on #349.

So: harmful on ROCm, neutral on Metal. Your close was right, and my argument against it was pointing the same direction my two earlier errors did — toward "this is worth adopting". Recording it here rather than letting it sit as an unanswered case for the patch.