ggml-openvino: add opt-in exact zero points for asymmetric weights

[osabnis]

Overview

The OpenVINO backend converts asymmetric quantized weights (Q4_K, Q4_1, Q5_1, and non-requantized Q5_K) to u4/u8 weights with an f16 scale and an integer zero point, round(m / d). Rounding the zero point shifts every weight in a group in the same direction by up to half a quantization step. For Q4_K on Gemma-4 and Llama, this corresponds to about 8.3% relative weight error and is the primary source of the accuracy gap between the OpenVINO GPU backend and the CPU backend.

This PR adds GGML_OPENVINO_EXACT_ZERO_POINT. When enabled, asymmetric quantized weights retain the exact f16 zero point, m / d, matching the existing path used for MoE expert weights. The graph remains a Subtract followed by a Multiply, so the fully-connected weight compression pattern is unchanged.

A new helper, ggml_openvino_use_exact_zero_point(), replaces the expert-only check in the three places that must agree on the weight layout: the buffer layout (f16 zero-point slot), process_weight_tensor, and extract_quantized_weights. The setting is also registered in the environment registry, model-cache settings, and docs/backend/OPENVINO.md.

The option is off by default, so default behavior and output are unchanged.

Accuracy

KL divergence of the next-token distribution against the CPU backend at context 1024 using GGUF Q4_K_M on Arc B390. Decode uses one token per batch; prompt uses 1024-token batches. OpenVINO results use master with f16 zero points enabled on the oneDNN path (see Additional information). Vulkan results use ggml-org b11236 on the same system.

Model	Decode default	Decode exact	Decode Vulkan	Prompt default	Prompt exact	Prompt Vulkan
Gemma-4 E2B	0.3634	0.0196	0.0187	0.3674	0.0251	0.0237
Llama-3.2 1B	0.0412	0.0019	0.0014	0.0539	0.0028	0.0020
SmolLM2 1.7B	0.0731	0.0103	0.0092	0.0934	0.0156	0.0143
Phi-3.5 mini	0.0312	0.0048	0.0040	0.0430	0.0066	0.0074
Phi-4 mini	0.0357	0.0108	0.0030	0.0480	0.0121	0.0051
Qwen3.5 4B	0.0250	0.0032	0.0039	0.0294	0.0034	0.0035

For decode, the percentage of cases where the OpenVINO backend selects the same top token as the CPU backend improves substantially with exact zero points:

• Gemma-4: 78.8% → 94.8% (Vulkan: 95.0%)
• Llama-3.2: 91.0% → 97.8% (98.1%)
• SmolLM2: 87.7% → 94.1% (95.6%)
• Phi-3.5: 93.1% → 96.9% (97.8%)
• Phi-4: 90.3% → 93.9% (97.2%)
• Qwen3.5: 94.1% → 97.9% (98.1%)

Phi-4 mini still has a gap relative to Vulkan because its output layer is requantized to 8-bit; that path is outside the scope of this PR.

Why it is opt-in

With current OpenVINO releases, the GPU plugin does not accept an f16 zero point on its oneDNN fully-connected path, so many of these layers fall back to an OpenCL kernel (130 of 261 fully-connected operations in a Gemma-4 decode step).

On OpenVINO master without the oneDNN and GPU-plugin changes described below, enabling exact zero points results in:

• Prompt processing (pp512): 67% slower on Gemma-4 and 61% slower on Llama.
• Decode at 1k context: 6.6% slower on Gemma-4 and 8.5% slower on Llama.
• GPU memory: +2.1 GB on Gemma-4 and +1.2 GB on Llama.
• Prompt accuracy: no improvement (Gemma-4 KL 0.367 → 0.374); the improvement is primarily seen during decode.

With f16 zero-point support in oneDNN and the GPU plugin, all fully-connected layers remain on the oneDNN jit:gemm path. For example, all 261/261 fully-connected operations in a Gemma-4 decode step use oneDNN. With that support, exact zero points cost 1–2% of decode throughput at 1k context, prompt processing stays within measurement noise, and peak GPU memory grows by at most 0.23 GB.

Testing

• test-backend-ops test on GPU: 8024/8024 passed with the switch both disabled and enabled.
• Verified that default-mode decode KL divergence matches the branch base within normal GPU run-to-run variation across all six models. SmolLM2, Phi-3.5, and Phi-4 were identical to six digits; Llama and Qwen3.5 differed only in the fifth significant digit; Gemma-4 measured 0.36351 versus 0.36340–0.36380 across two base runs.
• Re-measured the accuracy results after rebasing onto the current branch.
• Verified the exact-zero-point path on the OpenVINO CPU device: Llama-3.2 1B decode KL improved from 0.0407 to 0.0019.

Additional information

The GPU performance and prompt-accuracy results above require two changes outside llama.cpp:

• oneDNN: f16 weight zero-point support in matmul (RFC: uxlfoundation/oneDNN#6218)
• OpenVINO: allow f16 zero points on the oneDNN fully-connected path PR:38696

Once an OpenVINO release contains both changes, GGML_OPENVINO_EXACT_ZERO_POINT can be considered for enabling by default.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: Yes — AI was used for analysis and testing.

[osabnis]

Note on merging: The switch is off by default, so the default behavior and output remain unchanged. This can therefore be merged as soon as it is approved.

Enabling it by default should wait until an OpenVINO release includes both the oneDNN support for f16 zero points and the corresponding GPU plugin change. Without those changes, enabling the switch slows down prompt processing without providing a prompt-accuracy improvement (see the numbers in the description).