Skip to content
This repository was archived by the owner on Oct 8, 2026. It is now read-only.

quantize : choose quant types to hit a bpw or file size target (upstream #15550) - #35

Open
dzannotti wants to merge 1 commit into
masterfrom
halo/quantize-target-size
Open

dzannotti wants to merge 1 commit into
masterfrom
halo/quantize-target-size

Conversation

@dzannotti

@dzannotti dzannotti commented Oct 7, 2026 •

Copy link
Copy Markdown

Stacked on #34 because it needs the GGUF imatrix statistics from upstream ggml-org#14891, which #34 brings in. Once #34 is merged and its branch deleted, GitHub retargets this PR to master.

Port of ggml-org#15550 (EAddario, still open upstream) at 839a43500. It adds --target-bpw, --target-size, --state-file and --target-exclude to llama-quantize. For each tensor it picks the quant type that reaches the overall bpw or file-size target at the lowest estimated error, using the activation statistics that ggml-org#14891 writes into GGUF imatrices.

Measured

Qwen3-1.7B from Q8_0 (--allow-requantize). The imatrix comes from docs/ (60×512 tokens). Perplexity is on held-out tools/ + examples/ READMEs (60×512 tokens), Vulkan:

model size BPW PPL
Q8_0 (source) 2059 MiB 8.50 6.4084 ± 0.166
Q4_K_M + imatrix 1217 MiB 5.03 6.4925 ± 0.169
--target-bpw 5.0262 1217 MiB 5.03 6.4753 ± 0.168
--target-bpw 4.9 1187 MiB 4.90 6.5327 ± 0.170
--target-size 900m 894 MiB 3.69 6.8358 ± 0.179

Targets are hit precisely. At Q4_K_M's exact size the chosen mix loses about 20% less perplexity versus Q8_0 than Q4_K_M does. That is inside the error bars on this small model and calibration set, so treat it as encouraging rather than proven. Reusing --state-file brings a re-target down from minutes to 11 s.

…eam ggml-org#15550)

Port of ggml-org#15550 (EAddario, still open upstream) at
839a435, which adds --target-bpw, --target-size, --state-file and
--target-exclude: llama-quantize picks per-tensor quant types that reach the
target at the lowest estimated error, using the activation statistics that
ggml-org#14891 writes into GGUF imatrices.

Applied as its diff against its upstream base cb7934c, minus the PR's
stray uv.lock. In common/imatrix-loader.* the PR's stats loading is merged
with ggml-org#14891's n_layer_nextn.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

documentation Improvements or additions to documentation examples

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant