Repository navigation
ggml-hrx: PrismML tile bytes for the low-token SwiGLU (PQ2_0 1.7B decode) - #74
Merged
bong-water-water-bong merged 1 commit intoOct 2, 2026
Conversation
…ode) ggml_dequant_weight_tile_bytes had no case for PQ2_0 (format 72) or PTQ1_0 (73) and returned 0. #62 added both to ggml_dequant_weight_row_bytes but not here. The low-token SwiGLU (ggml_mul_mat_swiglu_f32_f32_lowtoken_dot, 1-5 tokens) sizes a weight row as K/256 x tile bytes, so every row read row 0's weights and the output collapsed ("red color color color the the the"). PQ2_0 is two 34-byte blocks per 256 values (68), PTQ1_0 two 28-byte blocks (56). Only models whose decode SwiGLU reaches that kernel were hit. On Ternary-Bonsai-1.7B, the hidden-2048 qwen decode norm publishes a q8_1 copy of its output, kquant decode steps aside, and the gate/up pair lands in the low-token SwiGLU. The 27B files and TQ1_0/TQ2_0 took kquant decode. Decode-shaped llama-perplexity (-c 128 -b 128 -ub 1, wikitext-2): Ternary-Bonsai-1.7B PQ2_0, 4 chunks: HRX0 before PPL 1,225,624 +/- 189,778 HRX0 after PPL 20.6537 +/- 4.797 CPU PPL 20.6972 +/- 4.810 Ternary-Bonsai-2-27B PTQ1_0, 2 chunks, low-token SwiGLU forced (GGML_HRX_DISABLE_DISPATCH=kquant.swiglu): HRX0 before PPL 18,534.8; after 11.1754 (default routing 11.1756) Greedy, "The three primary colors of light are": before " red color color color color theit the the ..."; after " red, green, and blue. These colors are used to create all other colors through mixing." (= CPU). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ggml-hrx: PrismML tile bytes for the low-token SwiGLU (PQ2_0 1.7B decode)
ggml_dequant_weight_tile_bytes had no case for PQ2_0 (format 72) or PTQ1_0
(73) and returned 0. #62 added both to ggml_dequant_weight_row_bytes but not
here. The low-token SwiGLU (ggml_mul_mat_swiglu_f32_f32_lowtoken_dot, 1-5
tokens) sizes a weight row as K/256 x tile bytes, so every row read row 0's
weights and the output collapsed ("red color color color the the the").
PQ2_0 is two 34-byte blocks per 256 values (68), PTQ1_0 two 28-byte blocks
(56).
Only models whose decode SwiGLU reaches that kernel were hit. On
Ternary-Bonsai-1.7B, the hidden-2048 qwen decode norm publishes a q8_1 copy
of its output, kquant decode steps aside, and the gate/up pair lands in the
low-token SwiGLU. The 27B files and TQ1_0/TQ2_0 took kquant decode.
Decode-shaped llama-perplexity (-c 128 -b 128 -ub 1, wikitext-2):
Ternary-Bonsai-1.7B PQ2_0, 4 chunks:
HRX0 before PPL 1,225,624 +/- 189,778
HRX0 after PPL 20.6537 +/- 4.797
CPU PPL 20.6972 +/- 4.810
Ternary-Bonsai-2-27B PTQ1_0, 2 chunks, low-token SwiGLU forced
(GGML_HRX_DISABLE_DISPATCH=kquant.swiglu):
HRX0 before PPL 18,534.8; after 11.1754 (default routing 11.1756)
Greedy, "The three primary colors of light are": before " red color color
color color theit the the ..."; after " red, green, and blue. These colors
are used to create all other colors through mixing." (= CPU).
Reviewed (PR-Agent duty): tile bytes match the row-bytes path (34 / 28 bytes per 128-value block, so 68 / 56 per 256). Found by the release format-matrix smoke test.
🤖 Generated with Claude Code