Repository navigation
ggml-hrx: GET_ROWS of IQ3_S and IQ4_NL rows on HRX (embeddings stay on HRX) - #77
bong-water-water-bong wants to merge 2 commits into
Conversation
The get_rows kernel already hands the IQ4_NL table and the IQ3_S grid to the shared dequantizer, but both types were refused twice: in the GET_ROWS matcher and in device_supports_op. IQ4_NL decodes correctly since #41 (no XOR 12 table remap). With both refusals gone, test-backend-ops GET_ROWS passes for the two-dimensional cases of both types (three runs, error <= 1e-7, the same cases q4_0 and q4_K pass), so embeddings stored as IQ3_S or IQ4_NL no longer go to the CPU. The separate refusal of batched IQ3_S sources (wrong rows) stays. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Twelve rows of 1024 random values per type, quantized by ggml_quantize_chunk at scales 2^-6..2^5. Every dequantized value is an f16 scale times small integers, so HRX's GET_ROWS must equal dequantize_row_iq4_nl and dequantize_row_iq3_s bit for bit, on the HRX device with no scheduler, through the get_rows kernel. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Review (PR-Agent duty): approved, merging after Sunday's release (stacked on #76). Removing the blanket IQ3_S/IQ4_NL GET_ROWS refusals in the matcher and in device_supports_op is right, because the kernel already decodes both tables; the batched-IQ3_S refusal stays. test-hrx-get-rows-iq is bit-exact for both types; Qwen3-0.6B goes from 2 graph splits to 1 at baseline KLD. The ggml-hrx.cpp change is a 4-line in-place deletion in AMD's file with its notice kept, which is fine under our fork rules. Stack at 97fcffe: test-backend-ops 1129/1129. |
|
Kept: format support in the shared dequantizer / GET_ROWS is plumbing, not a new kernel (Q2_0 is the 2-bit format the ternary goal uses). Waits for the release hold like everything else; needs a rebase onto f95f2db before merge. |
1 participant
GET_ROWS of IQ3_S and IQ4_NL rows on HRX0, so token embeddings stored in either type stay on HRX. Stacked on the Q2_0 PR (on #75).
Both types were refused twice, in the GET_ROWS matcher and in
device_supports_op, even thoughget_rows_f32already passes the IQ4_NL table and the IQ3_S grid to the shared dequantizer. IQ4_NL decodes correctly since #41. This removes both blanket refusals. The separate refusal of batched IQ3_S sources ("wrong rows") stays.Commits
1198e6f0: the two refusals removed (small in-place edits;ggml-hrx.cppkeeps its notice).97fcffe5:tests/test-hrx-get-rows-iq, 12 x 1024 random rows per type, quantized byggml_quantize_chunkat scales 2^-6..2^5. HRX must equaldequantize_row_iq4_nl/dequantize_row_iq3_sbit for bit (every value is an f16 scale times small integers).Validation, gfx1151:
The KLD matches HRX's baseline on this model (0.0032 with Q4_0 or NVFP4 projections). The remaining "not supported" cases are the same view/batched shapes q4_0 and q4_K show. Full suite on this branch: 1129/1129.
Not for the release pin.
🤖 Generated with Claude Code