Skip to content

Pin llama.cpp c9283cf: HRX ZAYA decode 67 -> ~90 tok/s (fork #36-#40) - #232

Merged
bong-water-water-bong merged 2 commits into
mainfrom
pin-hrx-zaya-decode
Sep 29, 2026
Merged

bong-water-water-bong merged 2 commits into
mainfrom
pin-hrx-zaya-decode

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Moves third_party/llama.cpp 45c1770 -> c9283cf (fork #36-#40). Regenerates the registry, and adds a docs/hrx.md entry.

ZAYA1-8B HRX0 decode 67 -> ~90 tok/s. Each step was measured in interleaved A/B runs on strixhalo:

Each new .loom file carries a reference kernel and check.case differentials that pass under iree-test-loom --sanitizer=access.

RFC #213 decode gate (tg128, HRX = this pin, Vulkan = weekly build, interleaved):

model HRX Vulkan HRX/Vulkan
Qwen3-0.6B Q4_K_M 320.8 356.4 90.0% (at the edge)
Qwen3-Coder-30B-A3B Q4_K_M 90.5 93.8 96.5%
ZAYA1-8B Q4_K_M 89.5 93 (fork Vulkan build) ~96%
Qwen3.8-27B UD-Q4_K_XL 7.25 12.48 58%, far off

The 27B is the open gap. HRX also draws 150 W and peaks at 91 °C, against Vulkan's 91 W and 59 °C. This UD file is mostly Q5_K and IQ4_XS, which miss HRX's Q4_K/Q6_K token-1 fast paths. Routing the generic WMMA matmuls to the decode GEMV only reaches 7.55 tok/s, so it needs its own profile (it's a hybrid with gated delta-net layers). Separately, the HRX IQ4 decode fix (fork branch 1bit/hrx-iq4-fix) isn't merged yet.

🤖 Generated with Claude Code

bong-water-water-bong and others added 2 commits September 29, 2026 14:57
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@context7

context7 Bot commented Sep 29, 2026

Copy link
Copy Markdown

Docs7 for 1bit-monster/engine

Result Status Action
Deployment ➖ Not used —
Content review ➖ Did not run. This site has no agent runs available this month. Wait for the monthly reset or check your Docs7 plan. —

Commit 901446d

@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

36 - Partially compliant

Compliant requirements:

  • The site template and layout are implemented
  • New blog posts are added
  • Blog cleanup with archive.tsv is included
  • copyright.py and pages.yml changes are included

Non-compliant requirements:

  • The specific implementation details of the site's CSS, header, and footer are not visible in the diff
  • The exact structure of the pages (home, overview, docs, blog, etc.) is not visible in the diff

Requires further human verification:

  • Visual verification of the site's appearance and layout
  • Confirmation that all pages are correctly implemented and styled

40 - Partially compliant

Compliant requirements:

  • The Laya commit has been updated in the diff

Non-compliant requirements:

  • The scripts/fetch-laya.sh verification and Laya tests are not visible in the diff

Requires further human verification:

  • Confirmation that scripts/fetch-laya.sh was run and verified
  • Confirmation that Laya tests were run and passed

39 - Partially compliant

Compliant requirements:

  • The zinc commit has been updated in the diff

Non-compliant requirements:

  • The build-zinc.sh and test commands are not visible in the diff

Requires further human verification:

  • Confirmation that build-zinc.sh was run and tested on Strix Halo
  • Confirmation that the specific model and prompt test was performed
⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Documentation Clarity

The documentation for the ZAYA decode fusions is detailed but could benefit from clearer explanation of how the performance improvements were measured and validated. Specifically, the mention of interleaved A/B runs and the reference to iree-test-loom --sanitizer=access should be clarified to ensure reproducibility.

- **ZAYA decode fusions (llama.cpp #36-#40).** ZAYA1-8B HRX0 decode 67 -> ~90 tok/s (Vulkan 93),
  each step measured in interleaved A/B runs:
  - #36 skips CONTs of fresh contiguous results in `src/models/zaya.cpp` (views keep theirs: on HRX
    their consumers take slower strided paths), bitwise the same output: +2.4%.
  - #39 runs the CCA convolution (Q/K concat, conv-state update, depthwise conv, grouped 2-tap conv
    and biases, ~35 graph nodes) as one dispatch (`zaya.cca_conv.decode_f32`): 72 -> 81 tok/s.
  - #40 runs the CCA query/key mixing and per-head norms (~30 nodes) as one dispatch
    (`zaya.cca_qk_norm.decode_f32`): 81 -> 90 tok/s.
  #39 and #40 accumulate in their own order, so output is not bitwise the same; teacher-forced top-1
  against the FP32 model stays 69/96 and batch-1 perplexity moves 33.31 -> 33.49 -> 33.45 (CPU backend
  34.30). Both match decode only (one token, one sequence). #37 and #38 remove runtime integer
  division from #35's and #32's kernels, which newer Loom rejects. Each new `.loom` file carries a
  reference kernel and `check.case` differentials that pass under `iree-test-loom --sanitizer=access`.
  `GGML_HRX_DISABLE_DISPATCH=zaya.cca` turns the two ZAYA kernels off for comparison.

@bong-water-water-bong
bong-water-water-bong merged commit 0311090 into main Sep 29, 2026
11 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the pin-hrx-zaya-decode branch September 29, 2026 18:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant