Skip to content

ggml-hrx: HIP kernels through HRX (plumbing + add-on hook) - #86

Merged
bong-water-water-bong merged 4 commits into
1bit/hrx-vulkan-patchedfrom
1bit/hrx-hip-plumbing
Oct 5, 2026
Merged

bong-water-water-bong merged 4 commits into
1bit/hrx-vulkan-patchedfrom
1bit/hrx-hip-plumbing

Conversation

@bong-water-water-bong

Copy link
Copy Markdown

Lets HRX run hipcc code objects as ggml-hrx kernels, beside the Loom kernels, and lets an external kernel add-on plug in at build time. Draft: do not merge yet. The engine release is on hold.

  • 616034b HIP plumbing:
    • Build: hip/ggml-hrx-hip.cmake compiles hip/kernels/*.hip with amdclang++ into raw code objects per GGML_HRX_HIP_TARGETS (default gfx1151) and embeds them in the library.
    • Runtime: hip-kernel-registry (a HipKernel description becomes a KernelDefinition in family hip), and hip-kernel-loader (the executable cache loads the embedded object instead of a Loom compile).
    • Matchers: registered in one place, hip-dispatches.cpp.
    • Docs and tests: README.md with the recipe; example hip_scale_f32 with its matcher (opt-in GGML_HRX_HIP_EXAMPLE_SCALE=1); the ggml-hrx-hip-smoke tool.
    • AMD files: one-line hooks only.
  • 8719cf4 add-on hook:
    • GGML_HRX_HIP_ADDON_DIR=<dir> adds <dir>/kernels/*.hip to the code-object build and compiles <dir>/*.cpp/*.h into ggml-hrx with GGML_HRX_HIP_ADDON defined.
    • It includes <dir>/addon.cmake if present and calls ggml_hrx_hip_addon_register(registry).
    • Generic attention-sink extension point: set_hip_attention_sink_hook. The dispatch falls back to Loom when no hook is set.
  • d5de4b1: three MUL_MAT_VEC_FUSION cases (one-token Q4_K/Q5_K gate/up + SwiGLU at outputs 128/384).

Checks on strixhalo (gfx1151), performance mode:

  • Without an add-on: ggml-hrx-hip-smoke PASS; test-backend-ops -o SCALE -b HRX0 4/4 on 3 runs with hip.scale_f32 matched, and 4/4 with the example off.
  • With an add-on, against the same kernels built in-tree:
    • the same matchers register (identical counts) on Qwen3-8B, GLM-4.7-Flash, ZAYA1-8B and Qwen3.6-35B;
    • llama-bench pp512/tg128 are within noise;
    • MUL_MAT 325/325, MUL_MAT_ID 108/108, MUL_MAT_VEC_FUSION 57/57, CONCAT 6/6.

🤖 Generated with Claude Code

bong-water-water-bong and others added 3 commits October 4, 2026 21:25
Each ggml/src/ggml-hrx/hip/kernels/*.hip is compiled by amdclang++ into a
raw code object per GGML_HRX_HIP_TARGETS entry (default gfx1151) and
embedded in the library. HRX loads it with hrx_executable_load_data, the
same call the Loom JIT result goes through. Neither the HIP runtime nor
ggml-hip is used.

- hip-kernel-registry: a HipKernel (bindings, uint32 launch parameters,
  workload parameters, launch-geometry function) becomes a
  KernelDefinition in family "hip". resolve_kernel_definition finds it
  there.
- hip-kernel-loader: KernelExecutableCache::get_or_compile takes the
  embedded code object and the launch geometry in place of a Loom JIT
  compile.
- hip-dispatches.cpp: the one place HIP matchers are registered.
- Example: hip_scale_f32 + dispatch-hip-scale.cpp (GGML_OP_SCALE),
  opt-in with GGML_HRX_HIP_EXAMPLE_SCALE=1.
- ggml-hrx-hip-smoke: loads, dispatches and checks a code object using
  libhrx only.
- hip/README.md: the recipe.

Upstream files get one-line hooks only (CMakeLists include, the resolve
fallback, the get_or_compile branch, a friend declaration and the
dispatch registration call).

Checks on strixhalo (gfx1151), performance power mode (120 W):
- ggml-hrx-hip-smoke, 6 runs: axpy 0/1000003 mismatches, LDS + wave32
  shuffle block sum max relative error 0 over 3907 blocks. Pointer args,
  by-value args and hidden kernargs (blockDim/gridDim) all work.
- test-backend-ops -o SCALE -b HRX0 with the HIP matcher, 3 runs: 4/4
  passed each run. GGML_HRX_LOG_DISPATCH shows hip.scale_f32 matched.
- ZAYA1-8B Q4_K_M llama-bench -fa 1 -p 512 -n 128 -r 3, two
  interleaved rounds against the same tree without this change:
  pp512 2037.7/2050.4 before vs 2052.3/2053.3 after; tg128 93.0/93.0 vs
  92.9/92.7. No regression with HIP matchers off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Kernels and matchers can live outside the tree. With
-DGGML_HRX_HIP_ADDON_DIR=<dir>:
- <dir>/kernels/*.hip are compiled and embedded with hip/kernels/*.hip
  (same targets and flags, looked up by stem; configure fails on a
  repeated stem);
- <dir>/*.cpp and *.h are compiled into ggml-hrx with GGML_HRX_HIP_ADDON
  defined, and register_hip_dispatches calls the add-on's
  ggml_hrx_hip_addon_register after the matchers in hip/;
- <dir>/addon.cmake is included if present (extra tools).
Without the option the build is unchanged: only the example matcher.

The attention-sink rescale that follows FlashAttention gets a generic
hook (set_hip_attention_sink_hook): an add-on kernel can take the
rescale, otherwise the Loom kernel runs as before.

README: the add-on layout, and the registry trap that every Fused
registration of a root op is tried before any SingleOp one regardless
of priority.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Output sizes 128 and 384, K 256/1024/2560, for HRX HIP kernels that
take output % 128 == 0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lities)

supports_op only hands a node to the dispatcher when eager_capability_declared lists its op,
so a HIP matcher for an op outside that list never fired: the node went to another backend.
hip_declare_eager_op(op), called from a matcher's register_* function while the registry is
built, adds the op to a set that eager_capability_declared consults for every op it does not
know (one line in its default branch). hip_eager_op_declared builds the registries on first
use, so the set is filled before the scheduler's first probe. Without a declaring matcher the
set is empty and the build behaves as before; the public tree names no add-on op.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Author

Added 3527b56 ggml-hrx: HIP matchers can declare the ops they claim (hip/hip-capabilities).

supports_op only hands a node to the dispatcher when eager_capability_declared lists its op, so a HIP matcher for an op outside that list could never fire. New hip/hip-capabilities.{h,cpp}: a matcher calls hip_declare_eager_op(op) from its register_* function while the registry is built; eager_capability_declared consults the set in its default branch (one line in ggml-hrx.cpp). hip_eager_op_declared builds the registries on first use so the set is filled before the scheduler's first probe. With no declaring matcher the set is empty and nothing changes; the public tree names no add-on op.

@bong-water-water-bong
bong-water-water-bong marked this pull request as ready for review October 5, 2026 11:20
@bong-water-water-bong
bong-water-water-bong merged commit 001832c into 1bit/hrx-vulkan-patched Oct 5, 2026
10 of 26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation ggml testing

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant