Skip to content

HRX: wrong answers for every prompt longer than one ubatch since fork #82 (graph inputs written back to host) #319

Closed

Description

@bong-water-water-bong

Found 2026-10-05 while root-causing #313 (strixhalo, performance mode). On the llama.cpp fork tip (e57beb9 = the pin merged in #311), Qwen3-0.6B Q4_K_M on HRX0 answers a 4,798-token prompt by repeating the question — 28/28 runs, same hash (8d294327), with and without the private add-on. Short prompts (one ubatch, ≤512 tokens) are unaffected, which is why the e2e and "Paris." gates passed.

Cause. qwen_attention_metadata binds the positions, the K/V cache indices and the mask as read_write, so they are queued as deferred host writebacks. Since fork #82 (e0f637b) every upload first publishes pending writebacks — which overwrites the next ubatch's fresh inputs with the previous ubatch's values. Every multi-ubatch Qwen prompt is affected.

Fix. Fork commit "ggml-hrx: never write a device-rewritten graph input back to host memory": a graph_input flag on such bindings — uploaded, never downloaded. Same PR carries "wait for a non-replay command program that reads registered host memory" (graph replay off answered "123" instead of 7341: the non-replay path returned before its own device-side input copy ran, and ggml overwrote the host buffer).

The NaN in #313 itself is a separate, third bug (Loom compiler GFX11 wave64 SGPR lane-mask hazard; see #313).

After the fork PR merges the engine pin moves; until then --device hrx with prompts over 512 tokens is wrong on main.

Activity

  1. bong-water-water-bong commented on Oct 5, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Fixed: graph inputs are never written back to host memory (fork #90, b4e7515), in the engine since llama.cpp pin 522dab47 (#320, main 11d6e2f). Verified on strixhalo at that pin: Qwen3-0.6B, 4,798-token needle prompt after a short request, 20/20 correct (hash 5fb9f8b5, names 7341), graph replay on and off identical.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions