rpc: avoid serializing buffers from other servers by hmirin · Pull Request #26500 · ggml-org/llama.cpp

hmirin · 2026-08-03T07:01:28Z

Overview

Goal: Support multiple llama.cpp RPC servers from a single client.
Bug: RPC buffer pointers were serialized even when they referred to buffers owned by a different RPC server, resulting in invalid pointers on the receiving server.
Fix: Include a remote buffer pointer only when the buffer belongs to the RPC socket receiving the graph.
Test: Add a two-server regression test for cross-server tensor serialization.

The bug caused the following error:

[create_node] invalid data ptr
[graph_compute] failed to create graph node 0

Additional Information

Complete reproduction gist

curl -fsSL https://gist.githubusercontent.com/hmirin/64cd976fe04e4a4b5405641c72647e10/raw/reproduce.sh -o reproduce.sh
bash reproduce.sh

Related changes:
- #21006: Reports the related invalid data ptr RPC error.
- #21030: Fixes Eval bug: PR20908 breaks rpc-server functionality when balancing split a model across multiple machines. #21006 for CPU and other non-RPC buffers. (Merged)
This PR applies the same zero-pointer serialization behavior as rpc : proper handling of data pointers to CPU buffers #21030 to RPC buffers owned by a different server.

Requirements

I have read and agree to the contributing guidelines.
AI usage disclosure: YES. The fix was created with Codex 5.6 Sol and Claude Code Fable. Fix is reviewed by the author and PR comment is written by the author.

himorishige · 2026-08-10T10:45:21Z

Tested this on 2x DGX Spark (GB10, sm_121, arm64, CUDA 13.0, driver 580.159.03) — it fixes the failure here.

Before the patch, both master (74ce157) and the rpc_tensor branch (db5abb4) abort as soon as the first graph reaches the remote worker:

[create_node] invalid data ptr[graph_compute] failed to create graph node 3085

Only the remote worker dies; the loopback worker on the client machine logs nothing unusual, and a single-worker setup never reproduces it — which lines up with the ownership problem described here.

With f0c483c applied on both nodes, two workers (one remote over a 200Gb QSFP direct link with RoCEv2 RDMA, one loopback) complete:

model	size	pp512	tg128
deepseek4 IQ2_M	84.68 GiB	338.36	20.19
deepseek4 MXFP4 MoE	144.44 GiB	369.69	16.49

The 144 GiB quant does not fit in a single node's 121 GiB of unified memory, so more than one RPC worker is the only way to run it at all. That is the case this patch unblocks for me.

Beyond llama-bench: llama-server over the same two workers survives repeated requests (with --cache-ram 0 from #26529), a ctx ladder of 2048/8192/32768 runs clean, and a 12-slot concurrency sweep finished with zero failed requests.

Same signature is reported in #26820; I left the reproduction details there.

dwhoop55 · 2026-08-12T11:00:32Z

I had the same issue ([create_node] invalid data ptr[graph_compute] failed to create graph node) when trying to run DeepSeek V4 Flash and Kimi K3 on multiple CPU-only RPC nodes (with RDMA transport; even thought it should be the same with TCP).

ggml-org/llama.cpp:master with this PR fixes the issue for DeekSeek V4 Flash and the unslothai/llama.cpp:kimi-k3-text-base with manually merging this PR does also fix running Kimi K3 on multiple CPU-only RPC nodes in my tests.

ludwigjossieaux · 2026-08-17T22:58:55Z

Confirming this also fixes GLM-5.2 multi-node RPC crashes (#26583), deterministic invalid data ptr at graph node 30 before, fully working after, on a mixed SM120 head + SM121 workers cluster with RDMA. MTP-over-RPC works too. Nice fix, thanks

fenneclabs · 2026-08-18T19:17:46Z

I have the same problem with DS4 REAP with RTX 4060Ti + 2x BC250 over RPC all on Vulkan.

@hmirin maybe resolving the branch conflict could help get this PR merged? Thanks for the fix :)

hmirin · 2026-08-21T16:34:43Z

Updated to keep up with the main branch.

Only include remote buffer pointers when the buffer belongs to the RPC dispatcher receiving the graph. Add a two-server regression test for cross-server tensor serialization. Assisted-by: Codex

[no ci]

68 upstream commits, 9 conflicted files. Brings qwen4exp fixes needed for Qwen3.8-Flash-Next: recurrent state rollback (ggml-org#28123), seq_cp/block position keying/mtmd input (ggml-org#27941), indexer head slicing (ggml-org#28023), and the central n_layer_nextn load (ggml-org#28159). Resolutions: - ggml-rpc.cpp: resolved entirely to the fork side again. Upstream's rpc_dispatcher/proto-6.0 port stays deferred; the fork keeps 5.1.3. Note that upstream ggml-org#26500 now implements the same foreign-endpoint skip in serialize_graph that the fork carries, so that one is a candidate to drop when the dispatcher port finally happens. - llama-graph.cpp: adopted upstream's fused ggml_swiglu_clamp for both the dense and MoE clamp paths and kept the fork's LLM_ARCH_GLM5NEXT in the condition. The fused kernel is min(gate,limit) / clamp(up,+-limit) / swiglu_split, i.e. bit-identical to the three-op sequence it replaces, and Vulkan implements it, so the boards get it too. Costs the ffn_*_clamped cb() trace names, same tradeoff upstream took for DSV4. - speculative.cpp: upstream ggml-org#27310 folded the DFlash encoder into the injection decode and deleted features_buf. The fork's non-finite feature scrub moves onto batch_inject.embd in place; the last_tap_nextn tap automerged. - dflash.cpp: kept the fork's Laguna kv_inp norm and deferred gated o_proj, layered on upstream's newly-populated wk_s/wv_s/wo_s scale arguments (previously NULL). The gated path latches wo_s alongside wo. - nemotron-h.cpp: dropped the now-duplicate NEXTN_PREDICT_LAYERS get_key, kept the fork's stricter assert and n_layer_nextn_per_head derivation. - llama-model-loader.cpp: kept the fork's rpc_preloaded skip ahead of upstream's use_mmap || lazy.has(cur) predicate. - tests: kept both sets of flash-attn cases; ported the fork's GLM5NEXT test_dsa_kpool/test_mtp onto upstream's int-verbosity logging convention. Windows CUDA build green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U3H5motr51eTWujztSXykc

…6.0.4) ggml-rpc.cpp is rebuilt from upstream's file: rpc_dispatcher (one worker per endpoint, message queue, futures for events) is the base, and the fork's features are layered on it as closure messages instead of a parallel rpc_stream implementation. Client contexts hold a shared_ptr<rpc_dispatcher> as upstream does; dispatchers are process-lifetime and survive a detach/reattach with their sockets dropped, so read ordinals stay monotonic without the old base folding. Kept, re-layered: full-duplex lanes (each lane its own dispatcher adopted onto a LANE_ATTACHed socket) and LANE_FENCE counting, the star cpy_tensor_async with real cross-endpoint event_wait, bf16 wire, the multi-slot graph cache with stable/quick/split uids and GRAPH_FORGET, peer push (async and the sync-client SPD path), sessions and hibernation, chunked cache reads and streamed SET_TENSOR, cache-limit LRU and loader preflight exports, IMATRIX_SQSUM, read ordinals, connect retry, and the whole server side (acceptor thread, lane serving, parked sessions). GGML_RPC_ASYNC=0 becomes the dispatcher's direct mode: messages execute inline on the caller, which is what the SPD stage pipeline measured as necessary. Gone: the rpc_stream/gate/event trio, send coalescing (measured a wash), the quiescent-stream ping (synchronize is a queue drain, as upstream), the foreign-endpoint node skip's private code path (kept as a node filter next to upstream's ggml-org#26500 descriptor blanking), and the CMD_STATS / WIRE_TRACE / TRACE_XDEV / GRAPH_OPS / QUICK_UID_OFF / SPLIT_UID_OFF diagnostics knobs. FREE_BUFFER, BUFFER_CLEAR, MEMSET_TENSOR and INIT_TENSOR carry no response, as in upstream 6.0. Wire is 6.0 with the fork's patch level 4; GRAPH_COMPUTE and GRAPH_RECOMPUTE carry a uid, so every daemon redeploys together. Unbuilt: verification is the temperature-0 text diff on a multi-board config once the fleet is rebuilt. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N6aMx4briWYDmJDphXuXFp

* rpc: avoid serializing buffers from other servers Only include remote buffer pointers when the buffer belongs to the RPC dispatcher receiving the graph. Add a two-server regression test for cross-server tensor serialization. Assisted-by: Codex * cont : add ref --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

Keep USB4 RPC, Flash-Next NextN, DFlash LM-head gather, meta shard guards, HIP TOP_K supports_op, and AVX-512 AllReduce. Take upstream radix TOP_K (ggml-org#27466) and RPC dispatcher check (ggml-org#26500) combined with view-src tensor data offsets.

* rpc: avoid serializing buffers from other servers Only include remote buffer pointers when the buffer belongs to the RPC dispatcher receiving the graph. Add a two-server regression test for cross-server tensor serialization. Assisted-by: Codex * cont : add ref --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

hmirin requested review from a team and ggerganov as code owners August 3, 2026 07:01

github-actions Bot added testing Everything test related examples ggml changes relating to the ggml tensor library for machine learning labels Aug 3, 2026

hmirin force-pushed the fix/rpc-multi-server-buffer-ownership branch from 6e24f65 to f0c483c Compare August 8, 2026 14:15

lich-77 mentioned this pull request Aug 17, 2026

RPC: GLM-5.2 crashes on multi-node CUDA RPC - invalid data ptr / graph_compute failed #26583

Open

hmirin force-pushed the fix/rpc-multi-server-buffer-ownership branch from f0c483c to e144634 Compare August 21, 2026 16:32

rpc: avoid serializing buffers from other servers

41d0d5f

Only include remote buffer pointers when the buffer belongs to the RPC dispatcher receiving the graph. Add a two-server regression test for cross-server tensor serialization. Assisted-by: Codex

hmirin force-pushed the fix/rpc-multi-server-buffer-ownership branch from e144634 to 41d0d5f Compare August 29, 2026 13:37

ggerganov approved these changes Aug 30, 2026

View reviewed changes

ggerganov reviewed Aug 30, 2026

View reviewed changes

Comment thread ggml/src/ggml-rpc/ggml-rpc.cpp

ggerganov added the merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge. label Aug 30, 2026

cont : add ref

a8f2e97

[no ci]

ggerganov merged commit a7cc83b into ggml-org:master Aug 30, 2026
1 check passed

liminfei-amd mentioned this pull request Aug 31, 2026

rpc : reject invalid top-level graph nodes in graph_compute #25670

Open

danielhanchen mentioned this pull request Sep 4, 2026

rpc: cherry-pick upstream #18626 and #26500 (real async + event APIs, RPC_PROTO 6.0) unslothai/llama.cpp#181

Closed

ggerganov mentioned this pull request Sep 4, 2026

ggml : bump version to 0.23.0 ggml-org/ggml#1618

Merged

BrewTestBot mentioned this pull request Sep 4, 2026

ggml 0.23.0 Homebrew/homebrew-core#302393

Merged

1 task done

loganbruns mentioned this pull request Sep 5, 2026

RPC: [create_node] invalid data ptr — split scheduler ships compute-flagged views to the wrong RPC worker (reproduced with deepseek4 and glm-dsa, >=2 workers, master) #28047

Closed

hmirin deleted the fix/rpc-multi-server-buffer-ownership branch September 7, 2026 18:46

Navigation Menu

rpc: avoid serializing buffers from other servers - #26500

rpc: avoid serializing buffers from other servers#26500
ggerganov merged 2 commits into
ggml-org:masterfrom
hmirin:fix/rpc-multi-server-buffer-ownership

hmirin commented Aug 3, 2026

Uh oh!

himorishige commented Aug 10, 2026

Uh oh!

dwhoop55 commented Aug 12, 2026

Uh oh!

ludwigjossieaux commented Aug 17, 2026

Uh oh!

fenneclabs commented Aug 18, 2026 •

edited

Loading

Uh oh!

hmirin commented Aug 21, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

hmirin commented Aug 3, 2026

Overview

Additional Information

Requirements

Uh oh!

himorishige commented Aug 10, 2026

Uh oh!

dwhoop55 commented Aug 12, 2026

Uh oh!

ludwigjossieaux commented Aug 17, 2026

Uh oh!

fenneclabs commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hmirin commented Aug 21, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

fenneclabs commented Aug 18, 2026 •

edited

Loading