Skip to content

rpc : proper handling of data pointers to CPU buffers - #21030

Merged
rgerganov merged 1 commit into
ggml-org:masterfrom
rgerganov:rpc-post-rce
Mar 27, 2026
Merged

rpc : proper handling of data pointers to CPU buffers#21030
rgerganov merged 1 commit into
ggml-org:masterfrom
rgerganov:rpc-post-rce

Conversation

@rgerganov

@rgerganov rgerganov commented Mar 26, 2026

Copy link
Copy Markdown
Member

The compute graph may contain tensors pointing to CPU buffers. In these cases the buffer address is serialized as 0 and sent over the wire. However, the data pointer is serialized as-is and this prevents proper validation on the server side. This patches fixes this by serializing the data pointer as 0 for non-RPC buffers and doing proper validation on the server side.

closes: #21006

@rgerganov
rpc : proper handling of data pointers to CPU buffers
57038f3 
The compute graph may contain tensors pointing to CPU buffers. In these
cases the buffer address is serialized as 0 and sent over the wire.
However, the data pointer is serialized as-is and this prevents proper
validation on the server side. This patches fixes this by serializing
the data pointer as 0 for non-RPC buffers and doing proper validation on
the server side.

closes: ggml-org#21006
@rgerganov
rgerganov requested a review from a team as a code owner March 26, 2026 15:06
@rgerganov rgerganov self-assigned this Mar 26, 2026
@rgerganov
rgerganov requested a review from ggerganov March 26, 2026 15:06
@rgerganov rgerganov mentioned this pull request Mar 26, 2026
Closed
@github-actions github-actions Bot added the ggml changes relating to the ggml tensor library for machine learning label Mar 26, 2026
@v0l v0l mentioned this pull request Mar 26, 2026
Merged
ggerganov
ggerganov approved these changes Mar 26, 2026
View reviewed changes

@ggerganov ggerganov left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merge?

@ggerganov

ggerganov commented Mar 26, 2026

Copy link
Copy Markdown
Member

@las7 Maybe also take a look as a quick follow-up on the previous patch.

@las7

las7 commented Mar 26, 2026

Copy link
Copy Markdown
Contributor

Taking a look

@stew675

stew675 commented Mar 26, 2026

Copy link
Copy Markdown

I am the original issue reported. I just applied @rgerganov 's patch to a pull of the latest source code and rebuilt. I can confirm that this change now fixes the issue.

@las7

las7 commented Mar 26, 2026

Copy link
Copy Markdown
Contributor

LGTM

@Xcc313r4n7

Xcc313r4n7 commented Mar 27, 2026

Copy link
Copy Markdown

Still failing on cross-architecture NVIDIA setup (RTX 5090 Blackwell + RTX 4090 Ada):

RPC server side:
[create_node] invalid data ptr
[graph_compute] failed to create graph node 5 (id=94071254079184)

Client side:
ggml-rpc.cpp:669: Remote RPC server crashed or returned malformed response
recv failed (bytes_recv=0, size_to_recv=8)

Note: Error changed from "invalid tensor: null buffer" to "invalid data ptr" — the PR fixed one check but there appears to be a second issue in cross-architecture setups.

Environment: RTX 5090 (compute 12.0) + RTX 4090 (compute 8.9)

@rgerganov

rgerganov commented Mar 27, 2026

Copy link
Copy Markdown
Member Author

Note: Error changed from "invalid tensor: null buffer" to "invalid data ptr" — the PR fixed one check but there appears to be a second issue in cross-architecture setups.

You need to apply the patch and rebuild both client and server sides. I suspect you only patched rpc-server.

@Xcc313r4n7

Xcc313r4n7 commented Mar 27, 2026

Copy link
Copy Markdown

Note: Error changed from "invalid tensor: null buffer" to "invalid data ptr" — the PR fixed one check but there appears to be a second issue in cross-architecture setups.

You need to apply the patch and rebuild both client and server sides. I suspect you only patched rpc-server.

Sorry about that, confirmed working

@rgerganov
rgerganov requested a review from CISC March 27, 2026 08:21
@rgerganov

rgerganov commented Mar 27, 2026

Copy link
Copy Markdown
Member Author

@ggml-org/maintainers fyi, this is good to go but needs a second approval

@rgerganov
rgerganov merged commit ba38f3b into ggml-org:master Mar 27, 2026
43 of 45 checks passed
slartibardfast pushed a commit to slartibardfast/llama.cpp-prism-avx2 that referenced this pull request Apr 12, 2026
@rgerganov
rpc : proper handling of data pointers to CPU buffers (ggml-org#21030)
adbecfb 
The compute graph may contain tensors pointing to CPU buffers. In these
cases the buffer address is serialized as 0 and sent over the wire.
However, the data pointer is serialized as-is and this prevents proper
validation on the server side. This patches fixes this by serializing
the data pointer as 0 for non-RPC buffers and doing proper validation on
the server side.

closes: ggml-org#21006
Seunghhon pushed a commit to Seunghhon/llama.cpp that referenced this pull request Apr 26, 2026
@rgerganov
rpc : proper handling of data pointers to CPU buffers (ggml-org#21030)
7d2e474 
The compute graph may contain tensors pointing to CPU buffers. In these
cases the buffer address is serialized as 0 and sent over the wire.
However, the data pointer is serialized as-is and this prevents proper
validation on the server side. This patches fixes this by serializing
the data pointer as 0 for non-RPC buffers and doing proper validation on
the server side.

closes: ggml-org#21006
my-other-github-account pushed a commit to my-other-github-account/llama.cpp that referenced this pull request May 15, 2026
@rgerganov
rpc : proper handling of data pointers to CPU buffers (ggml-org#21030)
f245011 
The compute graph may contain tensors pointing to CPU buffers. In these
cases the buffer address is serialized as 0 and sent over the wire.
However, the data pointer is serialized as-is and this prevents proper
validation on the server side. This patches fixes this by serializing
the data pointer as 0 for non-RPC buffers and doing proper validation on
the server side.

closes: ggml-org#21006
fukuro-kun pushed a commit to fukuro-kun/fukuro-llama-cpp-turboquant that referenced this pull request Jul 5, 2026
@rgerganov
rpc : proper handling of data pointers to CPU buffers (ggml-org#21030)
542ce50 
The compute graph may contain tensors pointing to CPU buffers. In these
cases the buffer address is serialized as 0 and sent over the wire.
However, the data pointer is serialized as-is and this prevents proper
validation on the server side. This patches fixes this by serializing
the data pointer as 0 for non-RPC buffers and doing proper validation on
the server side.

closes: ggml-org#21006
MrLordCat referenced this pull request in MrLordCat/llama.cpp-rdna-lab Jul 16, 2026
@rgerganov
rpc : proper handling of data pointers to CPU buffers (#21030)
8f8878a 
The compute graph may contain tensors pointing to CPU buffers. In these
cases the buffer address is serialized as 0 and sent over the wire.
However, the data pointer is serialized as-is and this prevents proper
validation on the server side. This patches fixes this by serializing
the data pointer as 0 for non-RPC buffers and doing proper validation on
the server side.

closes: #21006
@hmirin hmirin mentioned this pull request Aug 3, 2026
Merged
zommiommy pushed a commit to zommiommy/llama.cpp that referenced this pull request Aug 18, 2026
@rgerganov @zommiommy
rpc : proper handling of data pointers to CPU buffers (ggml-org#21030)
bc92f57 
The compute graph may contain tensors pointing to CPU buffers. In these
cases the buffer address is serialized as 0 and sent over the wire.
However, the data pointer is serialized as-is and this prevents proper
validation on the server side. This patches fixes this by serializing
the data pointer as 0 for non-RPC buffers and doing proper validation on
the server side.

closes: ggml-org#21006
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ggerganov ggerganov ggerganov approved these changes
@ServeurpersoCom ServeurpersoCom ServeurpersoCom approved these changes
@CISC CISC Awaiting requested review from CISC

Assignees

@rgerganov rgerganov

Labels

ggml changes relating to the ggml tensor library for machine learning

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Eval bug: PR20908 breaks rpc-server functionality when balancing split a model across multiple machines.

6 participants

@rgerganov @ggerganov @las7 @stew675 @Xcc313r4n7 @ServeurpersoCom