Eval bug: deepseek_v4 over RPC (2-node Metal): empty assistant turns degenerate output to "====", then a worker-side graph failure kills the whole server (RPC has no error recovery) #26384

Name and Version

llama.cpp b10208 (release builds, 2026-07-31) on both hosts. llama-server coordinator + ggml-rpc-server worker, Metal backend on both ends, RPC over a Thunderbolt 5 bridge (~0.5 ms RTT). Hosts: 2× MacBook Pro M4 Max 128 GB, macOS (arm64).

Operating systems

Mac

Which llama.cpp modules do you know to be affected?

llama-server, RPC backend (ggml-rpc)

Command line

Worker (host B, 10.55.0.2):

ggml-rpc-server -H 0.0.0.0 -p 50053 -d MTL0 -t 12

Coordinator (host A):

llama-server -m DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf \
  --rpc 10.55.0.2:50053 --device MTL0,RPC0 -ngl 999 --tensor-split 97,65 \
  -c 131072 --parallel 1 -b 2048 -ub 256 --flash-attn off \
  --cache-type-k f16 --cache-type-v f16 --jinja \
  --host 0.0.0.0 --port 9099 --temp 1.0 --top-p 1.0 --min-p 0.0

Model: unsloth/DeepSeek-V4-Flash-0731-GGUF UD-Q8_K_XL (161.9 GB; the checkpoint's routed experts are native MXFP4, so this quant is effectively a lossless repack). iogpu.wired_limit_mb=117760 on both hosts.

Problem description & steps to reproduce

Two problems that cascade. The first is arguably a model/template eval bug; the second is the one that hurts: any worker-side failure kills the whole multi-host deployment.

1) Eval: empty assistant messages in the history reliably degenerate into ==== walls

On a multi-turn /v1/chat/completions request whose history contains one or more assistant turns with content: "", generation reliably degenerates into long runs of ==== instead of coherent output:

curl http://HOST:9099/v1/chat/completions -d '{
  "messages": [
    {"role": "user", "content": "What is 2+2?"},
    {"role": "assistant", "content": ""},
    {"role": "user", "content": "Now multiply that by 3."}
  ]
}'

With clean histories the same rig is stable for hours (multi-turn, 10K+ token prompts, cached and uncached). Possibly related: sending enable_thinking: false produces the same ==== walls on the 0731 chat template (the model has no non-thinking mode — reasoning_effort is its control surface), which suggests the degeneration is "model pushed off its trained template" rather than a numeric fault. A server-side mitigation could be to normalize or reject empty assistant content at templating time (adjacent: #23506, which touched empty-generation parsing when continuing an assistant message).

2) RPC: a worker-side graph failure takes down both processes — no error path exists

During one of these degenerate generations, a graph computation eventually fails on the RPC worker. The worker then dies by assertion and closes its socket mid-operation:

• worker: ggml-rpc.cpp:1456/:1472 — GGML_ASSERT(status == GGML_STATUS_SUCCESS && "Unsuccessful graph computations are not supported with RPC")

The coordinator's next send/recv on the dead socket then aborts the entire server process:

• coordinator: ggml-rpc.cpp:30 — RPC_STATUS_ASSERT → GGML_ABORT("Remote RPC server crashed or returned malformed response")

Net effect: one bad request kills a 162 GB two-node deployment that takes minutes to reload, and every in-flight client with it. There is no reconnect, no per-request failure, no way to degrade gracefully.

Suggestions:

• (a) worker: on a failed graph computation (or any op error), return an RPC error frame to the client instead of asserting and closing the socket;
• (b) coordinator: treat a failed remote op — or an unexpectedly closed socket — as a failed graph compute for the affected request: fail that request (HTTP 5xx), free the slot, and keep serving (ideally attempt reconnect; even a clean per-request failure without process death would be a big step).

Prior art / adjacent issues: #14076 improved the crash message but kept the abort; #24325 notes the client assumes remote op support — same theme of coordinator/worker disagreement being fatal rather than recoverable.

Cross-references from sustained testing on this rig (b10208, Metal+RPC, UD-Q8_K_XL):

• Eval bug: DeepSeek V4 Flash forgot context when hit cache or use checkpoint in long context #25259 (prompt-cache amnesia on long context): did not reproduce here — multi-turn recall passed cached and uncached. Data point for that thread.
• DeepSeek-V4: quantized K-cache (--cache-type-k q8_0) produces garbage on all backends — attention rotation forces the model off its sparse attention paths #25382 (quantized K-cache garbage): closed, but as of b10208 a quantized K-cache still produced garbage for us on this arch, so we pin --cache-type-k f16 --cache-type-v f16. Mentioned as same-arch context, not the subject of this report.

First Bad Commit

Not bisected — b10208 was our first sustained multi-node deployment of this arch.

Relevant log output

# The crashing session's logs were overwritten by our automated recovery restart
# (worker log rotates on relaunch, coordinator log is truncated on start), so the
# assert strings above are quoted from the b10208 source rather than pasted from
# the incident. The observed sequence was:
#   1. degenerate "====..." stream begins (empty assistant turn in history)
#   2. rpc worker's socket closes mid-op (process gone)
#   3. coordinator llama-server aborts: "Remote RPC server crashed or returned
#      malformed response"
# Happy to re-trigger the crash and attach verbatim logs from both ends if useful.