track(server): reproducible cold/warm prompt-cache output difference on Qwen3-1.7B / GB10 #2175

The harness and verifier are distributed together in tools.tar.gz; extract it to obtain tools/run_server_cache.py and tools/export_current.py.

Summary and scope

Following the server observation on #2128, I independently reproduced a stable cache-state-dependent greedy reply with Qwen3-1.7B on GB10: cold requests produce A, actual prefix-cache hits produce B, and both cache-disabled controls reproduce A.

This is a tracking/reproduction report for the case left outside #2162. I am not claiming a new cache-corruption bug, stochastic variation within an arm, or a failure of the one-row SDPA fix. Generic split-prefill rounding is already documented. The purpose is to attach an exact independently reproducible case and establish whether this observation should be tracked separately or explicitly classified under that documented limitation.

Environment

• NVIDIA GB10 (sm_121), Linux aarch64.
• Unmodified upstream commit e466f5118b8e25d0c9a295ab017468db77b87313, the fix(cuda): add MLXCEL_SDPA_DETERMINISTIC for reproducible decode #2162 merge; this already includes fix(prompt-cache): stop a whole-prompt hit from duplicating the last token #2136.
• Release CUDA build: cargo build --locked --release --features cuda --bin mlxcel --bin mlxcel-server -j 2.
• CUDA compiler 13.0.88; cuDNN build headers 9.13.0; MLX dependency 81ba1c6a0e50a9268b931579c2d4f1158b9aab5a.
• cuDNN runtime 9.13.0 (cudnnGetVersion = 91300 from the same library path captured in the actual process maps); loaded CUDA runtime libcudart.so.13.0.96 and driver library libcuda.so.580.178.04. This is separate from the maintainer's cuDNN 9.27 environment in fix(cuda): add MLXCEL_SDPA_DETERMINISTIC for reproducible decode #2162.
• Server binary SHA256: e9d52181962009218f539569802d5fc2a389fb607d18ffad2b14b2f41dfdc608.
• Model: mlx-community/Qwen3-1.7B-4bit, pinned to 3b1b1768f8f8cf8351c712464f906e86c2b8269e.
• Model SHA256: 0e86d9677e519323849eac1bc272caae88567a481ff188c431f70be543d9995f; config SHA256: 507a6701220524eb8b283425bf0856a9ae4f21f4052e563896ddd668994b1dc7. All nine Qwen model/tokenizer/config files were rehashed after the experiments and matched the pinned manifest. The artifact includes that manifest; model weights are not redistributed.
• CUDA graphs left at their default setting. Requests serialized, no overlapping model workload. --parallel 4 preserves the paged-server configuration; it does not mean requests were issued concurrently. No logprob requests or forced synchronous-decode diagnostic toggle.

Results

Both MLXCEL_SDPA_DETERMINISTIC=0 and =1 completed 14 / 14 valid requests, for 28 / 28 total. They gave the same A/B split:

Every response reported prompt_tokens:42 and finish_reason:"stop"; none reached the 800-token cap. Across both modes, all 18 cold/disabled outputs were A and all 10 warm outputs were B.

A and B are SHA256 hashes of canonical assistant content, reasoning_content, and tool_calls, not the entire HTTP response. Request IDs, timestamps and usage do not enter the output hash:

• A: f226eab5468d501934ec58d8f9092c00666b38a7d7f34946a326481a579c6534
• B: fca63627c7003c038c1740cff0a0c702937cbd53bf76c06ceffd27ccafc4e2ae

The first differing Unicode character in content is at zero-based index 397; this is not a token index. For example:

• Cold/disabled: “It was a cold, gray morning when the wind began to howl. Elias had been on the deck for hours, sweeping the sand from the pier…”
• Warm: “It was a cold, gray morning when the wind picked up. Elias was alone in the tower, the only sound the creak of the old wooden floor…”

Cache use was verified rather than inferred from speed. Each warm request reported 32 cached tokens, while the scheduler metrics advanced by +2 hits / +64 reused tokens. Those are the observed counter deltas, not a claim that the request reused 64 tokens: the response-level value is 32. Cold and per-request-disabled requests reported zero cached tokens; the server-wide disabled arm omitted cached-token details and reported cache disabled. All non-warm requests had zero hit/reuse deltas, and both disabled controls had zero cache-store lookup/hit/insert deltas.

The pool block size was 32. Actual mlxcel_paged_decode_launches_total{path="gather"} deltas were 9,772 for each A response and 12,572 for each B response; fused_v2 and cascade deltas were zero. Thus the measured pool-backed path was gather fallback, not the fused paged-attention kernel. The source implements that fallback with gather_visible followed by attention_from_ptr; I did not trace the complete underlying GPU kernel sequence or infer that SDPA was bypassed.

No output variation was observed within any of the eight completed arm/mode groups. This distinguishes a reproducible cold/warm path difference from uncontrolled run-to-run variability in this finite sample; it is not a proof of determinism.

Exact request and reproduction

Resolve the pinned checkpoint above into a local directory. Use the linked harness with the pinned server binary and compatible CUDA/cuDNN runtime paths; it does not download models or manipulate existing services.

python3 run_server_cache.py \
  --binary /path/to/mlxcel-server \
  --model-dir /path/to/Qwen3-1.7B-4bit \
  --output ./server-det-off \
  --deterministic 0 --cycles 2 --max-tokens 800 \
  --port 18128 --wall-timeout 750

For the second condition use a new output directory and --deterministic 1. The example gives the harness a 750-second safety bound; the captured runs had remaining global bounds of 608.208 and 551.199 seconds respectively, without changing the request/model settings. The harness clears inherited MLX/MLXCEL/APC/LLAMA experiment toggles and known CUDA diagnostic overrides; retains explicitly identified runtime paths; and records effective settings and actual loaded library identities in private local output. The linked evidence exporter removes private paths and environment values before publication. Output directories must not already exist.

The underlying cache-enabled server command is:

MLXCEL_DEVICE=gpu MLXCEL_APC_TRACE=1 \
MLXCEL_SDPA_DETERMINISTIC=0 \
MLXCEL_CACHE_DIR=/path/to/isolated-runtime-cache RUST_LOG=info \
/path/to/mlxcel-server \
  -m /path/to/Qwen3-1.7B-4bit --alias cache-repro \
  --host 127.0.0.1 --port 18128 --parallel 4 --metrics --no-warmup

The disabled-server control adds --no-prompt-cache. MLX_USE_CUDA_GRAPHS is unset, not forced on. The exact /v1/chat/completions body is:

{
  "model": "cache-repro",
  "messages": [
    {
      "role": "user",
      "content": "Write a short story of about 600 words about a lighthouse keeper who finds a message in a bottle. Use vivid but plain language. /no_think"
    }
  ],
  "max_tokens": 800,
  "temperature": 0,
  "seed": 0,
  "stream": false,
  "user": "mlxcel-2128-cache-repro"
}

Only the per-request opt-out control adds "cache_prompt": false. The same payload is otherwise retained; no logprobs are requested. This is a chat-template-encoded server request, not a claim of tokenization identity with the raw CLI prompt used in the separate Llama follow-up.

For each SDPA mode, the harness performs:

1. Start a fresh server, reset the prefix cache, then cold → warm → warm.
2. Reset again, then cold → warm → warm.
3. Send three identical requests with cache_prompt:false in that server.
4. Restart the server, then cold → warm.
5. Start a fresh server with --no-prompt-cache, then send three repeats.

For every request it records HTTP completion, output/usage, before/after /v1/cache/stats, mlxcel_prompt_cache_hits_total, mlxcel_prompt_cache_prefix_tokens_reused_total, and mlxcel_paged_decode_launches_total labels. Cache resets are verified to clear entries; counters are compared as deltas. The paged cascade counter is not counted as a disjoint launch category from fused_v2.

Relationship to existing fixes and limits

• #2136 fixed the duplicate-last-token whole-prompt bug from fix(prompt-cache): a whole-prompt cache hit re-runs the last token on top of a cache that already holds it #1760. Its residual Qwen3-4B/Gemma-4 drift disappeared when both arms used --prefill-chunk-size 1. This is important prior art, not proof of the cause of this Qwen3-1.7B fixture.
• #2162's report leaves its server cache-state observation outside the PR. Our actual gather-fallback counters are reported above; they do not establish the same exact kernel sequence as that earlier observation. The deterministic SDPA switch is not a general promise of cold/warm cache-state-independent replies.
• Not executed in this report: equal-width --prefill-chunk-size 1 control, first-divergent-token/logit localization, or other GPUs/models. Logprobs were off to preserve the ordinary eligible scheduling path, so I do not claim an exact token divergence index or a measured top-two logit gap.
• The observed A/B split is consistent with the documented split-prefill explanation, but this experiment alone does not prove its internal numerical mechanism. It does not establish that either generated story is semantically incorrect.

Related work

The newly opened #2166 / #2167 engine-parity work includes a broader future cache-hit/miss baseline, and #2171 concerns the storage abstraction. This report provides a concrete fixture on the existing paged-gather path (serialized requests, server parallelism 4), not a competing implementation proposal or a claim to have run the planned B=1 dense/paged parity harness. It can serve as input to that baseline.

Supporting artifacts and requested disposition

• Sanitized requests, full outputs, cache/path telemetry, model manifest and SHA256SUMS
• Reproduction harness
• Offline evidence verifier

The artifacts omit private host paths, raw service environments, process maps and model weights. They preserve the numerical/run evidence and command placeholders needed to reproduce the case.

Please classify this concrete #2128 follow-up either as the already documented cache-state-dependent numerical limitation or as a separately actionable reproducibility issue. The documented cache_prompt:false workaround matched the cold output in the measured control. I am not requesting a change to #2162's scope or claiming its cuDNN root cause applies to these paged-server outputs.

Separate Llama CLI evidence: #2174. The CLI and server counts are not combined.