fix(server): investigate the "Failed to sync paged state ... got 0" warning spam in batched speculative decode #823

Summary

During batched speculative decode on CUDA, mlxcel-server emits the warning Failed to sync paged state for seq-N: CachePool: expected 32 paged layer lengths for seq-N, got 0 hundreds of times per burst. Requests still complete and return correct output, so this is not fatal, but the log spam is severe and the underlying Err return means the paged-state length sync is being skipped for those sequences on those ticks. This issue is to determine whether the skipped sync is genuinely benign (and the log should be downgraded) or is masking a real paged-KV desync.

Where it comes from

• src/server/batch/scheduler.rs:2685 — sync_sequence_storage logs the warning and swallows the error: if let Err(err) = self.model.sync_sequence_storage(seq_id, &mut self.cache_pool) { tracing::warn!("Failed to sync paged state for {seq_id}: {err}"); }.
• src/lib/mlxcel-core/src/cache.rs:6280 — the error is raised when the requested per-layer target lengths vector does not match the number of paged layers: return Err(format!("CachePool: expected {} paged layer lengths for {id}, got {}", state.layers.len(), target_lens.len()));. The observed message is "expected 32 ... got 0", i.e. an empty target_lens for a sequence whose paged state has 32 layers.

The surrounding comment in cache.rs notes this length sync is authoritative only for the model-owned paged path (gemma3/llama4/qwen3_5) and a no-op for the pool-backed dense-natural path (qwen3/llama3), because for the latter write_prefill already advanced the lengths in lockstep.

Reproduction

mlxcel-server -m models/qwen3.5-4b-4bit --draft-model models/qwen3.5-4b-dflash --port <p>
# fire concurrent chat completions; grep the server log for "Failed to sync paged state"

Observed on GB10 with qwen3.5-4b + dflash drafter, batched decode (batch_size=4), hundreds of lines per burst. Present on both v0.4.1 and current main.

What to investigate

1. Which sequences trigger it and when: are these speculative (draft) rows, freshly admitted sequences whose paged state is not yet initialized, or rows on a tick where no target lengths were computed? The "got 0" (empty target_lens) suggests sync_sequence_storage is called before, or on a path where, the per-layer targets were populated.
2. Whether the skipped sync is benign: if for these rows the lengths are already correct (as the dense-natural comment implies) or the row does not use the paged path this tick, the sync is legitimately unnecessary and the Err is a false alarm. If instead a real length divergence is being silently skipped, it is a correctness risk under paged KV.
3. The interaction with speculative decode specifically: the burst-heavy speculative path is where this shows up, so the draft/verify KV bookkeeping is the likely trigger.

Likely outcomes

• If benign: make sync_sequence_storage distinguish "no sync needed for this row this tick" from a genuine mismatch, so the empty-target_lens case returns Ok(()) (or is not called), and reserve the warn! for an actual length divergence. That removes the spam without hiding real errors.
• If not benign: fix the ordering so paged-state target lengths are populated before the sync runs for speculative/batched rows.

Refs

Surfaced during #818 release testing (the "Also observed" note). Sibling follow-ups: the FFI-boundary catch and the upstream MLX report. Independent of the #818 graph-cache fix.