fix(distributed): prefill handoff crashes the serving loop for model-owned paged families (gemma3 and friends) #708

Problem / Background

The disaggregated prefill node crashes its serving loop when it receives a request for a model-owned paged family (gemma3 / gemma4 / llama4 / qwen3_5 / qwen3_next). Observed with gemma-3-1b-it-4bit in the 3-node (prefill/decode/router) E2E harness: the prefill node logs Disaggregated serving role loop exited with an error: PagedBlockPool::read_block_contents: layer 0 has no pool tensors to read from (serving_mode=prefill-only), the router health loop marks the prefill peer down, and every subsequent request gets HTTP 500/503. A single request permanently kills the node.

Discovered on 2026-07-09 during issue #398 (PR #707): the first time the Gemma byte-fallback E2Es were ever actually executed. The tests disaggregated_router_completions_match_single_node_byte_fallback (added by PR #392 for issue #387) and the new chat-usage variant are unrunnable-by-construction with Gemma: PR #392's test plan explicitly noted it was never run ("no Gemma checkpoint available"), and gemma3 is a model-owned paged family, which the pool-block handoff (#125) never supported (the handoff scope is the pool-backed Fp16 families, qwen3 / llama3).

Root cause (verified on main at c259d23)

• CachePool::extract_paged_blocks (src/lib/mlxcel-core/src/cache.rs, around line 6283) guards against model-owned sequences with if !sequence.caches.iter().all(|cache| cache.is_paged_backed()) { return Ok(Vec::new()); }. For model-owned families, LanguageModel::make_caches() returns an EMPTY vec (e.g. src/models/gemma3.rs around line 1214), so .all() is vacuously true and the guard falls through.
• The in-code comment assumes empty caches implies an empty block table ("a paged-natural stub is vacuously pool-backed and falls through to the empty block-table fast path"), but model-owned families are routed through the paged backend for SHADOW block-table accounting (sync_paged_state_with_lengths), so state.layers[*].block_ids are populated while the pool tensors were never written (their KV lives model-internal). PagedBlockPool::read_block_contents (src/lib/mlxcel-core/src/cache/paged.rs around line 1402) then fails with "layer 0 has no pool tensors to read from".
• The error propagates out of prefill_request_for_handoff / prefill_text_request_for_handoff (src/server/batch/scheduler.rs) and the serving-role loop in src/server/model_provider/model_worker.rs treats it as fatal and exits, killing the node.

Proposed direction

1. Reject-cleanly (correctness): make extract_paged_blocks (or its callers on the handoff path) detect a model-owned sequence explicitly (not via the vacuous .all()) and return a per-request error like "the disaggregated handoff does not support model-owned paged families ()" so the router can surface a clean 4xx/5xx to the client. Alternatively reject even earlier at request intake on the prefill node based on model.sequence_state_layout().backend.
2. Survive per-request errors (availability): the prefill/decode serving-role loop must not exit on a single failed handoff; it should report the error frame for that request and continue serving. Reserve loop exit for transport-level/fatal conditions.
3. Test debt: retarget or gate the Gemma-based byte-fallback E2Es. PR fix(router): emit usage on the disaggregated /v1/chat/completions responses #707 already retargets the chat-usage E2E to MiniCPM-2B (llama-format export: pool-backed dense Llama arch with a byte_fallback SentencePiece tokenizer, checkpoint mlx-community/MiniCPM-2B-sft-4bit-llama-format-mlx). The fix(router): use worker's authoritative token count for usage #392 completions test should get the same retarget, and a new regression test should pin the clean-rejection behavior for a model-owned family.
4. (Optional, larger follow-up worth its own issue if pursued) actual model-owned handoff support by serializing the ModelOwnedSequenceState.

Acceptance Criteria

• A chat or completions request for a model-owned family through the 3-node disaggregated setup returns a clean per-request error (no usage of the pool-read path), and the prefill node keeps serving subsequent requests (verified by a follow-up request succeeding or failing identically, not by 503-peer-down).
• The prefill serving-role loop no longer exits on a per-request handoff error.
• A unit test pins extract_paged_blocks behavior for a model-owned sequence with a populated shadow block table (clean error or empty result, no pool read panic/error).
• disaggregated_router_completions_match_single_node_byte_fallback is retargeted to a pool-backed byte-fallback checkpoint (MiniCPM-2B llama-format) so it can actually run, or explicitly gated until model-owned handoff exists.

References

• Code: src/lib/mlxcel-core/src/cache.rs (extract_paged_blocks), src/lib/mlxcel-core/src/cache/paged.rs (read_block_contents), src/server/batch/scheduler.rs (prefill_request_for_handoff, prefill_text_request_for_handoff), src/distributed/disaggregated/handoff_impl.rs, src/server/model_provider/model_worker.rs (serving-role loop exit).
• History: Distributed block-table serialization (kv_cache_serde) #125 (pool-block transfer, pool-backed Fp16 scope), fix(router): use worker's authoritative token count for usage #392 / fix(router): carry the worker's true generated-token count over the disaggregated wire protocol #387 (byte-fallback E2E added but never run), fix(router): emit usage on the disaggregated /v1/chat/completions responses (streaming and non-streaming) #398 / PR fix(router): emit usage on the disaggregated /v1/chat/completions responses #707 (discovery context).