Eval bug: SIGSEGV in token-counting routes when the request arrives while the server is sleeping (stale vocab/mctx captured before the wake barrier) #29188

Name and Version

version: 0.4.0-dev
built with GNU 16.2.1 for Linux x86_64, GGML_CUDA=ON, Release

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

llama-server

Command line

Router parent:

llama-server --host 127.0.0.1 --port PORT --models-preset models.ini \
  --models-max 1 --no-models-autoload --api-key-file KEYFILE --no-webui

Children are spawned by the router from the preset with:

--sleep-idle-seconds 300 --ctx-size 65536 --batch-size 256 --ubatch-size 128 \
--flash-attn on --n-gpu-layers auto --parallel 1 --reasoning off \
--fit-target 2048 --no-warmup --no-webui --mmproj MMPROJ --no-mmproj-offload \
--image-min-tokens N

Problem description & steps to reproduce

Send a single request to any of the token-counting routes to a child that has
gone to sleep via --sleep-idle-seconds. The server segfaults.

Reproduced deliberately: one child idle for ~4h18m, one POST to
/v1/chat/completions/input_tokens, immediate SIGSEGV. Eight core dumps so far,
all on the same path, every one of them after a wake from idle sleep, never on a
cold start. No concurrency required — a single request is sufficient.

Affected routes (three handler lambdas, five registered URLs):

/chat/completions/input_tokens
/v1/chat/completions/input_tokens
/responses/input_tokens
/v1/responses/input_tokens
/v1/messages/count_tokens

/tokenize, /detokenize and ordinary /v1/chat/completions are NOT affected.

Cause

tools/server/server-context.cpp states the invariant at the top of
init_routes():

// IMPORTANT: all lambda functions must start with create_response()
// this is to ensure that the server_res_generator can handle sleeping case correctly

create_response() constructs server_res_generator, which calls
queue_tasks.wait_until_no_sleep() — the wake barrier.

Every route lambda obeys this except the three that call handle_count_tokens,
which evaluate ctx_server.vocab and ctx_server.mctx as call arguments
before the callee reaches its own create_response():

this->post_chat_completions_tok = [this](const server_http_req & req) {
    return handle_count_tokens(ctx_server.vocab, ctx_server.mctx, ctx_server.init_opt, req, TASK_RESPONSE_TYPE_OAI_CHAT);
};

(same shape for post_responses_tok_oai and post_anthropic_count_tokens)

While sleeping, destroy() has torn the model down:

void destroy() {
    spec.reset();
    spec_init.reset();
    ctx_dft   = nullptr;
    model_dft = nullptr;
    llama_init.reset();
    ctx_tgt   = nullptr;
    model_tgt = nullptr;
    mtmd_free(mctx);
    mctx = nullptr;
}

So the arguments captured at call time are a stale vocab (pointing into the
freed model) and a null mctx. handle_count_tokens then calls
create_response(), which waits for the wake; the reload installs a fresh
vocab on the member via llama_model_get_vocab(model_tgt) — but the copies
already passed as arguments are never refreshed. The null-mctx branch then
runs tokenize_mixed(vocab, ...) through the stale pointer.

Note this is an ordering bug, not a missing-reset bug: nulling vocab in
destroy() would not fix it, because the member is legitimately reassigned on
reload. The captured argument is the problem.

Suggested fix

Read ctx_server.vocab / ctx_server.mctx / ctx_server.init_opt inside
handle_count_tokens after create_response() returns, rather than passing
them in — or pass ctx_server by reference and dereference post-barrier.

Relevant context

destroy() has been corrected once before for omitted teardown (#23461). The
sleep feature also already shipped one use-after-free fix for a chat-template
pointer (#18228, commit 105e2f3). This looks like the same class, one pointer
further on.

In router mode the parent forwards the request straight through to the sleeping
child by design — ensure_model_ready() returns early on
SERVER_MODEL_STATUS_SLEEPING with the comment "child is sleeping but still
running; new request will wake it up" — so any router client whose first call of
a turn is a token count will hit this every time.

First Bad Commit

Not bisected. The token-counting routes were introduced in #23913.

Relevant log output

SIGSEGV, si_code: SEGV_MAPERR
top frame, 5 of 8 cores:  llama_vocab::impl::tokenize
top frame, 3 of 8 cores:  llama_vocab::impl::tokenizer_st_partition
reached from server_routes::handle_count_tokens