Skip to content

server: refactor sleep handling, allow access /metrics during sleep - #27376

Merged
ngxson merged 11 commits into
masterfrom
xsn/server_refactor_sleep
Aug 19, 2026
Merged

ngxson merged 11 commits into
masterfrom
xsn/server_refactor_sleep

Conversation

@ngxson

@ngxson ngxson commented Aug 19, 2026 •

Copy link
Copy Markdown
Collaborator

Overview

  • server_routes will read cached response during sleep
  • clarify the call order of on_sleeping_state

as a side-effect, /metrics can now be accessed during sleep

Requirements

@github-actions github-actions Bot added documentation Improvements or additions to documentation server labels Aug 19, 2026
@ngxson
ngxson marked this pull request as ready for review August 19, 2026 11:35
@ngxson
ngxson requested a review from a team as a code owner August 19, 2026 11:35
@ServeurpersoCom

ServeurpersoCom commented Aug 19, 2026 •

Copy link
Copy Markdown
Contributor

I think /slots should keep resetting the idle timer, only /metrics should skip it, otherwise monitoring no longer keeps the server awake but still forces a model reload on every polling round.

  • 3 nits :
  1. cached_metrics keeps the buckets as they were, so a sleeping server reports its last throughput on prompt_tokens_seconds and predicted_tokens_seconds forever, a reset_bucket() on the cached copy would give the 0 that the window actually holds.

  2. setting content_type before the is_error() check means an error result is now returned as JSON labelled text/plain, since error() only overwrites the status and the body.

  3. tmp.n_idle_slots is never read, to_metrics() only emits n_processing_slots and n_tasks_deferred.

I pushed directly:

@ngxson
ngxson force-pushed the xsn/server_refactor_sleep branch from 313e807 to 0baf73a Compare August 19, 2026 15:50
@ngxson

ngxson commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator Author

for /slots vs /metrics, it's better to separate them to 2 different tasks, which was done in c1fc93f

that also resolved the n_idle_slots being unused

@ServeurpersoCom ServeurpersoCom left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ServeurpersoCom

Copy link
Copy Markdown
Contributor

Nice. Splitting the task type kills the root cause instead of patching the symptom, and keeping the last window until a scrape reads it is more correct than dropping it on sleep

@ngxson
ngxson merged commit 947fd9b into master Aug 19, 2026
31 of 32 checks passed
tobocop2 added a commit to tobocop2/llama.cpp that referenced this pull request Aug 28, 2026
An upstream refactor can merge and compile clean and still break the memory
endpoints. The sleep refactor (ggml-org#27376) did exactly that: it moved /metrics
behind a cached snapshot, and the memory gauges went missing from the
sleeping path while the build stayed green.

Run tools/server/tests after the compile step, the same tests.sh upstream's
own server workflow uses. The build takes about 4 minutes and the tests
about the same, so the existing 60 minute job timeout still has room.
tobocop2 added a commit to tobocop2/llama.cpp that referenced this pull request Aug 28, 2026
Upstream restructured the code the memory work sits on.

server: refactor sleep handling (ggml-org#27376) split slot reporting out of
SERVER_TASK_TYPE_METRICS into a new SERVER_TASK_TYPE_SLOT_GET task,
rewrote get_metrics around a cached snapshot served during sleep, and
extracted the props body into get_res_props(). memory_data stays on
server_task_result_metrics, the memory gauges keep their place on
/metrics, and endpoint_memory is back in get_res_props().

/metrics during sleep must equal the last awake scrape. The model is
unloaded by then, so update_cached_responses() now renders the memory
series while the model is still loaded and use_cached_metrics() appends
the cached string. The rendering moved into render_memory_metrics() and
the breakdown into server_context_impl::get_memory_data(), so the awake
and sleeping paths share one implementation.

common: add json.h abstraction (ggml-org#27511) replaced nlohmann::json with
common_json, which has no find(). The gauge loop uses contains() and
at() instead.
thecodacus pushed a commit to thecodacus/llama.cpp that referenced this pull request Sep 7, 2026
…gml-org#27376)

* add cached responses

* refactor on_sleeping_state

* allow accessing metrics during sleep

* metrics task should not reset timer

* updated docs

* fix

* fix get_res_model_info

* add test

* fix a race condition

* split metrics and slots tasks / results

* should_reset_buckets
zbrad pushed a commit to zbrad/llama.cpp that referenced this pull request Sep 10, 2026
…gml-org#27376)

* add cached responses

* refactor on_sleeping_state

* allow accessing metrics during sleep

* metrics task should not reset timer

* updated docs

* fix

* fix get_res_model_info

* add test

* fix a race condition

* split metrics and slots tasks / results

* should_reset_buckets
pl752 pushed a commit to pl752/llama.cpp that referenced this pull request Sep 15, 2026
…gml-org#27376)

* add cached responses

* refactor on_sleeping_state

* allow accessing metrics during sleep

* metrics task should not reset timer

* updated docs

* fix

* fix get_res_model_info

* add test

* fix a race condition

* split metrics and slots tasks / results

* should_reset_buckets
frostyautumnleaf pushed a commit to frostyautumnleaf/llama.cpp that referenced this pull request Oct 5, 2026
…gml-org#27376)

* add cached responses

* refactor on_sleeping_state

* allow accessing metrics during sleep

* metrics task should not reset timer

* updated docs

* fix

* fix get_res_model_info

* add test

* fix a race condition

* split metrics and slots tasks / results

* should_reset_buckets
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation server

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants