Skip to content

fix(queen): say why the server hung; split 429 from 503 - #497

Merged
gHashTag merged 2 commits into
feat/queen-supervisorfrom
fix/liveness-forensics
Sep 21, 2026
Merged

gHashTag merged 2 commits into
feat/queen-supervisorfrom
fix/liveness-forensics

Conversation

@gHashTag

Copy link
Copy Markdown
Owner

Why

  • 11:19Z today the server froze for 13 min, logged nothing, and was killed by the liveness supervisor with no record of the cause.
  • The model ranking counted 429 and 503 as one failure; they need different fixes.

What

  • docker-entrypoint.sh: at the 3rd, middle and last missed probe, print from /proc the server's state/cpu/threads/rss/wchan and every descendant with age, cpu and command line. Verified in python:3.12-slim against a fake server (sleeping + spinning child).
  • overload-retry-fetch names the refusal cause; /queen/status models.ranking[].refusals counts by cause.

Tests

26 pass (ranking + retry); tsc clean.

🤖 Generated with Claude Code

gHashTag and others added 2 commits September 21, 2026 18:38
… ends it

Measured 2026-09-21 11:19: the server logged nothing for thirteen minutes
and was killed with no record of what it was doing. At the 3rd, middle
and last missed probe the supervisor now prints, from /proc, the server's
state, cpu seconds, threads, rss and wchan, and every descendant process
with its age, cpu and command line - which separates a CPU loop from a
wait on a child from memory pressure. Verified in python:3.12-slim
against a fake server with a sleeping and a spinning child.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
503 is the model overloaded (switching models helps); 429 is one key over
its rate (fewer lanes per key helps). The ranking counted both as one
failure, which hid which lever to pull. The retry wrapper now names the
cause ('429', '503', 'stream', 'threw') and /queen/status shows refusals
by cause per model.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@github-actions github-actions Bot added the fix label Sep 21, 2026
@github-actions

Copy link
Copy Markdown

❌ Tests failed — 11/2436 failed

Suite Passed Failed Skipped
✅ agent 87/87 0 0
✅ build 9/9 0 0
✅ cdp-protocol 5/5 0 0
✅ eval 93/93 0 0
✅ server-agent 272/272 0 0
❌ server-api 1252/1315 6 57
✅ server-browser 6/6 0 0
✅ server-integration 10/11 0 1
✅ server-lib 279/279 0 0
❌ server-pglive 1/3 2 0
✅ server-root 68/68 0 0
✅ server-skills 31/31 0 0
❌ server-tools 240/243 3 0
✅ shared 14/14 0 0
Failed tests
  • server-api — the note says who committed the branch, where it can be read > puts the salvage fact where the 1500-character cap cannot cut it
  • server-api — a turn the provider killed does not spend the issue > is judged and sent back, but charges neither counter
  • server-api — deriving work the repository already measured > carries the command that produced it
  • server-api — deriving work the repository already measured > gives every candidate a path that can be a boundary
  • server-api — deriving work the repository already measured > proposes nothing from code this project does not own
  • server-api — salvaging a turn that ended with its work uncommitted > refuses a worktree holding an unmerged path
  • server-pglive — the migration block, applied to a real PostgreSQL > creates every object it promises, and survives a second boot
  • server-pglive — the live gate > ran, or its absence is on the record
  • server-tools — navigation tools > new_hidden_page opens a hidden tab
  • server-tools — navigation tools > show_page restores a hidden page to visible
  • server-tools — window tools > create_hidden_window creates and closes a hidden window

View workflow run

@gHashTag
gHashTag merged commit 289e011 into feat/queen-supervisor Sep 21, 2026
17 of 21 checks passed
@github-actions
github-actions Bot deleted the fix/liveness-forensics branch September 27, 2026 04:43
dmitrii-f-t27 pushed a commit to dmitrii-f-t27/BrowserOS that referenced this pull request Oct 10, 2026
fix(queen): say why the server hung; split 429 from 503
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant