Skip to content

feat(queen): route bees to the best-measured worker model - #496

Merged
gHashTag merged 1 commit into
feat/queen-supervisorfrom
feat/model-ranking
Sep 21, 2026
Merged

gHashTag merged 1 commit into
feat/queen-supervisorfrom
feat/model-ranking

Conversation

@gHashTag

Copy link
Copy Markdown
Owner

Why

At 10 lanes nemotron-3-super-120b answered 1,369 requests in 20 minutes with 429/503 and bees finished 4 turns. Probing all 10 keys showed 503s spread across every key: the model is overloaded, not our keys. Five of seven would-be fallback models answered 410/404 the same afternoon.

What

  • lib/model-ranking.ts: per-candidate score = expected seconds per 500-token step = (500 / tok/s) / successRate. Inputs: probes every 5 min (speed, tool calling, 404/410) and every live answer. Only models that called a tool are eligible; a challenger must be 20% cheaper (no flapping); gone models re-probed after an hour.
  • overload-retry-fetch.ts: routeModel rewrites the request model per attempt; onOutcome feeds the ranking.
  • Worker endpoint only, and only requests that name a ranked candidate.
  • /queen/status gains models.chosen + models.ranking.
  • Off unless TRIOS_QUEEN_WORKER_MODEL_CANDIDATES is set.

Tests

bun test src/lib/model-ranking.test.ts src/lib/overload-retry-fetch.test.ts 24 pass; dispatch/tick/status/round suites 140 pass; tsc --noEmit clean.

🤖 Generated with Claude Code

At ten lanes nemotron-3-super-120b answered 1,369 requests in twenty minutes
with 429/503; every one of ten keys drew 503s, so the model was overloaded,
not the keys. Five of seven fallback models the same afternoon answered 410
or 404 - a fixed fallback list goes stale unnoticed.

lib/model-ranking.ts scores each candidate by expected seconds per 500-token
step, (500 / tokens per second) / success rate, from probes every five
minutes (speed, tool calling, 404/410) and every live answer the retry
wrapper sees. The retry wrapper rewrites the request's model to the ranked
best, per attempt. A model is chosen only after it has called a tool; a
challenger must be 20% cheaper to take over. /queen/status shows the ranking.

Off unless TRIOS_QUEEN_WORKER_MODEL_CANDIDATES is set.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

❌ Tests failed — 11/2436 failed

Suite Passed Failed Skipped
✅ agent 87/87 0 0
✅ build 9/9 0 0
✅ cdp-protocol 5/5 0 0
✅ eval 93/93 0 0
✅ server-agent 272/272 0 0
❌ server-api 1252/1315 6 57
✅ server-browser 6/6 0 0
✅ server-integration 10/11 0 1
✅ server-lib 279/279 0 0
❌ server-pglive 1/3 2 0
✅ server-root 68/68 0 0
✅ server-skills 31/31 0 0
❌ server-tools 240/243 3 0
✅ shared 14/14 0 0
Failed tests
  • server-api — the note says who committed the branch, where it can be read > puts the salvage fact where the 1500-character cap cannot cut it
  • server-api — a turn the provider killed does not spend the issue > is judged and sent back, but charges neither counter
  • server-api — deriving work the repository already measured > carries the command that produced it
  • server-api — deriving work the repository already measured > gives every candidate a path that can be a boundary
  • server-api — deriving work the repository already measured > proposes nothing from code this project does not own
  • server-api — salvaging a turn that ended with its work uncommitted > refuses a worktree holding an unmerged path
  • server-pglive — the migration block, applied to a real PostgreSQL > creates every object it promises, and survives a second boot
  • server-pglive — the live gate > ran, or its absence is on the record
  • server-tools — navigation tools > new_hidden_page opens a hidden tab
  • server-tools — navigation tools > show_page restores a hidden page to visible
  • server-tools — window tools > create_hidden_window creates and closes a hidden window

View workflow run

@gHashTag
gHashTag merged commit 283812d into feat/queen-supervisor Sep 21, 2026
17 of 21 checks passed
@github-actions
github-actions Bot deleted the feat/model-ranking branch September 27, 2026 04:43
dmitrii-f-t27 pushed a commit to dmitrii-f-t27/BrowserOS that referenced this pull request Oct 10, 2026
feat(queen): route bees to the best-measured worker model
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant