Repository navigation
fix(orchestration): wake an agent for new mail even while it still holds an unacknowledged batch - #33639
Conversation
…lds an unacknowledged batch An unacknowledged check batch used to silence every later nudge for that mailbox, so a coordinator or worker that ran check without --ack was never woken again. Both pointer lanes now defer instead: mail the held batch does not contain gets one nudge at the idle or turn-end edge, naming the batch and the exact ack command.
|
No actionable comments were generated in the recent review.
|
…let the agent replay the batch first - The nudge now says how to see the held delivery again before acking it, so an agent that lost it to /clear or compaction never acks unseen mail. - Chat lane: the send's own admission holds the nudge while a turn runs, closing the window where a turn starts during the journal read. - Terminal lane: a held-batch reminder's Enter waits for the next idle edge when the agent starts working inside the Enter delay. - Checked narrowing replaces the row assertion; tests move the exclusion check onto a real database and drop the real-time sleep.
…rminal pane A terminal coordinator holding an unacked Run delivery gets exactly one pointer naming it for newer mail; its Enter waits for the next idle edge when the agent starts working inside the Enter delay; a later idle edge stays quiet; acking returns the newer mail.
Merge stablyai/orca main b759a95 into fork main 83fb74d. 66 conflicts, mostly upstream's repo-wide comment audit (stablyai#33249): fork code kept with upstream comments. Code-level combinations: mailbox pointer delivery (stablyai#33639) with the fork's queued/injected outcome; held-batch lookup spans every outstanding typed delivery (fork schema v44); launched-agent write guard keeps the fork settle loop plus upstream's write-if-shell-provable policy; worker-start keeps the fork's refusal to guess the coordinator (upstream test adapted); monitored-work rows stay out of Claude agent view. Files over max-lines are split without behavior change (group settings dialog gets its own hook).
ELI5
An agent reads its orchestration mail with
check, which hands it a batch it must confirm withcheck --ack <deliveryId>. Until it confirms, Orca treated the agent as "still busy reading" and stopped telling it about any new mail, forever. So an agent that rancheckwithout--ack(which is exactly what the old nudge text told it to run) and then ended its turn was never woken again. In the ticket, a coordinator's mail sat unread for about 3 hours until a person typed into the chat.Now the unconfirmed batch only delays the nudge. When the agent's turn ends (chat) or it goes idle (terminal) and mail it has not seen is waiting, Orca nudges it once, names the batch it still holds, and gives the exact command to confirm it and get the new mail.
Root cause
Both pointer lanes returned early whenever the mailbox had an outstanding Delivery:
structured-mailbox-pointer-delivery.ts(if (!db || db.hasOutstandingMailboxDelivery?.(mailboxHandle)) return)mailbox-pointer-delivery.ts(same check)They parked no retry. The turn-end and idle edges do re-run delivery, but they hit the same early return. Nothing expires an outstanding Delivery and nothing re-runs delivery on ack, so only user input could break the wedge. The behaviour has been the same since chat coordinators first got Run mail.
What changed
One rule, in the shared selection both lanes already use (
mailbox-pointer-eligibility.ts):selectOrchestrationPointerBatchnow returns the mail the held Delivery does not contain, plus the held Delivery's id. Mail inside the held batch is never re-pointed.You have N new orchestration messages after delivery <id>, which you have not acknowledged. If you no longer have it, run `<cli> orchestration check --run <run>` to see it again; once it is handled, run `<cli> orchestration check --run <run> --ack <id>`.The replay command comes first so an agent that lost the batch to/clearor compaction never acks mail it has not seen. (Same formatter and send path as every other pointer.)delivered_atstamp on those messages, so no new state or flags.shouldReleaseOrchestrationPointer(terminal lane, between write and Enter) now releases a staged pointer only when achecktook that pointer's own mail into a batch, not whenever any batch is open.delivered_at, so a steer an unacked turn ignores would spend that mail's only nudge.beforeRun) re-checks this, because a turn can start (or a queued user send can run first) between the check and the send. A conversation the idle sweep closed counts as not working, so it is woken.getOutstandingMailboxDelivery, checked narrowing, no assertion) on the existingoutstanding_deliveriesview.Tests:
mailbox-pointer-eligibility.test.ts(real DB): held rows stayread: 0,delivered_atnull and are not selected; rows outside are; ack then reselect names the next held batch.structured-chat-coordinator-mail.test.ts: a turn that starts while the nudge reads the chat gets no steer; the nudge arrives at its end. Fails without the admission re-check.structured-chat-coordinator-mail.test.ts: real runtime, RPC and chat host with a fake provider. Coordinator runscheckwithout--ack, mail arrives mid-turn: no steer, one wake at turn end naming the batch; a further turn end with nothing new stays quiet; newer mail while idle wakes it once more;--ackthen returns the new mail. Fails on main (no wake).orchestration-mailbox-notification-consistency.test.ts: the terminal lane equivalent (previously asserted the bug: zero pointers), including a turn that starts inside the Enter delay. Fails on main.orchestration-dispatch-mailbox-delivery.spec.ts: worker (Dispatch) mailbox, terminal lane, across a restart: the concurrent follow-up now gets exactly one pointer naming the held batch.tests/e2e/orchestration-idle-mail-delivery.spec.ts(Electron, real PTY, fake agent): a terminal coordinator holding an unacked Run delivery gets exactly one pointer naming it; a turn that starts inside the Enter delay holds the Enter until the next idle; a later idle edge stays quiet;--ackreturns the newer mail. Whole spec: 13/13.structured-mailbox-pointer-delivery.test.ts: worker Dispatch mailbox, chat lane scheduling gate: parked while working, sent on the next journal edge. This is the STA-11631 case (chat worker holding an unacked batch).Real app before / after
Isolated headless Orca built from each tree (
ORCA_BACKGROUND_LAUNCH=1, tempORCA_DEV_USER_DATA_PATHandHOME), a real native-chat Claude coordinator on--model haikuthat created and bound the Run. Branch column is from head 4e820ef; main from 843e502. Native chat script: mail 1 arrives, the coordinator runscheck --runwithout--ack, then stays mid-turn (waiting on a file); mail 2 arrives mid-turn; the turn ends; observe for 4 minutes.orca-dev orchestration check --run run_61cf291ab98eto see it again; once it is handled, runorca-dev orchestration check --run run_61cf291ab98e --ack delivery_949400d6fdf8."check ... --ack delivery_949400d6fdf8, got mail 2, acked it, replied "received"delivered_atafter 4 minReal Claude Code terminal coordinator (
claude --model haikuin an Orca terminal, tempHOME/CLAUDE_CONFIG_DIR), same script:orca-dev orchestration check --run run_a61671e3af58to see it again; once it is handled, run... --ack delivery_a774bdad1b0f."delivered_atafter 4 minAn earlier run where mail 2 arrived while the coordinator was already idle and holding an unacked batch showed the same split: main never woke it (
delivered_atNULL after 8 min), the branch woke it once and stayed quiet for the next 6 minutes while the agent still did not ack.Platforms, remote, mobile
check --ackworks as before. No migration.Risk
checktook the pointer's own mail). A pointer staged for mail that a latercheckdid not include still submits, which is correct: that mail is still unseen.Known gaps
checkstill returns it. Fixing it would need a durable "Enter deferred" pointer phase.Supersedes #29006.
Fixes STA-11980
Fixes STA-11631