Repository navigation
fix(orchestration): wake a chat coordinator whose ended turn left a batch unacknowledged - #29006
Closed
brennanb2025 wants to merge 12 commits into
Closed
brennanb2025 wants to merge 12 commits into
brennanb2025 wants to merge 12 commits into
Conversation
…atch unacknowledged An unacknowledged Run delivery suppressed every later pointer to that mailbox, on the assumption that its consumer was still working through it. A coordinator chat whose agent stopped after `check` but before acknowledging (Orca restart, stop, or a turn that simply ended) never acknowledges, and only the suppressed pointer could start the turn that would, so every later worker report sat unpointed. The structured lane now re-derives that hold from the host's current-work projection: the batch suppresses a nudge only while the session's agent is working. The next check replays the batch as before. run-show now reports the coordinator a worker would reach and whether it can read mail (live, exited, unverifiable, or none bound), and a send into a Run whose coordinator is exited or unbound carries a recipient_unreachable warning.
1 task done
…erated, legacy and unbound Runs
- run-show on a worker host's stub of a federated Run reads `unverifiable`
("The Run is coordinated from another host.") instead of "none bound".
- The legacy and unbound Runs never take a coordinator: run-show omits the
coordinator field and send stays quiet, through one shared predicate.
- A terminal with no liveness observation yet reads that, not "no record".
- The "starts it if it is at rest" clause is printed only for session
coordinators; the terminal lane still skips that nudge.
- The send warning names the Run instead of a dangling "it".
The sender-name lookup's current-row read moves to runtime-terminal-hook-row.ts so the mail hand-off reader can use the same row.
…ffered again once if it was cut A batch `check` hands out now records the turn that fetched it (schema v44 holder columns on deliveries); a notice's turn is already in the chat journal. How that turn ended is derived on every pass: running, ended or stopped hold the batch's members out of new notices; a cut turn (Orca quit, provider died, agent restarted in the same shell) offers its unread mail once more, bounded per runtime for a restart and once in all for any other cut; unknown does neither. Notices no longer wait behind an unacknowledged batch in either lane: a busy chat gets the notice as a queued card, like any message, and a card whose mail was already taken is withdrawn when it drains. A batch held at quit is re-offered at launch.
… accurate run-start line `check` text output now says when a batch is one handed out before and not acknowledged, and how to get newer mail. A worker heartbeat into a Run no longer carries the coordinator-unreachable warning. `run start` states the coordinator clause for every live coordinator, not only one started by mail.
…d stops, errors and edits right - `check --wait` records the turn running when it hands the batch out, not when the wait began, so a wait that outlives its turn records no holder instead of an ended one. - A held chat turn whose end is not written yet reads stopped when a person's Stop decides it, so a refused interrupt that kills the child can't re-offer mail after the Stop. - A turn the provider ended with an error reads cut: it did not act on the mail, which is offered once more within the per-message budget. - A queued mail card the person edits stops being a mail notice, so the queue never withdraws their text as spent mail.
…le, separate re-offer budgets - A terminal holder is its pane, the process in it and the hook-reported turn. A pane outlives the handle Orca mints per launch, so a coordinator whose shell was replaced across a restart reads cut and gets its batch once more. Run mail routing and run-show find a terminal coordinator through its pane too. - A new agent conversation in the same shell (`/clear`, a fresh or resumed agent) reads unknown, not cut: the hook row keeps only the current turn, so a handled batch is never fed to it. - Handing a batch out marks its messages delivered, which replaces the per-pass exclusion; one SQL fragment answers "in that batch" everywhere, and both lanes share one synchronous settle. - A staged terminal pointer is released by any batch fetched after staging, holder or not. - Two re-offer budgets per message: once per launch after a restart, once in all after other cuts. - Heartbeats no longer wake a coordinator; they are state, read with the next `check`. - `run start` promises a start on mail only to chat and structured-worker coordinators. - The queued-card check takes the chat's session, fails open, and is wired through a typed helper.
8 of 9 tasks
# Conflicts: # src/main/native-chat/agent-session-journal/queued-message-text-update.test.ts # src/main/runtime/construction/runtime-service-state.ts # src/main/runtime/workspaces/workspace-process-listing.ts
…ever start a turn Built from the 10-15 sequence: reports queued as cards through a busy turn, then read by that turn's check; only the card still owed starts a turn, including after a restart.
…it was open The hook server keeps, per pane, the last main-agent turn whose agent process ended mid-way. The pane's next row (a resumed or fresh agent) overwrites the status row, never this record, so a reader can still tell a cut turn from one that finished. Written only at process end; a session boundary or a turn-clock change is not evidence of a cut. In memory, bounded, and exposed to the runtime as `getEndedOpenAgentTurnForPane`.
…sumed or not, is offered again - Each hand-off records the turn it reached as one JSON reference: `deliveries.holder` for a fetched batch, `messages.pointed_by` for a notice typed into a terminal (its pane, process and the time it was submitted). v44 is reshaped to these columns, and a database already stamped 44 by a development build gains them on the next open. - A terminal turn reads cut when its pane's process was replaced or the hook server saw the agent process end with it open, which covers an agent resumed with `--resume`. A typed notice whose turn was cut before it ran `check` has its mail offered once more, within the same budgets. - Reading a holder never fails the `check` it rides on: an unreadable host records none.
This was referenced Oct 16, 2026
Merged
4 tasks done
11 tasks done
Contributor
|
Closing in favour of #33639, which merged on 10-25 and covers this case. When a coordinator ends its turn holding an unacknowledged batch, it now gets one notice at the turn edge, naming the held delivery and the exact |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ELI5
When a chat that coordinates workers stopped at the wrong moment, the workers' reports kept arriving but nothing ever woke the chat to read them. The workers couldn't tell either:
run-showsaid the coordinator was "null", which looks the same whether the coordinator is a live chat or is gone. Now the next report wakes the chat, andrun-showandsendtell a worker whether its coordinator can actually read mail.What Changed
What the user saw (Run
run_afb61cba2e2d, Brennan's batch of 21 Linear issues, 10-14):orca orchestration check, which handed it a batch of 50 messages. Two seconds later Orca shut down mid-turn, and the chat never acknowledged the batch.worker_donereports (STA-10383 draft PR 28890, STA-10473, PR 28818), escalations ("Parent coordinator unavailable for required round-three answer", "Blocked: Brennan round-three decision pending") and go/no-go questions. The chat sat idle the whole time.run-show --jsonprintedcoordinator_handle: null, which is normal for a chat coordinator, so a working coordinator and a gone one looked the same. One worker (STA-10535) then sent its question to an unrelated session that had happened to message it last.Why it happened. A coordinator reads its mail with
check, which hands it a batch it must acknowledge on its nextcheck. Until then the batch counts as "outstanding". Both wake paths (a nudge typed into a terminal, or a turn sent into a chat) skip a mailbox with an outstanding batch, on the assumption that the coordinator is still working through it. Nothing ever checked that assumption:The same rule also strands a chat that is idle but still alive: if it ends a turn without acknowledging, no new report wakes it. That didn't happen in this incident, because the coordinator ran back-to-back turns until it died.
After:
structured-mailbox-pointer-delivery.ts): an outstanding batch blocks a nudge only while the chat's agent is actually working. That is read off the host's current-work projection, the same one the chat view uses, so a turn killed with Orca reads as not working. When the turn has ended, a new report wakes the chat (starting its agent if it is at rest). Its nextcheckreplays the old batch, as it already did.run-show: now returns acoordinatorfield, computed on each call and never stored:{ address, status: live | exited | unverifiable, reason? }, ornullwhen no coordinator is bound.coordinator:line.live, because mail starts it. A chat the user closed readsexited (The chat was closed.).send: a message into a Run whose coordinator isexitedor unbound still queues, but the result now carries the existingrecipient_unreachablewarning. The warning tells the worker the message waits unread and to tell the user rather than pick another recipient. Anunverifiablecoordinator stays quiet, because losing sight of a coordinator is not evidence it is gone.Why
The common pattern for a parent agent whose children report back is that a "the parent will pick this up itself" claim lasts only as long as the turn that made it. Once that turn is cut, the child's report wakes the parent with a new turn. Here that claim was a database row with no turn attached, so a dead turn kept it alive. Scoping the row to a working turn removes this whole class (killed, stopped, or simply ended without acknowledging) instead of special-casing Orca restarts.
Alternatives considered:
Differences from the common pattern:
check, and acknowledging protects a batch whose reader dies mid-read. Only the "still holding it" claim changes.Found in the same incident, fixed separately: a worker in this Run ran
run-useon its own parent Run three times (19:48, 20:31 and 23:24 UTC), each time silently taking the coordinator binding away from the real coordinator. Two more workers did the same in other Runs that night. That is a different structural fact (any caller can bind any Run, including the Run it works for), so it gets its own PR.Linked Issue
None (maintainer; diagnosed from the live orchestration database, read-only).
Visual Proof
N/A: no rendered UI change. The CLI
run-showtext gains onecoordinator:line.Testing
I manually tested these changes locally
Automated tests added/updated, or explained why not below
New flow test in
structured-chat-coordinator-mail.test.ts, on the in-process runtime, host and fake provider rig:checkreplays the held batch.run-showreadslive. After the user closes the chat it readsexited, and a worker'ssendcarriesrecipient_unreachable.With the wake change reverted, the test fails: the chat is never started (
expected 1 to be 2connections). With it, the test passes.Focused Node vitest on macOS: 1,217 tests across 107 files under
rpc/methods/orchestration, the orchestration session suites, the structured-chat coordinator suites, structured mailbox delivery and agent-facing parity; plus 194 tests across 22 files of the CLI orchestration handlers. All pass.Not run locally: typecheck and full lint (CI covers them). Not exercised: a real Orca app, SSH and Windows. The change reads only the local chat host's own projection, and the terminal and SSH paths are unchanged.
Review
Agent skill upstream boundary
Notes
coordinatoronorchestration.runShowis a new optional field, so old CLIs ignore it and a new CLI prints nothing extra against an old runtime. The send warning reuses an existing code. No schema change.unverifiableand raises no warning.run-showand per send into arun:mailbox.Author: @BrennanKB5
Checklist
N/Awith reasonpnpm lint,pnpm typecheck,pnpm test, andpnpm buildpass (or CI will cover; local preferred)