Skip to content

fix(orchestration): wake a chat coordinator whose ended turn left a batch unacknowledged - #29006

Closed
brennanb2025 wants to merge 12 commits into
mainfrom
brennanb2025/fix-orphaned-run-coordinator
Closed

brennanb2025 wants to merge 12 commits into
mainfrom
brennanb2025/fix-orphaned-run-coordinator

Conversation

@brennanb2025

@brennanb2025 brennanb2025 commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor
Files Added Deleted Net
Test 11 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​450 $\color{#cf222e}{\Huge{\mathbf{−}}}$​58 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​392
Prod 55 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​1309 $\color{#cf222e}{\Huge{\mathbf{−}}}$​130 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​1179

ELI5

When a chat that coordinates workers stopped at the wrong moment, the workers' reports kept arriving but nothing ever woke the chat to read them. The workers couldn't tell either: run-show said the coordinator was "null", which looks the same whether the coordinator is a live chat or is gone. Now the next report wakes the chat, and run-show and send tell a worker whether its coordinator can actually read mail.

What Changed

What the user saw (Run run_afb61cba2e2d, Brennan's batch of 21 Linear issues, 10-14):

  • A native chat coordinated the Run. At 23:50:35 UTC it ran orca orchestration check, which handed it a batch of 50 messages. Two seconds later Orca shut down mid-turn, and the chat never acknowledged the batch.
  • Over the next 5 hours, 39 more messages arrived for the coordinator and none were ever pointed to it. They included worker_done reports (STA-10383 draft PR 28890, STA-10473, PR 28818), escalations ("Parent coordinator unavailable for required round-three answer", "Blocked: Brennan round-three decision pending") and go/no-go questions. The chat sat idle the whole time.
  • Workers had no way to know. run-show --json printed coordinator_handle: null, which is normal for a chat coordinator, so a working coordinator and a gone one looked the same. One worker (STA-10535) then sent its question to an unrelated session that had happened to message it last.

Why it happened. A coordinator reads its mail with check, which hands it a batch it must acknowledge on its next check. Until then the batch counts as "outstanding". Both wake paths (a nudge typed into a terminal, or a turn sent into a chat) skip a mailbox with an outstanding batch, on the assumption that the coordinator is still working through it. Nothing ever checked that assumption:

  • A chat whose turn ended before acknowledging can only acknowledge in a later turn.
  • For a chat, the only thing that starts a later turn is the nudge this rule was suppressing.
  • So the batch stayed outstanding forever and silenced every later report. The coordinator's Run binding was correct the whole time: it named a chat that mail would have started.

The same rule also strands a chat that is idle but still alive: if it ends a turn without acknowledging, no new report wakes it. That didn't happen in this incident, because the coordinator ran back-to-back turns until it died.

After:

  • Wake (structured-mailbox-pointer-delivery.ts): an outstanding batch blocks a nudge only while the chat's agent is actually working. That is read off the host's current-work projection, the same one the chat view uses, so a turn killed with Orca reads as not working. When the turn has ended, a new report wakes the chat (starting its agent if it is at rest). Its next check replays the old batch, as it already did.
  • run-show: now returns a coordinator field, computed on each call and never stored: { address, status: live | exited | unverifiable, reason? }, or null when no coordinator is bound.
    • The text output gets a matching coordinator: line.
    • A chat at rest reads live, because mail starts it. A chat the user closed reads exited (The chat was closed.).
  • send: a message into a Run whose coordinator is exited or unbound still queues, but the result now carries the existing recipient_unreachable warning. The warning tells the worker the message waits unread and to tell the user rather than pick another recipient. An unverifiable coordinator stays quiet, because losing sight of a coordinator is not evidence it is gone.

Why

The common pattern for a parent agent whose children report back is that a "the parent will pick this up itself" claim lasts only as long as the turn that made it. Once that turn is cut, the child's report wakes the parent with a new turn. Here that claim was a database row with no turn attached, so a dead turn kept it alive. Scoping the row to a working turn removes this whole class (killed, stopped, or simply ended without acknowledging) instead of special-casing Orca restarts.

Alternatives considered:

  • Fence outstanding batches when a session exits. This depends on an exit event that a crash may never deliver.
  • Record which process holds each batch. This needs a schema change. The host already derives "is this agent working", so storing a second copy of that fact isn't needed.

Differences from the common pattern:

  • Intended: Orca keeps fetch-and-acknowledge mailboxes. In the common pattern, a report is the parent's next turn. Here workers and coordinators on terminals, chats and other hosts all read through check, and acknowledging protects a batch whose reader dies mid-read. Only the "still holding it" claim changes.
  • Intended: mail to a chat the user closed is kept and the sender is warned. In the common pattern, a report to an archived parent is dropped. Here the mail is still there if the user reopens that chat or another coordinator takes over the Run.
  • Temporary: a batch fetched by a turn that then died is re-pointed only when the next report arrives or the chat runs another turn. Live workers heartbeat every 5 minutes, so this covers a running Run. A final report that was already inside the dead batch still waits for the user's next message. Follow-up: point such a batch once when its turn ends.
  • Temporary: terminal coordinators keep the old rule. A terminal pointer is staged and then released if a batch is outstanding, and that release can't yet tell an old batch from one fetched in the meantime. Follow-up: apply the same working-turn rule there.

Found in the same incident, fixed separately: a worker in this Run ran run-use on its own parent Run three times (19:48, 20:31 and 23:24 UTC), each time silently taking the coordinator binding away from the real coordinator. Two more workers did the same in other Runs that night. That is a different structural fact (any caller can bind any Run, including the Run it works for), so it gets its own PR.

Linked Issue

None (maintainer; diagnosed from the live orchestration database, read-only).

Visual Proof

N/A: no rendered UI change. The CLI run-show text gains one coordinator: line.

Testing

  • I manually tested these changes locally

  • Automated tests added/updated, or explained why not below

  • New flow test in structured-chat-coordinator-mail.test.ts, on the in-process runtime, host and fake provider rig:

    1. The coordinator checks, then its agent stops before acknowledging.
    2. The next worker report starts the chat, and its check replays the held batch.
    3. run-show reads live. After the user closes the chat it reads exited, and a worker's send carries recipient_unreachable.
  • With the wake change reverted, the test fails: the chat is never started (expected 1 to be 2 connections). With it, the test passes.

  • Focused Node vitest on macOS: 1,217 tests across 107 files under rpc/methods/orchestration, the orchestration session suites, the structured-chat coordinator suites, structured mailbox delivery and agent-facing parity; plus 194 tests across 22 files of the CLI orchestration handlers. All pass.

  • Not run locally: typecheck and full lint (CI covers them). Not exercised: a real Orca app, SSH and Windows. The change reads only the local chat host's own projection, and the terminal and SSH paths are unchanged.

Review

Agent skill upstream boundary

  • Not applicable, or this change copies or mechanically translates no upstream skill-installer source, tests, fixtures, registry entries, path tables, comments, or documentation.

Notes

  • Wire: coordinator on orchestration.runShow is a new optional field, so old CLIs ignore it and a new CLI prints nothing extra against an old runtime. The send warning reuses an existing code. No schema change.
  • SSH and remote: a terminal coordinator on a host Orca can't see reads unverifiable and raises no warning.
  • Performance: one record lookup per run-show and per send into a run: mailbox.

Author: @BrennanKB5

Checklist

  • This PR is small and focused
  • I explained what changed and why (ELI5, the user-facing before/after, the mechanism, and why over the alternatives)
  • Before/after screenshots or videos attached for UI changes, or N/A with reason
  • Self-reviewed for correctness, security, and performance
  • Cross-platform, SSH/remote, and path/shortcut impact considered (or N/A)
  • pnpm lint, pnpm typecheck, pnpm test, and pnpm build pass (or CI will cover; local preferred)

…atch unacknowledged

An unacknowledged Run delivery suppressed every later pointer to that mailbox, on the
assumption that its consumer was still working through it. A coordinator chat whose
agent stopped after `check` but before acknowledging (Orca restart, stop, or a turn
that simply ended) never acknowledges, and only the suppressed pointer could start
the turn that would, so every later worker report sat unpointed.

The structured lane now re-derives that hold from the host's current-work projection:
the batch suppresses a nudge only while the session's agent is working. The next
check replays the batch as before.

run-show now reports the coordinator a worker would reach and whether it can read
mail (live, exited, unverifiable, or none bound), and a send into a Run whose
coordinator is exited or unbound carries a recipient_unreachable warning.
…erated, legacy and unbound Runs

- run-show on a worker host's stub of a federated Run reads `unverifiable`
  ("The Run is coordinated from another host.") instead of "none bound".
- The legacy and unbound Runs never take a coordinator: run-show omits the
  coordinator field and send stays quiet, through one shared predicate.
- A terminal with no liveness observation yet reads that, not "no record".
- The "starts it if it is at rest" clause is printed only for session
  coordinators; the terminal lane still skips that nudge.
- The send warning names the Run instead of a dangling "it".
The sender-name lookup's current-row read moves to runtime-terminal-hook-row.ts so the
mail hand-off reader can use the same row.
…ffered again once if it was cut

A batch `check` hands out now records the turn that fetched it (schema v44 holder columns on
deliveries); a notice's turn is already in the chat journal. How that turn ended is derived on
every pass: running, ended or stopped hold the batch's members out of new notices; a cut turn
(Orca quit, provider died, agent restarted in the same shell) offers its unread mail once more,
bounded per runtime for a restart and once in all for any other cut; unknown does neither.

Notices no longer wait behind an unacknowledged batch in either lane: a busy chat gets the notice
as a queued card, like any message, and a card whose mail was already taken is withdrawn when it
drains. A batch held at quit is re-offered at launch.
… accurate run-start line

`check` text output now says when a batch is one handed out before and not acknowledged, and
how to get newer mail. A worker heartbeat into a Run no longer carries the coordinator-unreachable
warning. `run start` states the coordinator clause for every live coordinator, not only one
started by mail.
…d stops, errors and edits right

- `check --wait` records the turn running when it hands the batch out, not when the wait began, so a
  wait that outlives its turn records no holder instead of an ended one.
- A held chat turn whose end is not written yet reads stopped when a person's Stop decides it, so a
  refused interrupt that kills the child can't re-offer mail after the Stop.
- A turn the provider ended with an error reads cut: it did not act on the mail, which is offered
  once more within the per-message budget.
- A queued mail card the person edits stops being a mail notice, so the queue never withdraws their
  text as spent mail.
…le, separate re-offer budgets

- A terminal holder is its pane, the process in it and the hook-reported turn. A pane outlives
  the handle Orca mints per launch, so a coordinator whose shell was replaced across a restart reads
  cut and gets its batch once more. Run mail routing and run-show find a terminal coordinator
  through its pane too.
- A new agent conversation in the same shell (`/clear`, a fresh or resumed agent) reads unknown,
  not cut: the hook row keeps only the current turn, so a handled batch is never fed to it.
- Handing a batch out marks its messages delivered, which replaces the per-pass exclusion; one SQL
  fragment answers "in that batch" everywhere, and both lanes share one synchronous settle.
- A staged terminal pointer is released by any batch fetched after staging, holder or not.
- Two re-offer budgets per message: once per launch after a restart, once in all after other cuts.
- Heartbeats no longer wake a coordinator; they are state, read with the next `check`.
- `run start` promises a start on mail only to chat and structured-worker coordinators.
- The queued-card check takes the chat's session, fails open, and is wired through a typed helper.
# Conflicts:
#	src/main/native-chat/agent-session-journal/queued-message-text-update.test.ts
#	src/main/runtime/construction/runtime-service-state.ts
#	src/main/runtime/workspaces/workspace-process-listing.ts
…ever start a turn

Built from the 10-15 sequence: reports queued as cards through a busy turn, then read by that
turn's check; only the card still owed starts a turn, including after a restart.
…it was open

The hook server keeps, per pane, the last main-agent turn whose agent process ended mid-way. The
pane's next row (a resumed or fresh agent) overwrites the status row, never this record, so a
reader can still tell a cut turn from one that finished. Written only at process end; a session
boundary or a turn-clock change is not evidence of a cut. In memory, bounded, and exposed to the
runtime as `getEndedOpenAgentTurnForPane`.
…sumed or not, is offered again

- Each hand-off records the turn it reached as one JSON reference: `deliveries.holder` for a
  fetched batch, `messages.pointed_by` for a notice typed into a terminal (its pane, process and
  the time it was submitted). v44 is reshaped to these columns, and a database already stamped 44
  by a development build gains them on the next open.
- A terminal turn reads cut when its pane's process was replaced or the hook server saw the agent
  process end with it open, which covers an agent resumed with `--resume`. A typed notice whose
  turn was cut before it ran `check` has its mail offered once more, within the same budgets.
- Reading a holder never fails the `check` it rides on: an unreadable host records none.
@Jinwoo-H

Copy link
Copy Markdown
Contributor

Closing in favour of #33639, which merged on 10-25 and covers this case. When a coordinator ends its turn holding an unacknowledged batch, it now gets one notice at the turn edge, naming the held delivery and the exact check --ack command, in both native chat and terminals. Thanks for the original idea here; it shaped that fix.

@Jinwoo-H Jinwoo-H closed this Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants