Skip to content

Continue threads after restarts behind an experiment - #6612

Draft
ymichael wants to merge 4 commits into
mainfrom
bb/continue-threads-after-restarts
Draft

ymichael wants to merge 4 commits into
mainfrom
bb/continue-threads-after-restarts

Conversation

@ymichael

@ymichael ymichael commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

Human comments

What was wrong

When bb restarts, whether from quitting it, a crash, an update, or a machine reboot, every running thread loses its turn and is left in error. Nothing tells the agent what it was doing, so the user has to notice and type "continue" in each thread. Retry plugins also see these as ordinary failures. A parent blocked in bb thread wait <child> could also get "will not reach idle by waiting alone… send a follow-up" during the seconds before the child's resume starts, and would then send the child a second instruction. This is the first half of #6127. The second half, updating when idle, will follow in a separate PR.

What changed

All of this is behind a new continueThreadsAfterRestarts experiment, which is off by default.

When bb records a resume (apps/server/src/services/threads/restart-resume.ts):

  • When the server shuts down, it records one resume message for each running thread. The message waits on a far-future time, so none can be sent while the server is still shutting down.
  • When a host daemon comes back as a new process (session-owner-side-effects.ts), or reports threads it no longer has (reconcileDaemonReportedThreads), bb records a resume before interrupting each cut turn. Idle threads whose awaited background work died get one too; background shells such as dev servers don't count.
  • At boot, each resume waits for its thread's turn if that turn is still running on a machine that kept going. Otherwise it becomes due.

What the message says: a cut turn gets "Continue where you left off." (system-message-restart-resume.md). An idle thread whose awaited background work died gets a note listing that work, saying it will no longer report back and asking the agent to restart what it still needs (system-message-restart-resume-background.md).

How resumes go out:

  • A few seconds apart per machine, children before their parents, and at the front of the thread's queue.
  • Messages already waiting for the thread are rescheduled just behind its resume.

When a resume is dropped (checked when it reaches the front of the queue):

  • the turn actually finished;
  • the user stopped it, or another non-restart interruption happened;
  • the user sent a newer message;
  • it would be the fourth automatic resume in a row.

Effects elsewhere:

  • A turn a restart cut off isn't announced as turn.failed, so retry policies don't also act. A resume that fails to send stops this suppression.
  • The parent of an interrupted child gets no "interrupted" notice. It hears when the child finishes, or from the existing failure notice if the resume fails.
  • The interruption error says "bb will send this thread a message to continue" instead of "Please retry".
  • SDK threads.wait (and bb thread wait) keeps waiting on a thread in error that has a message scheduled to send on its own. That covers a scheduled resume or a scheduled provider retry. Before, it failed with exit 4.

Other pieces:

  • Database queries in packages/db, the restart-resume notice kind and templates, a "Resumed after a restart" timeline row with a new built-in Replay icon, and the experiment's description.
  • Docs in docs/configuration.md, bb guide customization and bb guide thread-inspection.
  • No daemon protocol change, and no plugin SDK change (the version guard passes).

How you verified

Tests. Each fix was checked by undoing it and watching its test fail.

  • apps/server/test/threads/restart-resume.test.ts:
    • a daemon restart resumes cut turns and awaited subagents, but not a dev server;
    • no retry is announced, and the error detail changes;
    • resumes recorded at shutdown are dropped when the turn finished or the user stopped it, and sent when a daemon restart cut it;
    • a child is resumed before its parent, and the parent gets no notice;
    • the 3-in-a-row limit;
    • nothing happens with the experiment off.
  • apps/server/test/threads/turn-failed-retry.test.ts:
    • no turn.failed for a turn a restart cut, and the resume starts the next turn;
    • the announcement returns once a resume fails;
    • the resume is dropped when the user sent something newer.
  • apps/cli/src/__tests__/command-output/thread-wait.test.ts: waiting continues on a failed thread with a scheduled message, and still exits 4 without one.
  • pnpm exec turbo run typecheck lint --filter='...[origin/main]' passes (153 tasks).
  • Tests pass for @bb/server, @bb/db, @bb/domain, @bb/sdk, @bb/cli, @bb/templates, @get-bb/plugin-sdk, and the app's timeline and settings-view tests.
  • node .github/workflows/check-plugin-sdk-version.mjs reports the surface unchanged.
  • At b860c60bb3: full suites for @bb/server (3773), @bb/app (6521), @bb/cli (866), @bb/db (634), @bb/domain (294), @bb/sdk (128) and @bb/templates (46) pass, and typecheck --filter='...[origin/main]' passes.

Real runs, Oct 30 on a fresh pnpm start:worktree with real Claude Code (Opus 5.5) threads, at b860c60bb3. Each thread runs a 60 s foreground command, writes a second file, then replies ALL DONE.

Scenario Result
Experiment off, graceful restart mid-command error, "Please retry the thread to continue.", nothing sent (same as main)
Experiment on, graceful restart mid-command 1 resume parked at shutdown, sent after boot; the agent finished the task
Parent blocked in bb thread wait on a running child Child resumed first, parent got no "interrupted" notice, no second child
Thread stopped with bb thread stop before the restart, and an idle thread No resume
kill -9 of the server and daemon mid-command Resume created when the daemon reconnected; task finished
bb thread wait started right after boot, while the resume was queued Kept waiting, exit 0
Duplicates Exactly one resume per cut thread; queue empty afterwards

Things noticed during these runs, not changed here:

  • With the short "Continue where you left off." text, 3 of 4 resumed Claude Code agents read the cut tool call as the user's rejection ("picking up from the earlier rejection") and reran the step without checking whether it had already run. That was harmless for these commands, but would matter for a non-idempotent one.
  • Resumes go out on the 10 s due-scheduled-queue-dispatch sweep, so the 3 s delay and 5 s per-machine stagger round up to 10 s ticks. A child and its parent can go out in the same tick; children still go first.
  • While it waits (about 10 s after boot) the resume shows in the composer's Queue drawer as "From System", with the normal queue actions.

Earlier runs on the prototype branch, with the same reconnect code, also covered bb updates app apply with a parent and child, and quitting bb and starting it again.

Part of #6127; the update-when-idle half follows in a separate PR.

AGENT GENERATED

🤖 Generated with Claude Code

ymichael and others added 4 commits October 29, 2026 17:01
With the continueThreadsAfterRestarts experiment on, a thread that a quit,
crash, update or machine restart interrupted gets one message naming what it
was doing, such as the command it was running, so the agent checks it and
carries on.

bb records a resume for each running thread when the server shuts down, and
for each cut turn or awaited background work (not background shells) when a
host daemon comes back as a new process. Resumes go out a few seconds apart
per machine, child threads before their parents, ahead of messages already
queued for the thread. A resume is dropped if the turn finished, the user
stopped it, or the user sent something newer, and bb stops after three
automatic resumes in a row. A turn a restart cut is not announced to retry
policies, the parent of an interrupted child hears only when the child
finishes, and bb thread wait keeps waiting on a failed thread whose resume
is scheduled.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A graceful shutdown interrupts the provider, and Claude Code records the
cut tool call as rejected by the user. A resumed agent trusted that over
the resume message and stopped to ask, so the message now says a recorded
interruption or rejection was the restart.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ymichael added a commit that referenced this pull request Oct 30, 2026
## Human comments

## What was wrong

At boot the server starts its HTTP listener and loads plugins in the
background (`startServerPlugins`, which ends in
`markRegistrationsSettled`). A daemon that restarted along with bb
reconnects during that window. `handleHostSessionOpened` then
immediately interrupts the turns the new daemon process no longer has,
firing `turn.failed` and `thread.failed`. Plugin events go only to
plugins that have already loaded and are never replayed, so a plugin
still loading at boot never hears that the restart cut those turns. Turn
sends already wait for plugin startup (`whenRegistrationsSettled`).
Reconnect recovery didn't. This blocks plugins (and #6612) from reacting
to turns a restart cut.

## What changed

`apps/server/src/internal/session-owner-side-effects.ts`, in
`handleHostSessionOpened`:
- The replaced session is still retired right away (cancel its pending
disconnect, close its socket, tell terminal sessions).
- Recovery then waits for
`deps.providerRegistry.whenRegistrationsSettled()`, the same wait with
the same 30s cap that sends use. That covers interrupting environment
provisioning, pending interactions and active turns, settling dangling
background tasks, and `reconcileDaemonReportedThreads`. The
`/internal/session/open` response is held until recovery finishes, so
the daemon's WebSocket can't send events that conflict with the
active-thread snapshot it reported. Plugin startup doesn't depend on a
connected daemon, so this can't deadlock. After a boot, a machine shows
as connected only once plugins have loaded (or after 30s).
- A session replaced during the wait skips recovery. Its thread report
is out of date, and the newer session's handler recovers the host using
the replaced session as its previous one. Without this, an out-of-date
report could revive a thread (`run.started`) and its turn would then be
failed a second time.

No wire change, no protocol bump, no plugin SDK change.

## How you verified

- New
`apps/server/test/internal/reconnect-recovery-waits-for-plugins.test.ts`,
using a provider registry with `deferRegistrationsSettled: true`:
- A daemon restarting before plugins settle: the cut turn stays `active`
and no `turn.failed` fires. After settling, the thread is in `error`
with exactly one `turn.failed`. Fails without the wait: the thread was
already `error`.
- Two daemon restarts during the wait, where the first (out-of-date)
open reports the thread active: exactly one `turn.failed`. Fails without
the replaced-session check: `turn.failed` fired twice.
- `pnpm exec turbo run test lint --filter=@bb/server`: 368 files / 3789
tests pass, lint clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

> AGENT GENERATED

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@ymichael
ymichael marked this pull request as draft October 30, 2026 20:25

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant