Repository navigation
fix(server): never wedge a thread in stopping when sandbox teardown fails - #50
Merged
Merged
Conversation
Thread transfer impact✅ Thread transfer remains within every enforced ceiling.
Baseline: unavailable · PR result: Scenario and decoded snapshot size10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.
Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed. |
The sibling-removal phase of ContainerSandboxBackend.stop() listed the
thread's managed containers with `ps --format {{.ID}}` and excluded the
workspace container by comparing each listed ID against the workspace
NAME (t3-thread-<hex>). An ID never equals a name, and the listing
filters on exactly the labels the workspace container carries, so every
stop force-removed its own workspace container as a "sibling". The
`rm --force <name>` in #cleanup then failed, stop() collected the
failure and threw, SandboxLifecycleReactor.stop never dispatched
sandbox.stop.complete, and the decider left the thread in `stopping` --
where every thread.turn.start is rejected. A permanently unusable
thread on every orderly stop with a non-empty sibling listing.
List `{{.ID}}\t{{.Names}}` instead (same tab-column parsing as
reconcile's label listing; `--format` prints the plain name on both
docker and podman) and skip entries whose name is one #cleanup removes
by name -- the workspace container and, when configured, the egress
proxy sidecar. The per-candidate label-inspect verification before each
removal is unchanged.
The regression tests stub `ps` with a NON-EMPTY listing -- every prior
stop test stubbed it empty, which is exactly why this was missed -- and
answer whichever format the implementation asks for, so reverting the
fix fails them (verified: 2 failed on the reverted code, 27/27 pass
with it).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The decider accepts sandbox.stop from every non-terminal lifecycle -- including `unprovisioned`, `provisioning`, and `failed`, which never recorded a runtime -- and moves the thread to `stopping` unconditionally. The lifecycle reactor's stop handler then returned early when the sandbox had no docker/podman runtime, without ever dispatching sandbox.stop.complete: the thread stayed in `stopping` forever, where every thread.turn.start is rejected. This is the second independent cause of the stopping wedge, and the trap SandboxSettleCleanupReactor's narrow ready/paused guard was written around. Now the reactor always completes: when the sandbox projection records nothing to tear down, it skips the teardown and still dispatches sandbox.stop.complete so the decider can settle the lifecycle. When teardown IS attempted and runtimes.stop throws, the failure propagates to the worker's existing catchCause, which dispatches sandbox.operation.fail (stage `teardown`) carrying the runtime's real error message -- and the decider already resolves that from `stopping` to `failed`, a re-provisionable state, so no decider change was needed; a decider test now pins that transition. The settle-cleanup guard is deliberately left narrow in this PR; its comment now records that the underlying trap is gone. Reactor tests cover both paths: a `stopping` sandbox with no runtime dispatches exactly one stop.complete (verified failing on the reverted code), and a teardown failure dispatches operation.fail with the runtime's own message rather than propagating and wedging. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ProviderCommandReactor's per-thread provisionedTargets cache was only evicted from a snapshot the caller read before any lock wait, so an entry naming a container that a stop, an expiry, or a failure had already destroyed could still be served on a cache hit -- pointing the provider at a container name that no longer resolves. Validate at point of use instead: under the per-thread provision lock, a cache hit is served only if the CURRENT thread.sandbox projection is `ready` and still names the cached sandboxId/runtimeRef; anything else evicts the entry and re-provisions. Validating under the lock against a fresh projection read (rather than the caller's snapshot) also stops a stale pre-lock snapshot from throwing away a target another fiber just provisioned. Neither map needs bounding for memory -- one tiny entry per sandbox thread this server generation provisioned -- which is now documented at the declarations; staleness, not growth, was the defect. The regression test provisions, stops the sandbox through the real command path, and asserts the next turn re-provisions and hands the provider the new generation's container name, not the destroyed one (generation-suffixed names in the harness make the two distinguishable). Verified failing against an unvalidated bare cache hit: 3 failed, including this test; 53/53 pass with the fix. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
awtprod
force-pushed
the
impl/sandbox-stop-wedge
branch
from
August 22, 2026 02:08
0888a72 to
567eb73
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A thread whose sandbox teardown hits any snag gets stuck in
stoppingforever — every new turn is rejected and the thread is permanently unusable. Three release-blocking lifecycle bugs, all found on this branch and each sufficient to strand a user's thread on its own:stop()destroyed its own workspace container. The sibling-cleanup phase listed the thread's containers by ID but excluded the workspace container by comparing those IDs against its name — never equal — so every orderly stop force-removed its own workspace container as a "sibling". The later cleanuprmby name then failed,stop()threw, and the stop never completed. Fixed by listing{{.ID}}\t{{.Names}}and excluding by name (the egress sidecar, also removed by name later, is excluded the same way). The per-candidate label verification is unchanged.The lifecycle reactor early-returned without completing the stop. The decider accepts
sandbox.stopfrom every non-terminal lifecycle — includingunprovisionedandfailed, which never recorded a runtime — and moves the thread tostoppingunconditionally, but the reactor returned early for those threads without dispatchingsandbox.stop.complete. Now it always completes: nothing to tear down still dispatchesstop.complete, and a teardown failure flows to the existingsandbox.operation.failpath (stageteardown, carrying the runtime's real error), which the decider resolves fromstoppingtofailed— re-provisionable, not wedged. A decider test pins that transition.SandboxSettleCleanupReactor's narrow guard is deliberately left as-is; its comment now records that the trap it worked around is gone.Stale execution-target cache served dead containers.
ProviderCommandReactor's per-thread target cache was never invalidated when a sandbox stopped, expired, or failed, so a cache hit could hand the provider a destroyed container's name. Cache hits are now validated under the provision lock against the current sandbox projection (lifecycleready, samesandboxId/runtimeRef); anything else evicts and re-provisions. The maps are one tiny entry per sandbox thread, documented as such — staleness, not growth, was the defect.Every fix ships with regression tests that fail on the reverted code (verified by reverting each fix momentarily): the bug-1 tests stub
pswith a non-empty listing — every prior stop test stubbed it empty, which is why this survived — the bug-2 tests assert a completion or anoperation.failis dispatched in both no-container and teardown-failure paths, and the bug-3 test provisions, stops through the real command path, and asserts the next turn re-provisions rather than serving the dead generation's name.Stacked on #47 (
fix/sandbox-graceful-fallback); retarget tomainafter that merges.Verification:
vp checkexits 0,vpr typecheckexits 0, full server suiteCI=true pnpm -C apps/server testgreen (3263 passed, 19 skipped, 329 files).Claude Fable 5 via Claude Code