Repository navigation
bug(sandbox): on Kubernetes, deleting a sandbox waits out the pods' termination grace period #5025
Description
Problem
On Kubernetes, deleting a sandbox takes the pods' full termination grace period, 30 s by default. The agent pod's openshell-sandbox and the sandbox's os-supervisor-… pod both catch SIGTERM but neither exits, so the kubelet kills them when the grace period ends.
This is the Kubernetes form of #4635, which was closed as not planned. The cost here is larger: Kubernetes deletes the agent pod and its supervisor pod together, and every sandbox delete waits the full grace period.
Observed
OpenShell v0.1.5 (bdb956f2), with the Helm chart and Kubernetes driver, on kind v0.33.0 with kindest/node:v1.37.0. NemoClaw's live tests delete one to three sandboxes per test.
When the Sandbox objects are deleted, the agent container logs:
WARN openshell_sandbox::boundary_server::linux: Boundary gRPC exchange failed
WARN openshell_sandbox::boundary_server::linux: Sandbox Protocol connection lost; workload frozen pending authenticated recovery
Neither container logs anything after that. Both pods stay Running until the grace deadline: the deletion is requested at 03:48:46, the deadline is 03:49:16, and the pods are gone at 03:49:15.
PID 1 in both containers catches SIGTERM (bit 15 of SigCgt) and blocks nothing:
| Container | PID 1 | SigCgt |
|---|---|---|
agent |
/.openshell/runtime/openshell-sandbox --bootstrap /.openshell/state/bootstrap/boundary.json |
0000000000014442 |
supervisor |
/openshell-supervisor --backend-descriptor-file … |
0000000100004442 |
The gateway pods in the same namespaces exit within a second of deletion. With the Docker driver, the same openshell-sandbox exits promptly on SIGTERM ("sandbox boundary received SIGTERM before supervisor confirmation").
Effect
Every sandbox delete on Kubernetes takes at least 30 s. In NemoClaw's three-agent live test the deletes are about 34 of 81 s.
Proposal
After a termination signal, the boundary and the supervisor terminate the workload and exit within a bounded time. In particular, the boundary should not wait for supervisor recovery once it has received SIGTERM.
Activity
ci: raise the Live / Kind per-test budget to 150 s (#13960)
chore(deps): upgrade OpenShell to v0.1.6 (#13961)
cv commentedon Nov 2, 2026
Measurements from NemoClaw's live Kubernetes tests, on OpenShell v0.1.6 (kind,
linux_arm64, the test limited to 4 CPUs to match CI runners):linux_amd64runners (The three-agent OpenShift live-kind test runs at its 150-second limit on linux_amd64 NemoClaw#14087).SUPERVISOR_TERMINATION_GRACE_PERIOD_SECONDS = 30incrates/openshell-driver-kubernetes/src/sandbox_runtime.rsonmain, and there's no setting to shorten it.A fix that lets the pods exit on SIGTERM would take those 31 s off every Kubernetes sandbox delete.
test(e2e): stop the kind HCL case's cleanup from waiting out namespac…
FrostGod commentedon Nov 4, 2026
Root cause for the
openshell-sandboxhalf: after SIGTERM,servekills the workload and returns, thenrun_boundarydrops its Tokio runtime, which waits with no limit for every blocking task. The supervisor keeps an agentWaitexchange open for the life of the sandbox.ManagedProcess::waitblocks on a condition variable that only the agent's async reaper task wakes. When the shutdown cancels the reaper before it records the exit,Waitnever returns and the process stays up until SIGKILL.On k3s, deletes on main take 32–33 s with the agent container SIGKILLed. With the bounded shutdown in #5279, the agent exits on its own at about 3 s and
openshell sandbox deletetakes 4–6 s. The supervisor pod can still wait out the grace period; that part isn't addressed yet.