Self-hosted environment: worker spawned per claimed work item is told to shut down before executing (sandbox-per-session handoff broken; in-process pattern works) #1779

Summary

In a self-hosted environment (Managed Agents), the documented "sandbox per session" pattern — a dispatcher claims a work item via ant beta:worker poll --on-work spawn.sh, then spawns a fresh container whose worker process is supposed to execute that already-claimed item — reliably fails. The spawned worker starts fine, reaches api.anthropic.com fine, then shuts itself down within 2–6 seconds without executing anything (exit code 0). The session is left permanently stuck at stop_reason: requires_action and the work item is never re-offered to any worker.

We reproduced the identical failure with two independent client implementations:

• the ant CLI (ant beta:worker run), versions 1.15.0 and 1.17.0
• this SDK's EnvironmentWorker.handle_item()

Both fail at the same point, in the same way, which suggests the issue is server-side (or an undocumented requirement of the claim-handoff contract) rather than a client defect. Filing here because the SDK repro is the cleanest minimal case, and we don't have a direct support line.

Environment

• Environment ID: env_01UPQK3XYX67tnteLTvZw891 (type self_hosted)
• Agent ID: agent_014h5P54ckDkXpUG1NVdqY4G
• Runtime: Linux amd64, Kubernetes Job pod, Debian bookworm-slim base
• Reproduced identically across 5+ sessions, e.g. sesn_011zBktK84XqgQapwZCSunbU, sesn_017jQMiZ3mSv2bbtPFAWtayn, sesn_01GSgDdFAJ9JPKgx8oSt1UA6, sesn_01BjrqoHYCysBHz4tFboyfDU, sesn_014f4Y2BQrjJFaHZavJCxxVH
• SDK repro session: sesn_01GvoUeeRqmHe6dKbfPkCtcH

Reproduction steps

1. Create a self-hosted environment (POST /v1/environments, config.type: self_hosted); generate its environment key via the Console.
2. Run a dispatcher: ant beta:worker poll --on-work ./spawn.sh, where spawn.sh follows the documented shape — one fresh container per claimed work item, with ANTHROPIC_SESSION_ID, ANTHROPIC_WORK_ID, ANTHROPIC_ENVIRONMENT_ID, ANTHROPIC_ENVIRONMENT_KEY injected as env vars (our implementation spawns a Kubernetes Job rather than docker run; functionally equivalent).
3. Create an Agent (POST /v1/agents) and a Session (POST /v1/sessions referencing agent + environment_id).
4. Send a trivial user.message event, e.g. "List every file in /workspace and print their sizes."

SDK variant of the spawned worker (fails identically to the CLI)

async with AsyncAnthropic(auth_token=environment_key) as client:
    worker = EnvironmentWorker(
        client, environment_id=environment_id,
        environment_key=environment_key, workdir="/workspace",
    )
    await worker.handle_item(
        work_id=work_id, environment_id=environment_id,
        session_id=session_id, environment_key=environment_key,
    )

Observed behavior

• The dispatcher polls, claims exactly one work item (work_type=session), and spawns exactly one container per claim — this part works on every attempt.

• The spawned worker's logs show, consistently within 2–6 seconds of startup:

  msg="heartbeat reports shutdown" state=stopping
  msg="reconcile list failed" component=session-tool-runner error="context canceled"
  

  and the process exits 0. The SDK variant behaves the same (ran ~3s, exit 0, no error output).

• Meanwhile, via the Sessions API, the conversation proceeds normally on the model side: user.message → agent.thinking → agent.message → agent.tool_use (bash, evaluated_permission: allow), then stop_reason: requires_action referencing that tool_use, and session.status_idle.

• The tool_use is never fulfilled. No second work item is ever claimed for the session. It is stuck permanently.

Ruled out

• Our dispatch logic — found and fixed two real bugs in it (Jobs keyed by session_id instead of work_id; a heredoc quoting bug); failure persisted unchanged after both fixes.
• Env vars — verified all four are non-empty and correct on the running container via kubectl get pod -o jsonpath.
• Network — DNS + TLS to api.anthropic.com confirmed working from inside the spawned pod.
• --max-idle timing — tried 300s vs. the 60s default, plus explicit --environment-id; no change. Reverted to the exact zero-flag documented pattern for this report.
• Stale CLI — bumped 1.15.0 → 1.17.0 (the version pinned in the self-hosted-sandboxes doc's own Dockerfile examples as of 2026-07-17); identical failure.

Confirmed workaround (what narrowed it down)

The "in-process" pattern — a single long-running process that both claims and executes, ant beta:worker poll --workdir /workspace with no --on-work — works correctly on the first try: full tool-call cycle, results posted, clean stop_reason: end_turn.

So the failure is specifically in resuming an already-claimed work item in a process other than the one that claimed it. The observable shape (server heartbeat replying "shutdown" to the freshly spawned worker) is consistent with the claim/lease being bound to the claiming process's identity, so the handed-off worker is told to stop — but that's inference; we can't see the server side.

Question

Is there a known issue — or an undocumented handshake/requirement — for how a spawned ant beta:worker run / EnvironmentWorker.handle_item() process adopts a work item claimed by a different (dispatcher) process via the injected env vars? The session IDs above should allow server-side correlation. We've temporarily rearchitected to the in-process pattern, but we'd like to return to sandbox-per-session isolation once this is resolved.