Repository navigation
Concurrent prepare:* worktree tasks fail nondeterministically with a Bun-erased TypeError: undefined is not an object (evaluating 'get') #672
Description
Summary
Concurrently dispatched prepare:* (artifact-mirror / Worktree) tasks fail nondeterministically with TypeError: undefined is not an object (evaluating 'get'). Because these nodes are compiled with retries={0}, a single occurrence permanently fails the node and cascades into every downstream artifact contract.
This was the largest single source of node failures in an 18-hour local default-profile run: 55 of 214 NodeFailed events, 53 of them on prepare: nodes.
Evidence
Failure taxonomy for the run (214 NodeFailed events total):
| count | failure class |
|---|---|
| 97 | artifact-contract failure: artifact dependency has not … (downstream cascade) |
| 55 | TypeError: undefined is not an object (evaluating 'get') (53 on prepare: nodes) |
| 41 | unexpected status 402 Payment Required (separate provider-credit outage) |
| 9 | CLI timed out after 7200000ms |
| 5 | stream disconnected before completion |
| 4 | artifact schema validation |
| 3 | one each: workspace patch base tree mismatch, goal-plan child-count, 3600000ms timeout |
The 97 artifact-contract failures are consequences, not independent defects — which makes this TypeError the dominant root cause in the run.
Race signature
Two prepare: nodes dispatched in the same second, one succeeds, the other throws:
[00:14:53] → prepare:threat-model (attempt 1, iteration 0)
[00:14:53] → prepare:setup-foundry (attempt 1, iteration 0)
[00:14:55] ✓ prepare:threat-model (attempt 1)
error task execution failed nodeId=prepare:setup-foundry
iteration=0 attempt=1 maxAttempts=1
error="undefined is not an object (evaluating 'get')"
[00:14:56] ✗ prepare:setup-foundry (attempt 1): undefined is not an object (evaluating 'get')
A later burst shows the same pattern at higher fan-out — six prepare: nodes started within five seconds, some finishing and some failing with the identical message:
+03:01:12 NodeFailed ✗ prepare:<dynamic-goal-a> (attempt 1): undefined is not an…
+03:01:13 NodeFailed ✗ prepare:<dynamic-goal-b> (attempt 1): undefined is not an…
+03:01:13 NodeStarted → prepare:<dynamic-goal-c> (attempt 1, iteration 0)
+03:01:14 NodeFinished ✓ prepare:<dynamic-goal-d> (attempt 1)
+03:01:15 NodeFinished ✓ prepare:<dynamic-goal-c> (attempt 1)
+03:01:16 NodeFailed ✗ prepare:<dynamic-goal-e> (attempt 1): undefined is not an…
+03:01:17 NodeFailed ✗ prepare:<dynamic-goal-f> (attempt 1): undefined is not an…
Interleaved success and failure among simultaneously dispatched, otherwise-identical tasks is a concurrency race, not a per-node data problem.
The stack trace is erased
Every occurrence carries the same useless trace:
{
"name": "TypeError",
"message": "undefined is not an object (evaluating 'get')",
"stack": "TypeError: undefined is not an object (evaluating 'get')\n at run (node:async_hooks:68:37)\n at processTicksAndRejections (native:7:39)"
}Under Bun, the AsyncLocalStorage.run boundary discards the user frames, so the fault cannot be localised from run output at all. This is arguably the more serious half of the bug: an operator has no path from the symptom to a file and line.
Suspected cause
prepare:* tasks run prepareArtifactMirror(task) inside <Worktree>, and concurrent worktree preparation contends on the repository index. runGit in packages/runtime/src/workspace-handoff.ts passes env: undefined whenever no temporary index is supplied:
const env = index === undefined ? undefined : { ...process.env, GIT_INDEX_FILE: index };so index-scoped and non-index-scoped git invocations do not get a consistent environment. evaluating 'get' suggests a lookup against a map/registry that is momentarily absent, consistent with two preparations racing over shared state.
Not confirmed — the erased stack is why. Fixing the diagnostics is a prerequisite for confirming the cause.
Impact
- Largest failure source in a long real run, and each failure is terminal because
prepare:nodes compile withretries={0}. - Cascades: 97 further failures were downstream artifact contracts waiting on dependencies that never published.
- Undiagnosable from run output, so operators reach for unsupported recovery (see the control-seal issue filed alongside this one).
Acceptance criteria
prepare:*tasks preserve a usable stack trace across the BunAsyncLocalStorageboundary, naming a real file and line.- Concurrent
prepare:*dispatch at the configuredmax_parallel_nodesis race-free, with a regression test that dispatches several worktree preparations simultaneously and asserts all succeed. runGitsupplies a consistent environment on both the index and non-index paths.- A
prepare:failure does not silently become a terminal node failure withmaxAttempts=1.
Activity
aviggiano commentedon Aug 19, 2026
Material context for whoever picks this up: the retry budget from #673 does not reach these tasks.
prepare:is compiled withretries={0}hardcoded in the workflow template, somax_attempts: 2on the group defaults applies only to the agentic task, never the preparation wrapper. Every occurrence of this TypeError is therefore still terminal on first hit, and takes its downstream dependents with it.That matters because the failure is a race, which is the failure class a single retry absorbs best. Two
prepare:nodes dispatched in the same second, one succeeding and one throwing, withmaxAttempts=1printed on the failure line:Not proposing a fix here — it is deliberately deferred for now and tracked as a campaign risk downstream — but the
retries={0}detail is easy to miss when reading #673 as having covered retries generally.aviggiano commentedon Aug 20, 2026
Partially addressed by PR #689 (
fix/run-reliability-672-677→release/v0.1.0). Keeping this open — the PR references it rather than closing it, and that is correct.What landed: diagnosability only.
prepareArtifactMirrornow names each of its twelve preparation steps and wraps the throw,packages/runtime/src/templates/smithers/workflows/workflow.tsxL1502:with the original error preserved as
cause. The rationale is at L1488. Bun discards the user frames of an error thrown inside a task body, so all 53prepare:*failures in the measured run arrived with a stack of onlyat run (node:async_hooks:68:37)— 26% of every failure in that run with no file or line to bisect. Preparation is synchronous, so naming the steps recovers the one fact the erased stack cannot carry: which step threw.What did NOT land, and why this stays open:
prepare:tasks remain terminal. Themax_attempts: 2group budget from Shipped topology declares no max_attempts, so every agentic node compiles with retries: 0 and #572's bounded retries never engage #673 does not reach them: the preparation<Task>atworkflow.tsxL7026 is hardcodedretries={0}, independently of the group defaults, and unlike the goal-lane tasks it carries nocontinueOnFail. So aprepare:failure still fails the node and cascades downstream exactly as before. This is deliberate in the PR, not an oversight.In short: the 53 failures would now tell us where they came from, but the same run would still lose the same 53 nodes. Please do not read #689 merging as closing this.
Serialize concurrent prepare worktree tasks past the Bun-erased TypeE…
fix(runtime): stop retrying deterministic admission failures and labe…
fix(runtime): give agent retries a real wait, stop retrying determini…