Generated retry restoration fails on a transient linked-worktree index lock

Summary

On release/v0.1.0 (9b0bc8cab2e681bbffeee385c1f80f85a47087d3), the generated default workflow's retry restoration fails immediately when the bundled retry transaction briefly owns the selected linked worktree's index.lock. The lock disappears when that current transaction exits, but restoreWorkspacePatchPreparation invokes git read-tree --reset -u only once, so the logical retry fails before model execution.

This was reproduced with a generic temporary Git repository. No target identity or target content is included here.

Minimal generic reproduction

Create a temporary Git repository with one commit and a linked worktree.

Resolve its administrative index path:

index_path=$(git -C "$linked_worktree" rev-parse --path-format=absolute --git-path index)

In a background process, create ${index_path}.lock, retain it for 250 ms, then unlink it.
After 50 ms, invoke the primitive used by generated helper restoreWorkspacePatchPreparation once.
Invoke the identical command again after the owner exits.

Observed:

first_status=128
fatal: Unable to create '<temp-path>': File exists.
retry_after_transient_owner_exit=success

UltraFuzz-owned failure path

The generated helper currently performs one fail-fast call:

execFileSync("git", ["read-tree", "--reset", "-u", preparationTree], {
  cwd: workspaceRoot,
  stdio: ["ignore", "pipe", "pipe"]
});

In cloud recovery evidence, the retained orphan lock had already been removed. The bundled retry emitted RunStarted, then failed during workflow rendering at this helper; the lock was absent again after the transaction exited. That distinguishes a transient current-transaction collision from a stale retained lock.

Expected behavior

For this exact runtime-owned reset primitive, the generated workflow should recognize only Git's index-lock collision, wait for the validated selected linked-worktree lock to disappear, and retry within a small fixed bound. Other Git failures must remain immediate and terminal. The default topology definition and ordinary non-retry execution should remain unchanged.

Follow-up from a same-run cloud retry on the affected release:

The retry command itself received the scoped wrapper directory at the front of PATH, but generated task rendering restored the run's previously captured executable lookup and invoked the image Git path directly. The PATH-only compatibility shim was therefore not reached; the retained log showed the same one-shot read-tree --reset -u <tree> failure after 17 ms, rather than any bounded wait.

A generic reproduction is a launcher that receives a temporary PATH wrapper, then starts its generated callback with a previously captured PATH or absolute Git path. The launcher observes the wrapper while the callback bypasses it and reproduces the transient linked-worktree lock failure.

The local release compatibility fix remains confined to the already exact-task-gated retry. It now validates the canonical image Git executable, atomically retains it by inode, interposes the same bounded wrapper only for that retry subprocess window, and atomically restores the original executable afterward. Ordinary Git commands delegate unchanged. The focused regression invokes the captured executable path directly, proves a 250 ms regular lock clears successfully, proves an unrelated Git error is byte-for-byte unchanged, rejects a symlink lock, and verifies the original executable digest is restored. The full Modal package suite passes 553/553.

Follow-up from another same-run recovery on the affected release:

The retry reset was durably accepted, but the resumed engine emitted RunStarted, committed its recovery frames, and returned RunFailed before any new selected-task NodeStarted. The retry operation therefore rolled the reset back safely and no model request began.

Two disposable production-image probes narrowed the remaining compatibility gap without mounting or changing campaign state:

Calling the atomically interposed image Git path from an ordinary Node or Bun child waits through the bounded live lock and restores the original executable.
A Bun process honors a retry-scoped BUN_OPTIONS=--preload=<validated-file> hook, and that hook can synchronize the patched node:child_process binding before an ESM workflow imports execFileSync.

The local release compatibility fix now adds that process-local route only during the already exact-task-gated retry. The preload redirects only execFileSync("git", ["read-tree", "--reset", "-u", <40-hex-tree>], ...) to the existing bounded wrapper; every other child-process call and Git command remains unchanged. The preload lives in the worker-owned mode-0700 temporary directory and is removed with the existing wrapper cleanup.

The focused regression launches a Bun ESM renderer with its original PATH, retains a regular linked-worktree lock for 250 ms, proves the selected reset waits and succeeds, and retains the existing unrelated-error, unsafe-lock, executable-restoration, and cleanup assertions. The complete Modal package suite passes 553/553, with typecheck, build, formatting, and diff checks also passing.

Follow-up from the next recovery on the affected release:

The retry worker used the validated preload-enabled image and reached the generated retry transaction, but the retained workflow process still entered its imported execFileSync binding directly. It emitted RunStarted, committed both recovery frames, and returned the same transient selected-worktree read-tree --reset -u lock failure before a new task attempt began. The lock was absent again after process exit. This remains the same issue rather than a new bug.

The local compatibility fix now makes preload activation explicit at the already exact-task-gated worker boundary: only the selected retry is launched as the validated image Bun with --preload=<worker-owned-file>, while the inherited BUN_OPTIONS remains in place for a delegated Bun child. Ordinary resume, fresh execution, topology materialization, and unrelated subprocesses remain unchanged.

The focused regression executes the retry CLI under the explicit Bun/preload argv, delegates once to a second Bun process, imports the ESM child_process binding there, retains a regular linked-worktree lock for 250 ms, and proves the exact reset waits and succeeds. A disposable published-image probe exercises the same explicit-parent/delegated-child shape, waits 314 ms, restores the image Git executable digest, and leaves zero interposition files. The full Modal suite passes 553/553, with typecheck, build, formatting, and diff checks also passing.

Additional generic reproduction for the same transient linked-worktree collision:

The retry-only Git shim is now reached, but its current 100 x 50 ms bound can expire while a valid regular index.lock is still owned by a live Git process on slow durable storage. In a temporary repository with one detached linked worktree, hold only that worktree's regular index.lock for 6 seconds, then run the selected git read-tree --reset -u <40-hex-tree> primitive with the release's current 101-attempt/50-ms policy. All 101 attempts fail with status 128 after 5,774 ms while the regular lock is still present; the identical command succeeds immediately after the owner releases it. No symlink, special file, stale lock, target project, or external workflow data is involved.

The next local compatibility adjustment will keep the existing exact-command and exact-worktree safety checks and change only the bounded wait window. Unrelated Git commands and ordinary run/resume behavior remain unchanged.

Follow-up from the release-owned retained-retry path:

Executing worker SHA-256 was 58c001e3fac39ca870a22f9744ccfbbbeca115a54889b51a06522dddee5221e0, matching the focused/full-suite-tested build with the 60-second exact-command bound.
The release-generated workflow's static execFileSync call still failed at the exact selected git read-tree --reset -u <40-hex-tree> primitive.
RunStarted to RunFailed was about 299 ms. Therefore neither the 1,200 × 50 ms bound nor even a single bounded retry loop can explain this failure.
The emitted Git stderr is 522 bytes and byte-for-byte equal to the shim's accepted collision string, including the selected linked-worktree lock path and standard Git advice.
The isolated production-image regression still passes: explicit Bun preload, delegated Bun child, 7.5-second regular lock wait, canonical Git restoration, and zero leftovers.

Expected: the retained renderer must traverse the already exact-task-gated preload or canonical executable interposition before this command can reach image Git.

Actual: the real release-generated retained renderer bypasses both validated interception layers and reaches the native imported execFileSync binding directly.

This remains the same bug as #727 on a generic external Solidity target. No model invocation is involved.

Additional generic release-owned evidence from the retained retry renderer:

The retry preload loaded exactly once, but its selected-call count was zero.
The installed canonical Git wrapper recorded zero selected calls, collisions, successes, or exhaustions.
The retained renderer therefore held a child-process path below both JavaScript binding replacement and executable lookup replacement, and reached image Git directly.
No model request occurred; this failure is wholly inside the release retry path.

Minimal compatibility fix: enable a private, retry-scoped native spawn interposer only for the already exact-task-gated retry. It redirects only git read-tree --reset -u <40-lowercase-hex-tree> (including an absolute Git path) to the existing bounded wrapper. All other commands pass through unchanged. The interposer and trace live in the existing private temporary wrapper directory and are removed by the existing cleanup.

Focused reproduction now passes with a renderer that captured execFileSync before the preload and invokes absolute Git while a live regular lock is held. The full Modal package suite passes 553/553, along with typecheck, build, formatting, and whitespace checks.

This remains the same generic external-target bug as #727; no target identity, campaign data, model behavior, or contract finding is involved.

Additional generic production evidence after enabling the retry-scoped native spawn interposer:

The retry preload loaded once.
Its selected-call count was zero.
The native interposer selected-call count was zero.
The canonical executable wrapper recorded zero selected calls, collisions, successes, and exhaustions.
The retry failed at the same selected linked-worktree restore before any agent/provider request.

This proves Bun 1.3.0's retained renderer bypasses all three attempted boundaries: JavaScript binding replacement, canonical executable replacement, and libc LD_PRELOAD spawn interposition.

A minimal target-independent Bun probe identifies a narrower supported boundary: from the already retry-scoped preload, bun:test mock.module("node:child_process", ...) can replace the builtin module before the release-generated workflow's future static ESM import. That import receives the replacement while the rest of the child-process exports remain available.

This is further evidence for the same exhausted-task transient linked-worktree lock bug, not a new issue category. No target identity, source, prompt, model behavior, or finding is involved.

Additional same-bug evidence from the fully isolated default-topology retry path on the affected release:

The retry-scoped ESM module mock loaded exactly once.
Its selected-call count was zero, as were the native-interposer and executable-wrapper selected-call counts.
The retry command failed while dynamically importing the generated workflow, before it could take ownership of or reset the exhausted task.
The import executes the generated git read-tree --reset -u <40-hex-tree> restoration and encounters the selected linked worktree's exact regular index.lock collision.
The lock is gone after the transient owner exits, and no agent/provider request begins.

The successful standalone ESM mock probe and the zero real-renderer selection counts show that the bundled CLI imports the workflow in another isolated/bundled realm. Inner process interception is therefore not a reliable compatibility boundary for this retained renderer.

A minimal release-owned recovery boundary is the already exact-task-gated worker invocation: when retry-task fails before ownership with the exact selected linked-worktree lock signature, wait for that already canonically validated lock path to disappear and retry the whole command a small bounded number of times. Unrelated errors and ordinary run/resume must remain terminal.

This is further evidence for #727 on a generic external Solidity target. No target identity, source, path, prompt, business logic, model behavior, or finding is included.

Follow-up on the worker-owned whole-command compatibility boundary:

The first production attempt reached the new boundary and recorded one command attempt with zero lock waits.
The retained inner event stream appended no RunStarted or NodeStarted, so the command definitely failed before taking ownership of the exhausted task.
The classifier nevertheless failed closed because the CLI's rendered stderr omitted the assumed loadWorkflow function-name frame.
No agent/provider request began, and the durable retry state remained unchanged apart from the outer failed-attempt checkpoint.

The minimal follow-up removes only that brittle stack-text assumption. Around each already exact-task-gated retry-task invocation, the worker fingerprints the existing regular, canonical inner event stream by device, inode, byte length, and SHA-256. A whole-command retry is allowed only when the exact selected linked-worktree git read-tree --reset -u <40-hex-tree> lock signature matches and the stream remains byte-identical. Any appended ownership event, stream replacement, unsafe stream or lock, unrelated error, or exhausted bound remains terminal.

Focused generic tests cover all three cases: the pre-ownership collision waits and succeeds on command attempt two; unrelated stderr is invoked once; and the exact same lock signature is not retried after a RunStarted append.

This remains the same release-owned bug as #727 for a generic external Solidity target. No target identity, source, path, prompt, business logic, model behavior, or finding is included.

Follow-up for the same v0.1.0 Modal retry boundary on a generic external Solidity target:

The byte-stable ownership proof currently receives the outer UltraFuzz run root, then tries to fingerprint <outer-run-root>/smithers/logs/stream.ndjson. The retained runner stream is instead keyed by the already-known inner run ID at .smithers/executions/<inner-run-id>/logs/stream.ndjson.

Minimal generic reproduction:

Retain a failed inner execution and its canonical regular event stream.
Enter the exact exhausted-task retry path.
Fingerprint the stream through the outer run root before starting retry-task.

The worker raises ENOENT before the retry command is invoked. No ownership event can be evaluated, so the compatibility path remains blocked.

The narrow correction is to pass the existing inner run ID to the fingerprint helper and resolve the canonical .smithers/executions/<inner-run-id>/logs/stream.ndjson path. The existing regular-file, non-symlink, link-count, realpath, device, inode, size, SHA-256, exact-command-signature, and retry-bound checks remain unchanged. A regression should place the fixture stream at that canonical inner-execution path and append a RunStarted record there for the post-ownership negative case.

Follow-up for the same v0.1.0 Modal retry boundary on a generic external Solidity target:

The canonical inner-stream correction is now exercised in production. The retry command reached the worker-owned boundary and the before/after stream fingerprint was byte-identical, proving no inner ownership event was appended. The aggregate exact-collision classifier still returned false, so the worker correctly failed closed after one invocation.

The current diagnostic exposes only the aggregate signature result. It cannot distinguish which existing predicate rejected the command: the exact git read-tree --reset -u <40-hex-tree> form, the canonically derived linked-worktree lock path, or the legacy absence of ownership-token text in command output.

The narrow next step is diagnostic-only: retain the aggregate gate and separately count those three non-sensitive predicates in the failed checkpoint. No stderr, source path, repository data, prompt, or model output needs to be persisted. A focused regression can prove that the aggregate result and every component count are recorded without changing retry behavior.

Additional generic evidence shows why fixing this at the generated helper is necessary.

With a temporary repository, one linked worktree, and a fake retained runner, each normal resume can enter restoreWorkspacePatchPreparation, invoke the same one-shot git read-tree --reset -u <tree>, append a complete exact selected-lock RunStarted through RunFailed sequence, and exit before NodeStarted. The outer recovery then safely removes the proven orphan lock, but the next normal resume re-enters the same one-shot helper while a new current transaction owns the lock.

A four-command bound produced four exact signatures, three post-reset resumes, two validated post-reset terminal collisions, four RunStarted, four RunFailed, and zero NodeStarted. The retained stream stayed prefix-preserving. Raising that outer cap only increases the number of identical terminal sequences before exhaustion.

The narrow durable fix is therefore the one already described in this issue: retry the exact read-tree primitive inside the generated helper while the validated regular selected-worktree lock remains live, with a fixed delay/bound and fail-closed handling for unrelated errors, unsafe locks, changed path identity, or exhaustion.

Add bounded wait and stale-lock recovery to restoreWorkspacePatchPrep…

Reconcile #727 with the merged #691 scan: reword the prose that quote…

Navigation Menu

Repository navigation

Generated retry restoration fails on a transient linked-worktree index lock #727

Summary

Minimal generic reproduction

UltraFuzz-owned failure path

Expected behavior

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Generated retry restoration fails on a transient linked-worktree index lock #727

Description

Summary

Minimal generic reproduction

UltraFuzz-owned failure path

Expected behavior

Activity

aviggiano commented on Aug 21, 2026

aviggiano commented on Aug 21, 2026

aviggiano commented on Aug 21, 2026

aviggiano commented on Aug 21, 2026

aviggiano commented on Aug 22, 2026

aviggiano commented on Aug 22, 2026

aviggiano commented on Aug 22, 2026

aviggiano commented on Aug 22, 2026

aviggiano commented on Aug 22, 2026

aviggiano commented on Aug 22, 2026

aviggiano commented on Aug 22, 2026

aviggiano commented on Aug 22, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions