[aviggiano]

Stacked on #1211 and must merge after it. #1211 has since merged into main as a squash (c47c892), so this branch merges origin/main (7728e7b) on top of #1211's commits; its diff against main is this change alone, and a trial merge into the current main (13af837) is clean.

Problem

• Two ultrafuzz commands that append to one journal at the same time can lose a record, or tear the journal so that the run is stranded (pre-existing; measured on fix(artifacts): event appends and materialize keep working after the host clock steps back #1211, which listed it under "Deliberately not fixed"). Every strict JSONL journal appends in three steps with no lock: read the journal, validate the new records against it, then check the file size on the write descriptor and write at that offset.
  • The entry points take different locks or none. Sync takes .workflow-sync, launch takes the control lock, and resume, replay and fork take the lifecycle lock. cancelRun and materializeSelection take none. So ultrafuzz status --watch syncing while an operator runs ultrafuzz cancel appends to one events.jsonl from two processes.
  • Both writers can pass the size check and write at the same offset. When the second record is shorter, the rest of the first is left behind, and the next append and every replay of that journal fail with event journal record N is invalid strict JSON or contains a blank record. When it is not shorter, one record is gone although its writer reported success.
  • The same helper backs usage.jsonl, attempts.jsonl, and the .ultrafuzz/ materialize, clean and dashboard audit journals.
• In the fix(artifacts): event appends and materialize keep working after the host clock steps back #1211 harness (two processes, 300 materialize-selection events each, 1.8 KB payloads), the journal tore in 11 of 21 rounds without the lock. The other rounds refused 23 to 81 appends and lost up to 4 records each (Verification).

Root cause

appendStrictJsonlRecords and appendStrictJsonlRecordAfterTail (packages/artifacts/src/strict-jsonl.ts) read, validate and call appendBytesDurableAt without mutual exclusion between processes. The size fence in appendBytesDurableAt is an fstat followed by a positional write, so two writers can both pass it. The validation (timestamp order, duplicate IDs) also holds only against the tail that was read, so even an atomic append could not stop a concurrent record from breaking the journal's order.

Change

• A per-journal lock held for one append (packages/artifacts/src/journal-lock.ts, new, not exported from the package index). withJournalLock(journalPath, append) holds <journal>.lock while append runs.
  • Both append paths in strict-jsonl.ts run their existing bodies, moved unchanged into …Locked functions, inside it: from the read, through validation and fix(artifacts): event appends and materialize keep working after the host clock steps back #1211's build, to the fenced write. The size fence stays as a second check.
  • The lock is an O_CREAT | O_EXCL | O_NOFOLLOW file, mode 0600, holding {"host", "pid", "started"}. started is the process start time from /proc/<pid>/stat on Linux, so a later process given the same PID is not taken for the holder. The record is built before the file is created, so a holder killed right after creating it has the least time to leave it empty.
  • A waiter polls every 5 ms with Atomics.wait, the idiom eval-durable.ts and workspace-handoff.ts already use, because every caller is synchronous. After 30 s by the monotonic clock it throws ArtifactPathError with code journal-locked, naming the lock and its holder, and the append does not run. A waiter that has just removed an abandoned lock, or seen it go, tries again at once and checks its deadline only before sleeping, so it never gives up on a lock that is free.
• A crashed holder cannot wedge a journal.
  • A lock recorded on this host is judged by its process, and a live holder there is never broken. The next waiter breaks it at once when its PID is gone, when that process is a zombie its parent has not reaped yet, or when the process now using the PID started at another time. That last check also covers another user's process, for which kill(pid, 0) fails with EPERM while /proc/<pid>/stat stays readable.
  • Any other lock is broken once its mtime is 10 s old (JOURNAL_LOCK_STALE_MS). That covers an empty or unparseable lock, which a kill between the create and the record's write can leave, and a lock recorded by another host, whose PID cannot be checked here, such as one an earlier sandbox or container left on the same volume. A live append holds the lock for milliseconds. 10 s is well below the 30 s wait, so an append that was already waiting when such a holder died takes over within its own wait.
  • A power loss can leave either kind: the lock's creation reaches the disk with the journal's directory fsync, but its record and its later removal may not. After the reboot, a lock with this host's record is broken at once, because its process is gone.
• Breaking is serialized. A waiter breaks a lock while holding <lock>.break, taken with the same primitive, and removes the lock only if it is still abandoned when re-read. Without that, two waiters that both judged the lock abandoned could both remove it, and the second would remove the lock the first had just created. A stress run shows it: with the .break step removed, 5 of 10 rounds had two holders at once.
• No deadlock. The lock is taken inside the append, and nothing else is acquired while it is held. It therefore nests inside the sync, control and lifecycle locks and never the other way round. Release unlinks the lock only if it is still this process's inode, and never throws.
• Where the lock lives. Next to its journal: events.jsonl.lock, usage.jsonl.lock and attempts.jsonl.lock in the run root, and materialize-audit.jsonl.lock, clean-audit.jsonl.lock and dashboard-audit.jsonl.lock in .ultrafuzz/. It exists for the milliseconds of one append. I checked every directory walk that could reach those directories:
  • the report bundle copies a fixed list of top-level run files plus artifacts/, review/, events.index/ and smithers/logs;
  • run listings, the dashboard, clean, private eval evidence and the Modal runner enumerate runs/ entries, and listSafeFiles callers walk node artifact directories;
  • Modal staging, the deterministic archive and the sealed execution-snapshot inventories walk staging and snapshot directories, never the run root or .ultrafuzz/.
  • Readers never take the lock.
• Docs. docs/reference/artifacts-reports.md describes the lock after the event journal paragraph.

Size: journal-lock.ts +228, strict-jsonl.ts +33/−5, one test file +287, docs +15.

Deliberately not built

• Locking readers. replayEvents, status, report bundles and stats still read without the lock. A read that races a write already fails with file changed while it was read rather than returning a torn snapshot. A reader lock would also stop read-only run directories from being readable.
• Deduplicating under the lock. appendUsageEvents and appendNodeAttempts pick the new entries from a read outside the lock. Both run only inside a sync pass, which holds .workflow-sync. A racing duplicate would still be rejected by the locked validation (contains a duplicate identity), not written.
• Reusing an existing lock.
  • proper-lockfile (runtime's sync, control and lifecycle locks) throws ESYNC when its synchronous lockSync is given retries. Its staleness is an mtime age for every holder, so it depends on the wall clock, the clock that fix(artifacts): event appends and materialize keep working after the host clock steps back #1211 had to survive stepping back, and it would take over a live holder on this host stalled in a slow fsync. It is also not a dependency of @ultrafuzz/artifacts.
  • The closest precedent is withModalLaunchStateLock in packages/modal/src/launch-state.ts, also an O_EXCL file recording the PID and /proc start ticks. It is async, and @ultrafuzz/modal depends on artifacts, not the other way round. processStat here is therefore a second copy of its parseProcessStartTicks.
  • The mkdir lock in eval-durable.ts belongs to @ultrafuzz/evals, which depends on artifacts; it records no holder and never breaks a stale lock. A possible follow-up is for acquireEvalJournalLock to call withJournalLock.
• Closing the empty-lock window. A holder killed between the O_EXCL create and its record's write leaves an empty lock, and create-then-write cannot close that window, because the kill can land inside open() itself. Creating the lock with its record already in place (writing a temporary file, then link) would leave temporary files behind after a crash and needs hard links, which some network filesystems lack. What the window costs is a wait of up to 10 s for the next append, not a failure: in the SIGKILL stress below, 2 of 6 rounds stalled on an empty lock.
• Changing fix(artifacts): event appends and materialize keep working after the host clock steps back #1211's build order. appendEvent still builds its record before the read. Under contention the final event is often newer than the prebuilt one, so the record is rebuilt under the lock, which takes about 4 ms for a 1.8 KB payload. Without contention nothing changes.

Verification

New tests are in packages/artifacts/test/journal-lock.test.ts. Two bases were built from this branch's test file:

• Unlocked: strict-jsonl.ts from origin/claude/g3-artifacts-greptile-fixes (origin/main carries the same file), with this branch's journal-lock.ts in place so that the tests compile. Tests 4 to 7 exercise the lock alone, so they pass there.
• First revision: journal-lock.ts from this branch's first commit (a5d0134), before the takeover rules were widened, with an exported JOURNAL_LOCK_STALE_MS added for the test's import.

Test	Unlocked	First revision	Here
1. appends from concurrent processes all land in the event journal, which still replays: 3 processes × 50 appends	fails 16 of 16 runs: 13 with AssertionError: events reported as appended are missing, listing event IDs that appendEvent returned but the journal does not contain, and 3 torn, with event journal contains a blank record at line 24 (twice) or line 20	passes	passes in all 25 runs: 10 in a row, 12 with six copies of the file running at once, and 3 with real fsync instead of eatmydata
2. an append waits while another process holds the journal's lock, then appends after its record	fails 16 of 16: the append does not wait, so the journal reads [waited, holder] instead of [holder, waited]	passes	passes
3. a journal lock left by a process killed while holding it is taken over: appendStrictJsonlRecords after a SIGKILLed holder	fails 16 of 16 with true !== false: the append ignores the lock, which is still there afterwards	passes	passes: the append runs at once and removes the lock
4. a journal lock whose holder was killed but is not reaped yet is taken over: the test waits without turning its event loop, so its SIGKILLed child stays a zombie	passes	fails 10 of 10 with journal-locked, held by the zombie	passes
5. a journal lock whose process ID was reused, even by another user, is taken over: PID 1 (EPERM unless root) and the test's own PID, each recorded with a start time one tick early	passes	fails 10 of 10 with journal-locked, held by process 1	passes
6. an append that cannot take the journal's lock within its wait fails without running	passes	passes	passes: ArtifactPathError code journal-locked, and the append never ran
7. an empty journal lock, or one recorded on another host, is taken over once it is stale: each is held while new, then taken over when 11 s old by an append with waitMs: 0	passes	fails 10 of 10: the 11 s old empty lock is still held. With only the old acquire loop restored, it fails 3 of 3: the append removes the lock, then gives up.	passes

The file takes about 3 s (3.0–3.1 s with eatmydata, 3.3–3.9 s without). Tests 4 and 5 skip where /proc/self/stat does not exist.

The review's probes against this build:

• a lock recorded by another host an hour ago is taken over in 1 ms, and a fresh one after 10 s;
• a lock naming PID 1 with the wrong start time is taken over in 1 ms;
• a zombie holder's lock is taken over in 1 ms;
• an append that was already waiting when an empty lock appeared appends after 10 s; before, it failed after 30 s.
• Under Bun, which runs parts of the runtime suite, a SIGKILLed holder's lock, the PID 1 lock and both of the other host's locks are taken over the same way.

Harness: the #1211 fixer's contend.mjs, unchanged. Each round, two processes each append 300 materialize-selection events with 10 copies (1.8 KB) to one journal. "Refused" counts appends that threw.

Build	Rounds	Torn rounds	Other rounds
Unlocked: #1211's branch (84fe94c) and origin/main 7728e7b, whose packages/artifacts/src is identical, 7 rounds each in the first run; the unlocked base above, 7 more in this run	21	11: contains a blank record at line 222, 31 and 201, and records 82, 179, 69, 87, 427, 388, 171 and 76 invalid strict JSON. In this run's torn rounds, every later append failed: 378 to 565 of them per round.	refused 23 to 81 appends; lost 0 to 4 records
This branch	7	0	every round: 600 of 600 appended, 0 refused, 601 replayed, 0 lost

A round takes 5.0–5.9 s here against 3.1–4.0 s for the unlocked rounds of this run that did not tear, under the same background load, because the appends now run one at a time.

Stress, with scratch scripts:

• Breaking race: a holder is SIGKILLed inside the lock, then 8 processes start together and append 20 events each. In 10 of 10 rounds, 160 of 160 were appended, none was lost, and no .lock or .break file was left. With the .break step removed from the build, 5 of 10 rounds had two holders at once: 4 refused 1 or 2 appends with append path changed size before repair, and 1 lost a record that its writer had reported as appended.
• SIGKILL stress (the review's script): 8 contenders take the lock in a loop, a PID-stamped marker file marks the critical section, and one contender is SIGKILLed every 5–45 ms. 6 rounds of 8 s, 1,725 kills in all: 0 double holders, 0 failed acquisitions.

Suites on this branch:

• artifacts: the full suite, 332/332.
• runtime:
  • materialize, clean, audit-contracts, data-governance, source-revision, verified-output and trusted-cli: 97/97. trusted-cli.test.js's closures copy the artifacts dist/.
  • lifecycle-inspection, the whole file: 55/55.
  • The two dynamic-lifecycle tests that read events.jsonl: 2/2.
  • The 145 runtime.test.js tests whose names match synchronization, syncRun, resume, cancel, replay, fork, materialize, pause, lifecycle or event journal: 136 pass and none fail. The other 9 are Bun adapter contract tests, which skip under node; the package script runs them under bun test.
• dashboard: contracts.test.js 5/5, and the two audit tests in dashboard.test.js, including dashboard audit append refuses a final-component symlink: 2/2.
• cli: the 57 report bundle, report render and stats tests in cli.test.js, whose runs append events and ledgers and are then bundled: 57/57.
• The runtime, dashboard and cli runs used an artifacts dist/ that differed from the final build only in still exporting JOURNAL_LOCK_WAIT_MS. The artifacts suite and the checks below ran on the final commit.
• Checks, all passing:
  • npx prettier --check and npx eslint on the changed files.
  • CI=1 ESLINT_PLUGIN_DIFF_COMMIT=origin/claude/g3-artifacts-greptile-fixes pnpm -w lint:strict:ci, and the same with the merge base with origin/main.
  • pnpm -w lint.
  • pnpm --filter @ultrafuzz/artifacts typecheck.
  • pnpm -w knip, and also with ci: fail on unused exports, types and duplicates #1219's exports,types,duplicates added, which reports nothing in these files.
  • node scripts/docs-check.mjs.

Risk / compatibility

• Journal format. Unchanged. The lock is a separate file, and older builds read journals written here.
• Mixed versions. A process built before this change does not take the lock, so it can still race a new one until it exits. The size fence still refuses most of those races.
• Directory permissions. An append now needs to create a file in the journal's directory. Before, an append to an existing journal needed only the file to be writable. Run roots and .ultrafuzz/ are created writable. In the read-only run root of a synchronization that cannot take its lock is reported instead of thrown, a pass stops at the sync lock before it appends anything.
• Waiting. A waiting append blocks its process's event loop, because every caller is synchronous. The wait is normally one append, a few milliseconds, and never more than 30 s. An empty lock left by a kill costs the next append up to 10 s, and a backward clock step while it exists lengthens that by the size of the step.
• Another host. A lock recorded by another host is aged by this host's clock against the lock's mtime. A holder there that stalls inside one append for more than 10 s can have its live lock broken, and so can one whose clock, or whose file server's, runs more than 10 s behind this host's; the size fence then refuses most of what follows, as before this change. A clock that runs ahead delays the takeover by the difference.
• Containers. A holder counts as on this host by its hostname alone, not its PID namespace or boot. Processes that report the same hostname but run in different PID namespaces, and share a run directory, can therefore break each other's live locks, because a PID from the other namespace looks gone or reused here; their appends are then left to the size fence, as before this change. The reverse, a dead holder's PID and start time matching a live process here, needs a sandbox with its own kernel and a fixed hostname, and keeps the lock held until that process exits.
• Overlap. No open PR touches strict-jsonl.ts or journal-lock.ts. fix(runtime): a renderer or projection change no longer strands runs whose dynamic groups expanded #1216, feat(topology)!: a failed property lens no longer skips the rest of the campaign #1198, refactor!: remove per-node cloud execution (execution.mode = "cloud") #1197 and fix(runtime): record final-report producer selections in the run instead of querying smithers #1183 edit docs/reference/artifacts-reports.md elsewhere and merge with this branch cleanly in that file. A trial git merge-tree with each open PR conflicts in none of this branch's files; the conflicts with feat(runtime)!: run lifecycle commands from the pnpm-patched install instead of per-command npm installs (#921 step 1) #1201, refactor!: remove per-node cloud execution (execution.mode = "cloud") #1197, fix(runtime): record final-report producer selections in the run instead of querying smithers #1183 and feat(runtime): let agents record roadblocks in a run-local friction log #1162 are in other files and appear against origin/main alone. ci: fail on unused exports, types and duplicates #1219 turns on knip's unused-export check, which this branch passes.

Changelog entry

Concurrent ultrafuzz commands, such as ultrafuzz status --watch while ultrafuzz cancel runs, no longer lose or tear records in a run's events.jsonl, usage.jsonl or attempts.jsonl, or in the .ultrafuzz/ audit journals. Each append now holds a short-lived <journal>.lock next to the journal. A later append takes over at once a lock left by a killed process on the same host, and after 10 seconds one it cannot check, such as a lock an earlier sandbox left on the same volume.

Greptile follow-up

Both comments describe limits this PR already lists under Risk, and neither makes anything worse than before it.

• Same hostname, different PID namespaces (comment). Two containers that share a run directory and report one hostname can misjudge each other's live holder and remove its lock. Their appends then rely on the size fence, as every append did before this PR. Concurrent appenders in separate PID namespaces sharing one run directory is not a supported deployment. Modal's resume starts a new container only after the old one is gone.
• Clock skew or a stalled append on another host (comment). The 10-second age rule applies only to a lock that cannot be checked: one recorded by another host, or one with no parseable record. A lock recorded on this host is judged by its holder's PID and start time, so a long append here is never broken. The rule exists to recover the lock a dead container leaves on a shared volume. Concurrent appends from two hosts with more than 10 s of skew are outside what the lock can arbitrate, and again fall back to the size fence.

🤖 Generated with Claude Code

[greptile-apps]

Live container lock removed

If two containers share a journal directory and report the same hostname but use different PID namespaces, this check can mistake the other container’s live holder for a missing or reused process. It then removes that holder’s lock. Both commands can read the same journal length and write at the same offset, losing a record or tearing the journal.

Knowledge Base Used: Artifact management

Prompt To Fix With AI

This is a comment left during a code review.
Path: packages/artifacts/src/journal-lock.ts
Line: 137

Comment:
**Live container lock removed**

If two containers share a journal directory and report the same hostname but use different PID namespaces, this check can mistake the other container’s live holder for a missing or reused process. It then removes that holder’s lock. Both commands can read the same journal length and write at the same offset, losing a record or tearing the journal.

**Knowledge Base Used:** [Artifact management](https://app.greptile.com/monad-foudnation/-/custom-context/knowledge-base/monad-developers/ultrafuzz/-/docs/artifact-management.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Clock skew breaks live locks

If hosts sharing a journal have clocks more than 10 seconds apart, this check can mark another host’s live lock as abandoned during an ordinary append. A live append that lasts past the threshold has the same problem. Removing that lock lets two writers read the same journal length and write at the same offset, so records can be lost or the journal torn.

Knowledge Base Used: Artifact management

Prompt To Fix With AI

This is a comment left during a code review.
Path: packages/artifacts/src/journal-lock.ts
Line: 140

Comment:
**Clock skew breaks live locks**

If hosts sharing a journal have clocks more than 10 seconds apart, this check can mark another host’s live lock as abandoned during an ordinary append. A live append that lasts past the threshold has the same problem. Removing that lock lets two writers read the same journal length and write at the same offset, so records can be lost or the journal torn.

**Knowledge Base Used:** [Artifact management](https://app.greptile.com/monad-foudnation/-/custom-context/knowledge-base/monad-developers/ultrafuzz/-/docs/artifact-management.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.