Skip to content

fix(engine): a failed parallel capture names the worker that failed, not the peers it stopped - #7803

Merged
miguel-heygen merged 1 commit into
heygen-com:mainfrom
wonjangcloud9:fix/parallel-failure-names-cause
Nov 15, 2026
Merged

miguel-heygen merged 1 commit into
heygen-com:mainfrom
wonjangcloud9:fix/parallel-failure-names-cause

Conversation

@wonjangcloud9

Copy link
Copy Markdown
Contributor

What

When one worker of a parallel capture fails, the render's failure message now carries that worker's error and a count of the peers it stopped, instead of listing every cancelled peer around it.

Why

A 1800-frame render of mine failed on a nearly full disk (0.9.43, 6 workers). The reason was in the message, but as the fourth of six entries:

✗  Render failed
   [Parallel] Capture failed: Worker 0: Parallel worker cancelled; Worker 1: Parallel worker cancelled; Worker 2: Parallel worker cancelled; Worker 3: Disk capture may need ~1520.2 MB of temporary frame storage (measured from the first 10 frames), but only 154.4 MB is free at … Free up disk space, or re-run this render with --workers 1 … ; Worker 4: Parallel worker cancelled; Worker 5: Parallel worker cancelled

The first fatal failure aborts the pool (createPoolFailureHandler), each aborted peer is classified cancelled, and executeParallelCapture joins every failed worker's message in id order. So the cancellations are a consequence of the one real failure, and they come first whenever the failing worker is not worker 0. The same line is repeated in the progress row and in the Failure summary log.

Related work

  • fix(cli): a low-disk render refusal names the extracted-frame cache and clean #7695 (open, mine) makes the pre-render low-disk refusal name the extract cache. This is the other disk message, and the change here is not specific to disk: it applies to any failure that aborts the pool.
  • No open or closed issue or PR covers this. I searched "Parallel worker cancelled", "worker cancelled failure message" and "Capture failed cancelled workers".

How

packages/engine/src/services/parallelCoordinator.ts: a new formatPoolFailure(errors) builds the message.

  • If some failures are not cancelled and some are, it prints only the former and appends (N other workers were stopped after this failure).
  • If every worker was cancelled (an outside abort) or none was, the message is the same list as before.

Only the message text changes. The failure's kind, cause, workerDiagnostics and pageErrorLines are built as before, and the message still starts with [Parallel] Capture failed: Worker N: …, which isRecoverableParallelCaptureError and the producer tests match on.

Test plan

Unit (packages/engine):

  • parallelCoordinator.test.ts: a new case for formatPoolFailure (three stopped peers, one stopped peer, no peers, all cancelled).
  • parallelCoordinator-peerAbort.test.ts: the existing two-worker peer-abort case now also checks the rejected message. On main (04ee926d) with the fix stashed it fails with expected '[Parallel] Capture failed: Worker 0: …' to contain '(1 other worker was stopped after thi…'; with the fix both files pass (64 tests).
  • vitest run src/services/ in packages/engine: 2292 passed. The 2 failures are beginFrameReuse.browser.test.ts (HeadlessExperimental.beginFrame: Target closed), which fail the same way on main on this macOS machine.
  • packages/producer: renderOrchestrator.test.ts, captureStageError.test.ts, renderEventPublisher.test.ts (vitest, 321 passed) and captureStreamingStage.test.ts (bun test, 48 passed).
  • tsc --noEmit, oxlint, oxfmt, fallow audit --base origin/main and the comment ratchet are clean.

Real CLI: a 3600-frame 1080p project (the blank template over a noisy full-frame background that pans for 120 s), rendered with TMPDIR on a 1.3 GB disk image so the capture runs out of room: hyperframes render --workers 4 -q draft.

Before (main 04ee926d):

✗  Render failed
   [Parallel] Capture failed: Worker 0: Parallel worker cancelled; Worker 1: Parallel worker cancelled; Worker 2: Disk capture may need ~2712.6 MB of temporary frame storage (measured from the first 10 frames), but only 1174.7 MB is free at /Volumes/hfsmall/tmp/hf-render-VBzUWr/captured-frames. Disk capture stores every frame as an image file (JPEG, or PNG for alpha). This render stores frames because it captures with 4 workers. Free up disk space, or re-run this render with --workers 1 to stream frames into the encoder instead. One worker is slower, so keep the default worker count for renders that fit on disk.; Worker 3: Parallel worker cancelled

After:

✗  Render failed
   [Parallel] Capture failed: Worker 1: Disk capture may need ~1412.1 MB of temporary frame storage (measured from the first 10 frames), but only 1172.8 MB is free at /Volumes/hfsmall/tmp2/hf-render-daeG5H/captured-frames. Disk capture stores every frame as an image file (JPEG, or PNG for alpha). This render stores frames because it captures with 4 workers. Free up disk space, or re-run this render with --workers 1 to stream frames into the encoder instead. One worker is slower, so keep the default worker count for renders that fit on disk. (3 other workers were stopped after this failure)

(The estimate differs between the two runs because it is measured from whichever worker's first 10 frames land first.)

🤖 Generated with Claude Code

…not the peers it stopped

When one capture worker fails, the pool aborts its peers and each of them
reports "Parallel worker cancelled". The pool message listed every worker in
id order, so the one real cause sat between cancellations:

  Worker 0: Parallel worker cancelled; Worker 1: Parallel worker cancelled;
  Worker 2: Disk capture may need ~2712.6 MB ...; Worker 3: Parallel worker cancelled

The message now carries the failures that are not cancellations and counts
the peers that were stopped because of them. A pool where every worker was
cancelled keeps the full list.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

@miguel-heygen miguel-heygen left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified at d30ece6: formatPoolFailure keeps the real cause(s) and counts the cancelled peers; all-cancelled (outside abort) and no-cancelled keep the old list. The message still starts with [Parallel] Capture failed: Worker N:, kind/cause/workerDiagnostics unchanged, and nothing outside the coordinator matches on "Parallel worker cancelled". Tests pin all four shapes plus the peer-abort path. CI green.

@miguel-heygen
miguel-heygen added this pull request to the merge queue Nov 15, 2026
Merged via the queue into heygen-com:main with commit db6b747 Nov 15, 2026
91 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants