execution/stagedsync: parallel exec loops forever with zero progress when commitment files lag state domains (step-misalignment error silently swallowed)

[yperbasis]

Summary

On v3.5.0 with parallel execution, a node can get permanently stuck repeating the Execution stage with zero progress and no error logged — execution "runs" every cycle, executes 0 blocks in ~300µs, and loops forever.

Root cause is two stacked bugs, both originating in #20805 ("parallel commitment calculations implemented", b72aa7b4f7):

1. Trigger: the commitment domain .kv files lag the account/storage/code domain .kv files by one step (state at step 2314, commitment at 2313). This is the same root cause as execution/commitment: parallel executor skips step-boundary commitment checkpoint → snapshot domain/commitment inconsistency (blocks v3.6 release) #21992 / execution/stagedsync: checkpoint commitment at step boundaries in parallel exec #22092 (parallel exec skips the step-boundary commitment checkpoint), but a distinct, runtime symptom — execution/commitment: parallel executor skips step-boundary commitment checkpoint → snapshot domain/commitment inconsistency (blocks v3.6 release) #21992 assumes "block execution is unaffected". executeBlocks has a guard that returns snapshot step misalignment ... files need rebuilding on every cycle.
2. Mask: that error is silently swallowed — it is errors.Join'd with the apply loop's ErrLoopExhausted, and the Warn is gated on !Is(ErrLoopExhausted), so nothing is logged and the stage "succeeds" with unchanged progress. The sync loop reads ErrLoopExhausted as "more work pending" and reruns forever.

The code is identical on main, so this affects main as well.

User report (v3.5.0)

The following four lines repeat without end:

[INFO] [06-29|14:14:58.105] [4/6 Execution] parallel done            in=317.884µs buf=0B/512.0MB blk=25423801 blks=0 blk/s=0 txs=0 tx/s=0 gas/s=0 stepsInDB=1.00 step=2314.0 alloc=9.2GB sys=10.2GB isForkValidation=false isApplyingBlocks=true
[INFO] [06-29|14:14:58.115] BuildFilesInBackground                   step=2313 lastInDB=2314 targetStep=2314 reorgSafeBlock=25423752 reorgSafeStep=2313.99 reorgSafeOK=true
[INFO] [06-29|14:14:58.117] [4/6 Execution] parallel starting        from=25423802 to=25423848 limit=25428801 initialTxNum=3615626461 initialBlockTxOffset=0 initialCycle=true isForkValidation=false isApplyingBlocks=true
[INFO] [06-29|14:14:58.117] [4/6 Execution] parallel executed        blk=0 blks=0 blk/s=0 txs=0 tx/s=0 gas/s=0 exec=0 repeat%=0.00 abort=0 invalid=0 tgas/s=0 tcpus=0.0 tdur=0s exec=0s(0.00%) read="0s(0.00%),a=0s,s=0s,c=0s" rd="0,a=0,s=0,c=0" wrt=0 rd/s=0 wrt/s=0 buf=0B/512.0MB bdur=0s alloc=9.2GB sys=10.2GB isForkValidation=false isApplyingBlocks=true

No WARN/ERROR is ever emitted.

Analysis (initial)

Inferred from the logs and source; not yet confirmed against the reporter's datadir. The "step misalignment" guard is the most likely trigger given the step-boundary evidence below; a missing/nil block or a BAL-decode failure would be hidden by the exact same swallow and are the alternative pre-execution errors to rule out.

Decoded symptom

• parallel executed blk=0 blks=0 → lastExecutedBlockNum == 0: no block ever produced a blockResult.
• done in=317.884µs → the whole parallel-exec call returns in ~300µs: far too fast to read or execute anything → it errors out before dispatching the first block.
• done blk=25423801 → lastCommittedBlockNum is stuck at startBlockNum - 1.
• parallel starting from=25423802 to=25423848 → there is real work to do (46 blocks), yet 0 are done.

Layer 1 — trigger: commitment files lag the state domains

executeBlocks opens with a hard guard (execution/stagedsync/exec3.go:547-555, added by #20805 itself):

cmtStep  := execRoTx.StepsInFiles(kv.CommitmentDomain)
acctStep := execRoTx.StepsInFiles(kv.AccountsDomain)
storStep := execRoTx.StepsInFiles(kv.StorageDomain)
codeStep := execRoTx.StepsInFiles(kv.CodeDomain)
maxStateStep := max(acctStep, storStep, codeStep)
if maxStateStep > cmtStep {
    return fmt.Errorf("snapshot step misalignment: state domains (accounts=%d, storage=%d, code=%d) ahead of commitment=%d — snapshot files need rebuilding", ...)
}

This fires when the commitment domain files lag the state-domain files — exactly the inconsistency described in #21992 / #22092 (parallel exec skips the step-boundary commitment checkpoint, so a block straddling a step edge leaves commitment .kv inconsistent with the state-domain .kv).

The step math lines up precisely:

• stepSize = 1,562,500 (3,615,626,461 ÷ 2314 ≈ 1,562,500.6 — matches the archive step size cited in execution/commitment: parallel executor skips step-boundary commitment checkpoint → snapshot domain/commitment inconsistency (blocks v3.6 release) #21992).
• Step 2313→2314 boundary is at txNum 3,615,625,000.
• Last committed txNum is 3,615,626,461 — only 1,461 txNums past the boundary, so block ~25423801 is the block straddling the step edge. The state domains froze to step 2314; commitment couldn't checkpoint at the boundary and is stuck at 2313.
• BuildFilesInBackground step=2313 lastInDB=2314 reorgSafeStep=2313.99 corroborates: commitment frozen at 2313 while the DB/state is at 2314.

Layer 2 — why it's silent and loops forever

In parallelExecutor.execImpl (execution/stagedsync/exec3_parallel.go):

1. executeBlocks returns the misalignment error before dispatching any block → pe.blockExecutors stays empty.
2. The errgroup cancels → execLoop drains, sees 0 pending blocks, and execLoopExitCheck returns nil → closeApplyChannels().
3. The apply loop sees applyResults closed with reachedMaxBlock=false and lastBlockResult.BlockNum=0 → returns &ErrLoopExhausted{From: 25423802, To: 0} (exec3_parallel.go:481).
4. pe.wait() returns the real executeBlocks error → execErr = errors.Join(ErrLoopExhausted, misalignmentErr) (exec3_parallel.go:673-678).
5. The Warn at exec3_parallel.go:688-689 is gated on !(Is(context.Canceled) || Is(&ErrLoopExhausted{})). Because the join Is ErrLoopExhausted, the Warn is skipped and the real error is never surfaced.
6. exec3_parallel.go:718 then updates the stage to the unchanged lastCommittedBlockNum and returns the ErrLoopExhausted-flavored error. sync.go treats ErrLoopExhausted as "more work pending" → the stage reruns → back to step 1. Forever, silently.

Evidence mapping

Observation	Cause
blk=0 blks=0, lastExecutedBlockNum=0	error before any block dispatched
done in=317µs	guard fails at the top of executeBlocks
done blk=25423801 stuck	stage progress = unchanged lastCommittedBlockNum
no WARN/ERROR	error joined with ErrLoopExhausted, Warn skipped
step=2313 vs lastInDB=2314	commitment files lag state-domain files

Proposed fixes

A. Unstick an affected node (recovery). Realign commitment files to the state domains:

erigon integration commitment rebuild --datadir <dir> --chain <chain> --no-history

Confirm first by comparing the max end-step of commitment.*.kv vs accounts/storage/code.*.kv under <datadir>/snapshots/domain/. Nodes that downloaded snapshots produced by a pre-#22092 binary may carry this misalignment.

B. Prevent recurrence. #22092 (checkpoint commitment at step boundaries in parallel exec). Note this only prevents new misalignment — it does not repair an already-misaligned datadir and does not fix the silent loop.

C. Fix the silent loop itself (independent bug). A persistent pre-execution failure in executeBlocks (this guard, a missing/nil block, a BAL-decode failure, …) must not be masked by ErrLoopExhausted. In execImpl, when pe.wait() returns a non-Canceled/non-ErrLoopExhausted error, it should take precedence over the apply loop's ErrLoopExhausted (override rather than join) so the Warn fires and the stage fails loudly. This is arguably the higher-priority fix: it would have turned #20805's consequence into a one-line fatal instead of an endless quiet spin, and it guards the whole class of "executeBlocks errored before dispatch" cases.

Related

• parallel commitment calculations implemented #20805 — introduced both the misalignment and the guard (b72aa7b4f7)
• execution/commitment: parallel executor skips step-boundary commitment checkpoint → snapshot domain/commitment inconsistency (blocks v3.6 release) #21992 — same root cause, framed as a CommitmentRoot integrity-check false positive ("block execution is unaffected"); this issue is the runtime counterpart
• execution/stagedsync: checkpoint commitment at step boundaries in parallel exec #22092 / [3.6] execution/stagedsync: checkpoint commitment at step boundaries in parallel exec #22097 — step-boundary commitment checkpoint fix (main / performance-v36)

[VBulikov]

yperbasis added Imp1 as Importance

[VBulikov]

yperbasis changed Importance from Imp1 to Imp0