execution/commitment: parallel executor skips step-boundary commitment checkpoint → snapshot domain/commitment inconsistency (blocks v3.6 release)

[awskii]

erigon seg integrity --check=CommitmentRoot reports a false recomputed root does not match verified root on commitment .kv files whose checkpoint sits on a complete block boundary and whose block has account changes. This gates the v3.6 snapshot-release automation (erigon-snapshots-automation) on all chains (sepolia, ethmainnet, hoodi). The published snapshots are correct — it is a false positive in the check's recompute.

Symptom

integrity error: recomputed root does not match verified root:
  0xf236e7… != 0x646f53… : in v2.1-commitment.2624-2656.kv with startTxNum=1025000000, endTxNum=1037500000

Data is valid

For each failing file the check reaches the recompute only after verifying stored root == canonical block-header root (checkCommitmentRootViaFileData) and stored root == SeekCommitment file-loaded trie root (checkCommitmentRootViaSd). Both pass. Only checkCommitmentRootViaRecompute (db/integrity/commitment_integrity.go, TouchChangedKeysFromHistory(blockMinTxNum, info.txNum+1) + ComputeCommitment) diverges.

Trigger (precise)

Fails iff info.txNum == blockMaxTxNum (PartialBlock()==false, complete block) and the block has account churn.

• partial-block files: always pass.
• clean-boundary, accountTouches==0 (chiado 160-168): pass.
• clean-boundary, accountTouches>0 (sepolia 2624-2656, hoodi 272-276): fail.

Reproduced, step-independent, identical roots

• snap36-sepolia: references=false, step 390625, v2.1-commitment.2624-2656.kv: recomputed 0xf236e7 ≠ stored 0x646f53.
• arb-dev /erigon-data/sepolia: references=false, step 1562500, archive, v2.1-commitment.656-664.kv, same txn range 1025000000-1037500000: identical 0xf236e7 ≠ 0x646f53.

Ruled out (by experiment)

RootHash() side-effect on the restored trie; state-reader txnum (info.txNum vs endTxNum-1); touch-set completeness (full-range touch); empty/tombstone branches (honored on read; unfold errors on them, recompute returns a wrong root not an error); step-size / migration (identical at 1562500 and 390625; datadir built fresh, not rebased); file version (v1.1/v2.1 confounded with partial/clean — a v2.1 partial file passes).

Why prod doesn't hit it

Production builds commitment incrementally and reuses persisted memoized stateHash; it never re-folds these cells. The recompute invalidates and re-folds them, exposing a latent divergence. Must not be "fixed" by disabling memoization in the check or relaxing the check.

Leading lead

accountTouches==0 passes → the diverging quantity is an account leaf hash recomputed at a complete block boundary, most likely the account's storageRoot.

Next step

TRIE_TRACE_BLOCK=11105912 on v2.1-commitment.2624-2656.kv (sepolia) → the exact diverging cell. Untested axis: references=true datadir.

Reproduces on older build cb894784, so likely preexisting in the recompute/fold path (or the #21026 FilesOnlyStateReader check), surfaced now because v3.6 runs CommitmentRoot in automation and the finer step size makes clean-boundary files common.

[VBulikov]

yperbasis added Imp1 as Importance

[awskii]

Root cause: parallel executor skips the step-boundary commitment checkpoint

This isn't a check false-positive. CheckCommitmentRoot correctly detects that the account/storage domain values are inconsistent with the commitment branches frozen into the same step .kv — for files whose checkpoint lands on a complete block boundary with account churn.

Where

Parallel execution disables step-boundary commitment, and nothing else saves it:

• execution/stagedsync/exec3_parallel.go:250 — pe.rs.StateV3.SetSkipStepBoundaryCommitment(true).
• execution/state/rw_v3.go:388 and :429 — the compute+save is gated ... && !rs.skipStepBoundaryCommitment, so at (txNum+1)%stepSize == 0 nothing is checkpointed in parallel mode.
• The setter comment (rw_v3.go:439) says step-boundary commitment "would clear the Updates buffer mid-batch" and assumes "the calculator handles this" — but execution/stagedsync/committer.go only computes at block boundaries (br.lastTxNum). The step-end checkpoint is dropped.

Mechanism

When a block straddles a step edge:

• the block's early account/storage writes (txNums in step S) land in step S's account/storage .kv;
• commitment is only checkpointed at the last complete block (info.txNum, before those writes);
• no step-end checkpoint reconciles them.

Step S's commitment .kv therefore lags its domain .kv by the straddling block's early writes. The stored root still equals the header (the checkpoint is a valid complete-block root), so checkCommitmentRootViaFileData and checkCommitmentRootViaSd pass; checkCommitmentRootViaRecompute reads the fuller step-S domain and diverges.

This matches every observed case:

• complete-boundary + churn → fail;
• partial-block file (info.txNum == endTxNum-1) → pass (commitment and domain describe the same partial state);
• accountTouches == 0 → pass;
• reading values at info.txNum vs endTxNum-1 → identical wrong root (both see the same step-S domain).

Evidence

• Reproduces on a clean re-exec from the base (drop state ≥ step N + drop chaindata + re-execute) under parallel execution — fresh step files still fail. Not inherited corruption, not db/state: fix unwind restoring stale values across step boundaries #21981.
• Per-cell confirmation: commitment.VerifyBranchHashes (execution/commitment/verify.go, MemoizationOff=true) recomputes each branch cell's stateHash from the domain values and finds hundreds of mismatches per failing file (e.g. 371 / 633 / 258 in three single-step files), clustered on the contracts that churn across the step edge. A passing file (different step range) gives 0 mismatches, so the verifier is sound.
• Passes on sequential / v3.4 producers, which save step-boundary checkpoints.

Introduced by

b72aa7b4f7 (#20805, parallel commitment) — the commit that added SetSkipStepBoundaryCommitment.

8f6a7736e7 (db/state: dedup latest-values per (key,step) in domain flush) is a related nearby fix and is not on this branch; worth pulling, but it doesn't substitute for a real step-end checkpoint.

Impact

Commitment root stays correct (== header) → block execution / consensus are unaffected. But published snapshots carry domain values inconsistent with the commitment → eth_getProof, debug_executionWitness, and historical state reads return wrong values. Regenerating via re-exec does not fix it (the parallel producer reintroduces it every run), so this is a release blocker for any snapshot built with parallel execution.

Fix

1. Parallel calculator saves a partial commitment checkpoint at (txNum+1)%stepSize == 0, then continues the straddling block from it (reset block flags). Must handle the Updates-buffer-clear-mid-batch concern that motivated the skip.
2. Producer guard: before building step S, require its commitment KeyCommitmentState to decode to txNum == stepEnd-1 (invariant: every state step has a step-end checkpoint).

Workaround

Generate snapshots with sequential execution (skip flag never set → step-boundary checkpoints saved). Slower, but produces consistent snapshots that pass CheckCommitmentRoot until the parallel fix lands.

[AskAlexSharov]

#21996 (comment)
(maybe need cherry-pick to main)