Parallel-exec: route the per-tx writeset through one faithful path; remove normalizeWriteSet/calcState; carry serial-finalize signals on ExecutionResult

[mh0lt]

Follow-up to #21136. The recurring parallel-exec-vs-serial divergences (SELFDESTRUCT/recreate, EIP-161 empty-removal, EIP-7708 burn logs, fork-transition, coinbase) all come from the post-execution wrapper re-deriving "what did this tx do" via the heuristic normalizeWriteSet(versionWritten-log) path and the calcState re-derivation, instead of using the faithful MakeWriteSet-style output that the validated worker already produces. ~half of genuine parallel-executor PRs over the last 2 months are this family. This issue is the consolidation that makes those bugs structurally impossible.

Target per-tx flow (incremental — does NOT require the full "IBS = pure transaction-state" rework)

1. Parallel-execute txs (unchanged).
2. Validate + serialize via BlockSTM (unchanged) — at this point we have a verified version. The validated worker's IBS already ran MakeWriteSet into a LightCollector → result.CollectorWrites = every dirty object's final state.
3. Per-tx serial-finalize — done from data + the writeset, no reconstructed IBS:
   • per-tx delta = filterWritesByVersionMap(result.CollectorWrites, vmWrites) (faithful, pruned to the keys the tx actually touched — both filterWritesByVersionMap and CollectorWrites already exist).
   • accumulate tip serially → final coinbase / burnt balances → patch the writeset (read pre-block balance from sd).
   • EIP-161 empty-removal: a writeset entry that's {bal=0, nonce=0, empty codehash} → convert to a delete.
   • EIP-7708 transfer/burn logs: compute from the writeset + a self-destructed-addresses list carried on ExecutionResult (see below) — delete the minIBS + postApplyMessageFunc reconstruction (exec3_parallel.go ~line 2807; the minIBS exists only to SetBalance then LogSelfDestructedAccounts reads it).
   • SD storage cascade if still needed (or it falls out of the collector's final state).
4. Pass the finished tx output to the commitment calculator: the one writeset → applyVersionedWrites writes it into SharedDomains/sd.mem (the public DB the calc reads from), AND VersionedWrites.TouchUpdates(updates) folds it into the calc's commitment.Updates. Calc reads post-block siblings from sd.mem via asOfStateReader.

Block-end finalize (block reward, withdrawals, EIP-4788/7002/7251 syscalls) genuinely runs state transitions and keeps a real IBS — it's outside the per-tx flow and not in scope here.

Concrete changes

• ExecutionResult gains SelfDestructedAddresses []common.Address (regardless of residual balance) — populated by the worker; consumed by per-tx serial-finalize for EIP-7708 and (potentially) the SD storage cascade.
• txResult.writes = filterWritesByVersionMap(result.CollectorWrites, be.blockIO.WriteSet(txIndex)) instead of normalizeWriteSet(...).
• the commitment calc consumes txResult.writes.TouchUpdates(...) — delete calcState.ApplyWrites/FlushToUpdates.
• delete normalizeWriteSet and resolveStorageWrites (the latter is already DEPRECATED dead code).
• unify the per-tx and block-finalize collector paths (LightCollector vs versionedWriteCollector — one of them).
• IBS.VersionedWrites(checkDirty) stays, used only for BlockSTM read/write dependency tracking — it must stop being the apply/commitment payload.

Validation gate before deleting anything

Remove the EXEC3_PARALLEL-gated SkipLoad/t.Skips from #21136 one by one — if the consolidated path is faithful, test_double_kill, test_dynamic_create2_selfdestruct_collision, the prague test_system_contract_deployment variants, tipInsideBlock, and TestEIP7708BurnLogWhenCoinbaseSelfDestructs should all pass without per-bug heuristics. That's the success criterion.

Eventual rework (separate, larger)

Extract the whole serial flow (MakeWriteSet → commitment fold → domain write → BAL) to run post-validation as the single path; serial becomes a 1-tx-at-a-time consumer of it; IBS becomes a pure per-tx transaction-state. This issue is the stepping stone.

[mh0lt]

Phased migration plan

Goal: replace the heuristic per-tx writeset reconstruction with the faithful MakeWriteSet-derived path that's already (mostly) wired but sidelined. Each phase is independently mergeable and validated by un-gating one or more of the #21136 EXEC3_PARALLEL-gated skips.

What dissolves vs. what survives

normalizeWriteSet's heuristics all dissolve because MakeWriteSet/updateAccount reads each dirty object's final state, not a write-event log:

• sdSet / sdEarlier ("was this address self-destructed, and when?") — gone; the final state of a SELFDESTRUCT'd-then-recreated account just is the final state.
• "fill missing account fields" — gone; LightCollector.UpdateAccountData already emits all fields of each dirty object.
• no-op filtering / stale-incarnation filtering — gone; filterWritesByVersionMap(CollectorWrites, vmWrites) already prunes to exactly the keys the tx's IBS actually wrote (that's the whole point of the filter — it exists, it's just barely used).
• EIP-161 empty-removal — gone from here; serial's updateAccount(IsSpuriousDragon, …) already turns an empty dirty account into a DeleteAccount, so CollectorWrites already carries the delete.
• calcState.ApplyWrites/FlushToUpdates (incl. its Deleted-clearing logic) — gone; fold the writeset into the commitment Updates directly via VersionedWrites.TouchUpdates.

What survives (faithful helpers, not heuristics):

• SD storage cascade: for an address with a SelfDestructPath=true entry, emit StoragePath=0 for every slot currently committed under it (vm.StorageKeys ∪ domainStorageKeys). This isn't a heuristic — it's the parallel-side mirror of serial's DomainDelPrefix(StorageDomain, addr), needed because the commitment calc folds per-slot and can't prefix-delete. Stays as a small helper layered on top of filterWritesByVersionMap.
• per-tx serial-finalize arithmetic: accumulate tip serially → final coinbase/burnt balances → patch the writeset; EIP-7708 transfer/burn logs. Already mostly in result.finalize / postApplyMessageFunc; consolidate, and (Phase 3) make it IBS-free.

Phases

Phase 0 — shadow-compare (no behavior change). In be.nextResult, compute both normalizeWriteSet(rawWrites,…) (current) and filterWritesByVersionMap(result.CollectorWrites, rawVmWrites) + SD-cascade (proposed). Behind a debug flag (or always, in tests), log/assert the diffs. The diffs are the bug surface; the proposed path should be the correct one wherever they disagree. De-risks Phase 1.

Phase 1 — make the per-tx delta the faithful one. txResult.writes = filterWritesByVersionMap(result.CollectorWrites, rawVmWrites) + SD-cascade, instead of normalizeWriteSet(…). Keep normalizeWriteSet in the tree one cycle (unused) as a fallback. Un-gate frontier/opcodes/test_double_kill and cancun/eip6780_selfdestruct/test_dynamic_create2_selfdestruct_collision_two_different_transactions (root cause: the validated tx's empty-account/recreate effect not reaching the apply path because normalizeWriteSet reconstructed instead of using CollectorWrites). Verify make test-all parallel + blocktests-stable-parallel shard.

Phase 2 — fold commitment from the same writeset. Calc consumes txResult.writes.TouchUpdates(updates) per tx instead of calcState.ApplyWrites + FlushToUpdates. Delete execution/stagedsync/calc_state.go. Verify the trie input is identical (the Updates buffer dedupes per-key per-block, so per-tx folding == per-block flushing).

Phase 3 — carry serial-finalize signals on ExecutionResult; kill the minIBS. Add ExecutionResult.SelfDestructedAddresses []common.Address (all addresses SELFDESTRUCT'd, regardless of residual balance), populated by the worker. Per-tx serial-finalize computes EIP-7708 burn logs from that list + the writeset (final balance from the writeset or a sd read) — no minIBS/postApplyMessageFunc reconstruction. Un-gate TestEIP7708BurnLogWhenCoinbaseSelfDestructs.

Phase 4 — delete normalizeWriteSet and resolveStorageWrites (the latter is already DEPRECATED dead code). Unify the per-tx and block-finalize collector paths (LightCollector vs versionedWriteCollector — keep one). IBS.VersionedWrites(checkDirty) stays, used only for BlockSTM dep tracking — assert (lint/comment) it's no longer the apply/commitment payload.

Phase 5 (out of scope here, larger) — extract the whole serial flow (MakeWriteSet → commitment fold → domain write → BAL) to run post-validation as the single path; serial becomes a 1-tx-at-a-time consumer; IBS becomes a pure per-tx transaction-state.

Open questions for Phase 0

• Does the worker call FinalizeTx/MakeWriteSet for every tx today, or only when needed? (result.finalize in exec3_parallel.go runs MakeWriteSet-ish into the LightCollector — confirm result.CollectorWrites is populated for all txs, not just fee/BAL.)
• tipInsideBlock (Cancun gasUsed +4800) and the prague test_system_contract_deployment variants — do these fall out of Phase 1, or are they a separate cause (gas accounting / fork-transition)? If separate, they stay gated until addressed independently.

[mh0lt]

Heuristic-removal checklist (the "group" that moves together)

Process per heuristic: (1) identify the target, (2) ensure a test pins the correct replacement behavior — write one if missing, (3) change + run. These can't be pulled one at a time — normalizeWriteSet/calcState heuristics mutually compensate, so Phases 1+2 land as one atomic swap (with normalizeWriteSet kept dead one cycle as a fallback). Below: each heuristic → its faithful replacement → test status.

#	Heuristic to remove	Replaced by	Test status
1	normalizeWriteSet sdSet pre-scan + drop account-field writes for SD'd addrs	CollectorWrites shows the SD'd account's final state (deleted, or recreated) directly — no need to reason about "was it SD'd"	covered: TestRecreateAndRewind, TestSelfDestructReceive, TestCVE2020_26265; need: unit test on the new path's writeset for "SD in tx A, recreate in tx B"
2	normalizeWriteSet sdEarlier → fill missing fields with zero	CollectorWrites of the re-creating tx shows the (empty) account	gated repro: test_double_kill, test_dynamic_create2_selfdestruct_collision (un-gate as Phase-1 validation); covered: TestSelfDestructReceive
3	normalizeWriteSet StoragePath no-op filter (incl. SD-aware baseline)	filterWritesByVersionMap prunes to vmWrites keys; CollectorWrites only has slots WriteAccountStorage touched	covered: TestDeleteRecreateSlots, TestDeleteRecreateSlotsAcrossManyBlocks
4	normalizeWriteSet fill-missing-account-fields loop	LightCollector.UpdateAccountData emits all fields of every dirty object (mirrors serial MakeWriteSet)	need: test that a storage-only-touched address still gets its account fields in the writeset
5	normalizeWriteSet EIP-161 empty-removal section + emptyRemoval param	serial's updateAccount(IsSpuriousDragon,…) already turns an empty dirty account into a DeleteAccount → CollectorWrites carries it	covered: TestEIP161AccountRemoval
6	normalizeWriteSet stale-incarnation skips (w.Version.Incarnation != incarnation)	CollectorWrites is from the validated incarnation's IBS — no stale entries	covered by version-validation tests; verify no new coverage gap
7	calcState.ApplyWrites Deleted-clearing (sdThisCall) + the 3-way Deleted flush branches in FlushToUpdates	txResult.writes.TouchUpdates(updates) folds directly: SelfDestructPath/DeleteAccount entry → DeleteUpdate; normal account write → `Balance	Nonce
8	calcState lazy-load (ensureAccount/ensureStorage reading the domain)	nothing — ComputeCommitment does its own sibling reads via asOfStateReader; TouchUpdates just emits touched keys	n/a (deleting calc_state.go covers it)
9	(done — e7c4c41c3d) IBS.Selfdestruct's StoragePath=0 versionWritten loop	the SD storage cascade in the writeset path (survives, see below)	covered: TestSelfDestructKeepsDirtyStorageReadableSameTx

Faithful helpers that survive (still need tests)

Helper	Why it stays	Test status
SD storage cascade (sdStorageSlots = vm.StorageKeys ∪ domainStorageKeys, layered on filterWritesByVersionMap)	parallel-side mirror of serial's DomainDelPrefix(StorageDomain, addr) — the calc folds per-slot and can't prefix-delete	covered: TestDeleteRecreateAccount (genesis storage), TestCVE2020_26265 (batch-local storage); need: explicit unit test that a SELFDESTRUCT'd address gets StoragePath=0 for every committed slot (genesis + batch-local + prior-block)
per-tx serial-finalize fee arithmetic (accumulate tip → coinbase/burnt balances → patch writeset)	block-level serial state can't be done speculatively per-tx	covered: engine-API TestEngineApi*
EIP-7708 burn logs from ExecutionResult.SelfDestructedAddresses + writeset (Phase 3)	replaces the minIBS/postApplyMessageFunc reconstruction	gated repro: TestEIP7708BurnLogWhenCoinbaseSelfDestructs (un-gate as Phase-3 validation); need: unit test for the burn-log computation given a SelfDestructedAddresses list + a writeset

Missing tests to write before the swap

1. unit: new writeset path produces correct output for "SD addr in tx A, recreate-via-transfer in tx B" (covers Pull from go-ethereum up to 26aea73 (7 Feb 2019) #1, Pull from go-ethereum up to 2f24e25 (6 Mar 2019) #2).
2. unit: storage-only-touched address gets account fields in the writeset (covers Rename package paths to run unit tests separately #4).
3. unit: VersionedWrites.TouchUpdates → correct commitment.Update for a SelfDestructPath entry (vs a normal account write) (covers Rename LDBDatabase to BoltDatabase, fix some compile errors in tests #7).
4. unit: SD storage cascade deletes every committed slot of a SELFDESTRUCT'd address (genesis + batch-local + prior-block) (covers the surviving helper).
5. unit: EIP-7708 burn-log computation from (SelfDestructedAddresses, writeset) (covers Phase 3's surviving helper).

[mh0lt]

Cross-block read-cache consolidation (sub-task — requires archaeology first)

Separate from the per-tx writeset work above, the parallel path has (at least) three overlapping "current state of address X" caches that close the cross-block timing hole (block N+1's workers run before block N's apply has reached sd.mem):

1. BlockStateCache (rw_v3.go:1026) — fed by applyVersionedWrites as it writes; CachedReaderV3 reads it; Flush replays the writeLog into sd.mem at block end. Has the Flush-to-sd.mem machinery — the canonical one if we keep a cache.
2. StateV3Buffered.accounts (the bufferedAccount/deleted map) — maintained by versionedWriteCollector.UpdateAccountData/DeleteAccount "for the cross-block timing hole bridge" (its own comment).
3. the SharedDomains-level warmup/state cache (doms.GetStateCache(), validated against parent hash; #43a64a0b66 "fix two cache bugs" touched it).

Each has its own delete/staleness rules (delete handling is the perennial bug — e.g. BlockStateCache.DeleteAccount not clearing currentAccounts so a later put resurrects the account). Same "multiple re-derivations" pattern as normalizeWriteSet/calcState, applied to the cross-block read view.

IMPORTANT — these were added during fix processes, reactively, to patch specific timing holes. Do NOT consolidate/remove without doing the archaeology first:

• Enumerate, for each of Pull from go-ethereum up to 26aea73 (7 Feb 2019) #1/Pull from go-ethereum up to 2f24e25 (6 Mar 2019) #2/Pull go-ethereum up to 26aea736 (7 Feb 2019) #3, exactly which timing hole / which bug it closes (git-blame + the test it was added with).
• Decide the target: (a) one cross-block bridge populated from the single post-validation writeset (CollectorWrites), deleting the other two; or (b) drop cross-block pipelining entirely (block N+1's workers wait for block N's apply → no cache needed, less parallelism).
• Prove the target covers every hole Pull from go-ethereum up to 26aea73 (7 Feb 2019) #1/Pull from go-ethereum up to 2f24e25 (6 Mar 2019) #2/Pull go-ethereum up to 26aea736 (7 Feb 2019) #3 covered today; add a test pinning each.
• Only then change the code.

Until that's done, the caches stay as-is — not touched in the whack-a-mole, not "simplified" opportunistically.

[mh0lt]

Additional issue found while validating: per-block changeset accumulator not reliably installed → domains.GetDiffset(N, hash): not found

Same family (a separately-orchestrated derivation the parallel scheduler hand-rolls — here the per-block changeset accumulator — that slips out of sync under load). Surfaces as intermittent updateForkChoice: [4/6 Execution] domains.GetDiffset(N, 0x…): not found — a later unwind/FCU can't revert block N because block N's diffset is empty/missing. Best repro: the execution/execmodule package under its own t.Parallel() load (~1-in-3), e.g. TestValidateChainAndUpdateForkChoiceWithSideForksThatGoBackAndForwardInHeight; also hits assorted rpc/jsonrpc tests (TestTraceTransaction, TestGetStorageValues_*, TestRawTransactionVmTrace, …) under make test-all load. Not gateable — non-deterministic, different tests each run. No WARNING: DATA RACE (it's a logical race, not a memory race).

Diagnosis

The parallel exec keeps a single pe.currentChangeSet accumulator, installed/cleared by the exec loop per block:

• exec start: seed for startBlockNum (exec3_parallel.go:191-193).
• per block: at the block's blockResult → SavePastChangesetAccumulator(blockHash, N, currentChangeSet), then SetChangesetAccumulator(nil) / currentChangeSet = nil (:870-877).
• install for the next block: if pe.blockExecutors[N+1] exists at this moment → new StateChangeSet + SetChangesetAccumulator (:919-921).

The window: after block N's blockResult is processed (accumulator cleared, pe.blockExecutors map can go empty) but before block N+1's processRequest arrives, the install at :919-921 does nothing (N+1 not in the map yet). Then processRequest lands, sees the map empty, and schedules block N+1 directly (:1009-1027, the scheduleable path) — with no accumulator installed. N+1's ApplyStateWrites then writes DomainPuts into a nil domainWriters[idx].diff → N+1's changeset stays empty → its diffset is never saved (memory) or flushed (DB) → a later unwind across N+1 fails. Load-sensitive because whether the map empties in that gap depends on how fast executeBlocks pumps blocks vs. the exec loop drains them.

(Related, separate scenario: pe.shouldGenerateChangesets is computed once per batch from startBlockNum — shouldGenerateChangeSets(cfg, startBlockNum, maxBlockNum) — whereas serial decides per block (exec3_serial.go:88). For a multi-block batch [start, …] whose start + MaxReorgDepth < maxBlockNum but whose later blocks are within MaxReorgDepth of the tip, parallel generates no changesets for the batch while serial would generate them for the tip-adjacent blocks → unwind across that boundary fails. Batch boundaries shift under memory pressure, so this is also load-sensitive. Both should be fixed together.)

Proposed fix (for the follow-up PR — NOT #21017)

• Make changeset coverage per-block, matching serial: replace pe.shouldGenerateChangesets bool with pe.shouldGenerateChangesetForBlock(n) = shouldGenerateChangeSets(cfg, n, pe.maxBlockNum). Set forcePerBlockCompute from the batch's last possible block (min(startBlockNum+blockLimit, maxBlockNum)), not its start.
• Make the accumulator install robust against blocks scheduled out-of-band: track pe.currentChangeSetBlock; in processResults (the single exec-loop point that drains worker results), before be.nextResult(txResult), ensureChangesetAccumulator(txResult.Version().BlockNum) — if the block needs a changeset and the current accumulator isn't for it, install a fresh one (the previous block's was already saved at its blockResult). Also call it at the blockResult save point so empty blocks get an (empty) changeset saved. Then :919-921's install becomes the fast-path, ensure the safety net (both idempotent via currentChangeSetBlock). Keep all SetChangesetAccumulator calls in the exec loop (the comment's single-writer invariant holds — processRequest runs in executeBlocks, so it must NOT install; the exec loop must).

Validation gate

Run execution/execmodule package + make test-group {execution-other, execution-tests, core-rpc} GO_FLAGS="-race" under EXEC3_PARALLEL=true, ~20 iterations each — no GetDiffset(…): not found.

[mh0lt]

Note on the GetDiffset fix above: it (the ensureChangesetAccumulator install-robustness, the pe.currentChangeSet/currentChangeSetBlock rotation in the exec loop, and the existing changesetMu band-aid in SharedDomains) is itself a band-aid on the parallel orchestration — it exists only because changeset generation is hand-coordinated against the exec-loop/apply-loop/calc handoff. In the Phase-5 rework (serial flow — including changeset generation via applyVersionedWrites→DomainPut→diffset — runs post-validation, one block at a time) there is no rotation window and no concurrent reader, so this whole machinery is on the removal list, alongside normalizeWriteSet/calcState/resolveStorageWrites and the redundant cross-block caches. i.e. the tactical fix unblocks the parallel -race jobs now; it does not become permanent — track it for deletion when the changeset-generation moves post-validation.

[mh0lt]

Add to the missing-tests list (item 6): a deterministic repro for the per-block changeset-accumulator install hole. It should drive the parallel exec so that a block gets scheduled via processRequest with an empty blockExecutors map mid-batch (the out-of-band path), then unwind across that block, and assert doms.GetDiffset(N, hash) returns the block's diffset rather than "not found". Phrase the assertion in terms of behavior — "every executed block's state changes are present in its diffset, so any unwind across it can revert" — not the mechanism, so the test survives the Phase-5 rework that deletes the currentChangeSet/ensureChangesetAccumulator/changesetMu machinery. (If a fully-deterministic scheduling repro is impractical, a stress variant — N iterations of a small multi-block insert + unwind under EXEC3_PARALLEL=true with -count/-race — counts, but a deterministic one is preferred.)

[mh0lt]

Status: the changeset-accumulator install fix is now PR #21145 (standalone, off main, single commit). It is intentionally a band-aid on the parallel changeset orchestration — when the Phase-5 rework moves changeset generation post-validation (one block at a time, no rotation window), the whole pe.currentChangeSet/pe.currentChangeSetBlock/ensureChangesetAccumulator/changesetMu machinery introduced/touched there comes out. Re-address (remove) this when doing the rework. mh/ci-exec-mode-matrix (#21017) picks it up automatically the next time it merges main.

[mh0lt]

Remaining parallel-only flake to fix for reliable CI: Wrong trie root of block 1 under load

Distinct from the GetDiffset race (now fixed in #21017). Repro: TestValidateChainAndUpdateForkChoiceWithSideForksThatGoBackAndForwardInHeight → [5/5 Execution] Wrong trie root of block 1: <computed> expected <expected> — ~1/12 with -parallel 1, ~4/15 under the execution/execmodule package's concurrent t.Parallel() load. EXEC3_PARALLEL=false is 15/15 clean (so not global-state contamination — it's an internal parallel-exec race). Under -race it still flakes with no WARNING: DATA RACE — a logical race. Pre-existing (was on the failing list of the CI run on d3121487b7).

The calc computes block N's commitment from (a) cc.state/cc.updates (block N's touched keys, folded into the trie) and (b) sd.mem/files for sibling reads via asOfStateReader (txNum = blockN.lastTxNum + 1). Ruled out so far:

• sd.mem.GetAsOf(domain, key, ts) does filter by txNum (versioned dataWithTxNum slices; returns the entry where ts > entry.txNum && (last || ts <= next.txNum)) — so a "future" block-N+1 write at txNum > blockN.lastTxNum+1 is correctly excluded.
• SharedDomains.SetChangesetAccumulator takes changesetMu — so the exec-loop's accumulator swap is serialized against the calc's LockChangesetAccumulator window in computeWithBlockAccumulator. (The new ensureChangesetAccumulator uses the same locked path — no new race.)
• inline TouchKey from applyVersionedWrites is disabled in parallel mode (SetDisableInlineTouchKey(true)), so the apply loop doesn't mutate the commitment context concurrently with the calc.
• cc.state (calcState) and cc.updates are mutated only from the calc goroutine, sequentially per block.

Remaining candidates:

1. Stale-incarnation in normalizeWriteSet/versionMap — account-field paths (BalancePath/NoncePath/IncarnationPath/CodeHashPath) are not incarnation-checked in normalizeWriteSet; w.Val is resolved via vm.Read(addr, path, txIndex+1). Under heavy re-execution the versionMap holds multiple entries for the same txIndex (different incarnations); if vm.Read returns a stale incarnation's value, block N's writeset is wrong → wrong root. Load-sensitive (more contention → more re-executions).
2. Unwind residue in sd.mem for fork re-validation — TestValidateChainAndUpdateForkChoiceWithSideForksThatGoBackAndForwardInHeight does FCU chain-switches → unwindExec3 → sd.mem.Unwind(unwindToTxNum) (drops entries with txNum > unwindToTxNum), then re-executes the fork. If the re-execution's calc reads sd.mem and an entry that should have been dropped wasn't, sibling reads are wrong → wrong root. (Unwind is latestStateLock.Lock()-guarded and happens before re-execution, so the window would have to be elsewhere.)

Next: instrument computeAndCheck to dump block N's touched-key set (from calcState, before ResetBlockFlags) on a mismatch and diff a failing run vs a passing one — different touched-set ⇒ candidate 1; same touched-set ⇒ candidate 2 (or the fold's reads). Needs a focused session.

#21017 currently carries: the 5 exec-mode fixes + the GetDiffset changeset-accumulator fix + the gated skips + the dead-Trim-comment cleanup. Reliably-green parallel matrix is blocked on this flake.

[mh0lt]

Phase 1 trace verification on PR #21207 branch

Reproduced test_double_kill deterministically on worktree-eip-8037-bal-devnet-7 (head 3fac59fe9d) — 3/3 fail in parallel, 3/3 pass in serial; only 2 failures remain in the blocktests-stable-parallel shard, both this test (Frontier + Homestead). Trace matches the stated Phase-1 root cause exactly.

Scenario (single fixture, 0xca72…8180 is the contract whose runtime is 0x30FF = ADDRESS SELFDESTRUCT):

• Block 1: deploy the SD-to-self contract.
• Block 2 tx1: call it → SD Pull from go-ethereum up to 26aea73 (7 Feb 2019) #1.
• Block 2 tx2: call it again at the same address → pre-Spurious-Dragon touch creates an empty account at 0xca72… (since IsSpuriousDragon == false on Frontier/Homestead).

Serial finalize emits the empty account into the trie:

2 (1.0) Update Account Data (*state.Writer): ca72…8180 balance:0 nonce:0 codeHash:c5d2…(empty)

Parallel finalize at the same point is silent for 0xca72…:

2 (1.0) Updated Balance: ca72…8180: 0 (0)
2 (1.0) Update Account ca72…8180
                                    # ← no "Update Account Data (*state.LightCollector): ca72…" follow-up

The sender's Update Account Data (*state.LightCollector) IS emitted in the same tx, confirming the collector is wired — it's specifically the touch-creates-empty-account effect on the SD'd contract that's lost. The version-map view (SelfDestructPath=true from tx1) causes normalizeWriteSet(rawWrites, …) to reconstruct the contract's final state from the write-event log instead of from the worker IBS's MakeWriteSet/updateAccount output that's already in result.CollectorWrites. The reconstructed state doesn't see the touch, so the trie misses the account → state-root mismatch at block 2.

This matches the issue's Phase 1 text verbatim:

Un-gate frontier/opcodes/test_double_kill … (root cause: the validated tx's empty-account/recreate effect not reaching the apply path because normalizeWriteSet reconstructed instead of using CollectorWrites).

So the artifact is: deterministic repro + per-tx trace pinpointing the missing LightCollector write at finalizeTxSimple → normalizeWriteSet call site (exec3_parallel.go:2631), as the concrete check for the Phase-1 swap.

Reproduce in ~10s (after building make evm on the PR branch + having test-fixtures-cache/eest_stable populated):

ERIGON_EXEC3_PARALLEL=true ./build/bin/evm blocktest --workers 1 \
  --run 'test_double_kill\[fork_Frontier' --jsonout \
  test-fixtures-cache/eest_stable/fixtures/blockchain_tests/frontier/opcodes/test_double_kill.json \
  | grep -v '^\[[A-Z]' | jq '.[] | select(.pass == false)'

Trace with TRACE_TRANSACTION_IO=true TRACE_ACCOUNTS=0xca728f188aff48461fac0669f88c3eedf93c8180,0x7ca39f391b88b588d392b89bee0ad30552467002.

PRs #21211 (drop dead finalize variants) and #21212 (extract runPostApplyMessageOnMinIBS) are the structural prep ahead of this — they don't change the normalizeWriteSet path, so the trace and repro above will still apply verbatim to PR 3 (the Phase-1 swap).

[mh0lt]

@/tmp/comment-updated.md

[mh0lt]

Additional repro for the cluster: chiado parallel exec from 0 → block 21 nonce-propagation failure

Logging this here rather than as a separate fix so it's tracked against the Phase-1 consolidation instead of accumulating yet another independent edge-case patch. Data point on a non-test workload that fits the same thesis (heuristic writeset re-derivation diverging from the worker's faithful CollectorWrites).

Repro

On current origin/main (f05a65e691):

EXEC3_PARALLEL=true ./build/bin/erigon \
    --datadir <chiado-datadir-with-block-snapshots> --prune.mode=archive --chain=chiado \
    --sync.loop.block.limit=10000 --log.console.verbosity=3 --batchSize=512mb

Deterministically fails on the first cycle:

exec loop error: invalid block: could not apply tx 21:0 [44:0xd5489a42f0e14478826ccbf465694ad6301a1624f9915604288a444297223f64]:
nonce too high: address 0x14747a698Ec1227e6753026C08B29b4d5D3bC484, tx: 1 state: 0

What I traced

With TRACE_ACCOUNTS=0x14747a698Ec1227e6753026C08B29b4d5D3bC484 TRACE_BLOCKS=1,21 TRACE_TRANSACTION_IO=true TRACE_APPLY=true:

0x14747a69… is the sender of block 1, tx 0 — and also the coinbase of block 1 (per ibs.Prepare: sender: 14747a69…, coinbase: 14747a69…, dest: 3000…0001). It's a zero-gas-price ~259k-gas tx (balance unchanged, val: 0). The pre-validation SetNonce bumps its nonce 0→1; it's then idle in blocks 2..20. Block 21 has another tx from it with nonce 1, which then fails because state says 0.

The worker IBS does call the LightCollector with the right post-tx state:

1 (0.0) Update Account Data (*state.LightCollector): 14747a698ec1227e6753026c08b29b4d5d3bc484 balance:0,nonce:1,codehash:c5d2460186…

But the per-block apply layer shows:

1 apply:del code+storage: 14747a698ec1227e6753026c08b29b4d5d3bc484
1 apply:del account: 14747a698ec1227e6753026c08b29b4d5d3bc484

i.e. applyVersionedWrites receives SelfDestructPath=true with no field writes (d.balance == d.nonce == d.incarnation == d.codeHash == nil) and takes the pure-delete branch at execution/state/rw_v3.go:177. So somewhere between LightCollector saying Nonce: 1 and applyVersionedWrites running, the per-tx writeset for 0x14747a69… got converted to a pure SD.

resetObjectChange.dirtied() fix does NOT clear this case

Tried the journal asymmetry fix from your test_double_kill diagnosis above (resetObjectChange.dirtied() returning ch.account, true). Chiado still fails identically at block 21. That makes sense — 0x14747a69… is a fresh address never seen pre-block-1, so its first getStateObject(addr, true) goes through createObject(addr, previous=nil) → createObjectChange (not resetObjectChange). Confirms this is a different instance in the cluster.

Candidate location

The writeset transformation that drops NoncePath=1 and inserts SelfDestructPath=true for a non-empty account most likely lives in:

• normalizeWriteSet's EIP-161 empty-removal stanza (exec3_parallel.go ~3457+) — should be gated on s.nonce == 0, so shouldn't trip with NoncePath=1 in filtered; worth confirming the acctStates[0x14747a69…] actually sees the LightCollector's Nonce=1 and not a synthesized Nonce=0.
• Or the block-finalize MakeWriteSet path at exec3_parallel.go:2929 (the VersionedWriteCollector collector, separate from the per-tx LightCollector) — 0x14747a69… being the block-1 coinbase, AuRa Finalize touches it; if the finalize-side writeset arrives at apply with SD=true and no fields it'd produce exactly this trace.

Either way it fits the Phase-1 thesis: the heuristic-rebuilt writeset diverges from the worker's faithful CollectorWrites. Tracking it here alongside test_double_kill / test_dynamic_create2_selfdestruct_collision etc. so Phase 1 is the resolution path rather than a separate patch. If the Phase-1 swap doesn't subsume it, happy to drill in then — or defer to whoever owns the consolidation. (Not opening a separate issue or proposing a patch — explicitly avoiding the independent-edge-case-fix mode.)

Build hash: f05a65e69192d979ffa658d9481fd3ae584631f4. Recipe + traces ready if useful.