Parallel-exec: residual functional failures skipped to land the exec-mode CI matrix

[mh0lt]

Tracking issue for parallel-exec test failures that were SkipLoad/t.Skip-gated (under EXEC3_PARALLEL=true only) so the serial-vs-parallel CI matrix (#21017 / mh/ci-exec-mode-matrix) can land green. Serial coverage of these is unaffected.

Skipped under parallel exec

• TestExecutionSpecBlockchain/cancun/eip6780_selfdestruct/test_double_kill.json — wrong trie root at block 2. Block 1 deploys a self-destructing contract; block 2 tx0 CALLs it (SD), tx1 0-value-CALLs it. Serial re-creates it as an empty account ({0,0,empty,0}); under parallel the validated tx1's empty-account re-creation never reaches the apply/commitment path (writeset-reconstruction layer drops it). Same family as execution/stagedsync, db/state: parallel-commitment correctness for reorg/unwind + SD recreate #21088 / state, stagedsync: parallel-exec wrong-root fix for SD vs EIP-161 emptyRemoval #21032.
• TestExecutionSpecBlockchain/cancun/eip6780_selfdestruct/test_dynamic_create2_selfdestruct_collision_two_different_transactions.json — wrong trie root at block 2; CREATE2 colliding with an address self-destructed in a prior tx. Same family.
• TestExecutionSpecBlockchain/prague/eip7002_el_triggerable_withdrawals/test_system_contract_deployment.json and .../eip7251_consolidations/test_system_contract_deployment.json — only the fork_CancunToPragueAtTime15k-...-deploy_on_fork_block-{zero,nonzero}_balance variants; system-contract deployment on the fork-transition block diverges under parallel.
• TestLegacyBlockchain/ValidBlocks/bcEIP3675/tipInsideBlock.json (Cancun + Prague) — parallel over-counts block gasUsed by ~4800; coinbase-touching txs (COINBASE/BALANCE/NUMBER/SSTORE).
• TestEIP7708BurnLogWhenCoinbaseSelfDestructs (execution/execmodule) — parallel finalize uses a minIBS that only carries coinbase/burnt via SetBalance, so it doesn't know the coinbase was self-destructed → no EIP-7708 Burn log. Needs an ExecutionResult.SelfDestructedAddresses field (regardless of balance) + LogSelfDestructedAccounts looking up the current balance for those.

Flaky under heavy parallel test load (not gated; pass on -count=3)

TestLongerForkHeaders, TestLongerForkBlocks, TestReorgShortBlocks (execution/tests), TestCustomTraceReceiptDomain (execution/stagedsync), TestSetHead (rpc/jsonrpc). The parallel reorg path has timing sensitivity — needs a hardening pass.

Architectural follow-up

These are all the same family — the post-execution wrapper re-derives "what did this tx do" (per-tx writeset via versionWritten→VersionedWrites filter→normalizeWriteSet→applyVersionedWrites; commitment-calc state via calcState.ApplyWrites; BAL recording) and diverges from what serial computes by observing the IBS directly. ~half of genuine parallel-executor PRs over the last 2 months are this family. Direction: extract the serial flow (MakeWriteSet etc.) to run post-validation as the single path (IBS becomes a pure per-tx transaction-state). Incremental first step: route the per-tx delta through versionedWriteCollector for all txs (today only the block-finalize IBS uses it), then have the commitment calc consume the same — collapsing normalizeWriteSet + calcState.ApplyWrites into pass-throughs and removing the SD/empty/recreate heuristics.

[mh0lt]

Map of the competing "what did this tx do" mechanisms

The fix-refix cycle exists because there are 5+ overlapping reconstructions of the per-tx delta; the faithful one is sidelined and the heuristic one is load-bearing.

#	Mechanism	Where	Faithful to serial?	Currently used for
1	IBS.VersionedWrites(checkDirty)	intra_block_state.go:2520	partial (write-event log, not final-state snapshot)	BlockSTM dep tracking and be.blockIO.WriteSet(txIndex)
2	normalizeWriteSet(VersionedWrites)	exec3_parallel.go:3015	no — heuristics (sdSet/sdEarlier, no-op filter, stale-incarnation filter, fill-missing-fields, EIP-161 empty-removal, SD storage cascade)	txResult.writes → applyVersionedWrites (domain) and calcState.ApplyWrites (commitment) — the load-bearing path
3	LightCollector.TakeWrites() = result.CollectorWrites	rw_v3.go:710; workers at exec3_parallel.go:686	yes (it is MakeWriteSet into a collector) — but emits stale fields for unmodified loaded accounts	fee/coinbase balance adjustments + BAL only; not the main writeset
4	filterWritesByVersionMap(CollectorWrites, vmWrites)	exec3_filter.go:14	yes (#3 ∩ #1's keys)	exists, barely used — this is the bridge that makes #3 usable as the writeset
5	versionedWriteCollector.Writes()	rw_v3.go:543; exec3_parallel.go:2524/2807	yes	only the block-finalize IBS — so finalize is faithful, per-tx is heuristic #2
6	resolveStorageWrites	exec3_parallel.go:3434	n/a	DEPRECATED — dead, delete
7	calcState.ApplyWrites/FlushToUpdates	calc_state.go	no — re-derives state from #2's output, own Deleted/SD heuristics	parallel commitment calc
8	VersionedWrites.TouchUpdates	versionedio.go:503	yes (mechanical)	the calc's intended fold; calcState (#7) shadows it
—	IBS.MakeWriteSet/updateAccount	intra_block_state.go:2112	reference	serial executor

Convergence target (pieces mostly exist)

• per-tx delta = filterWritesByVersionMap(result.CollectorWrites, rawVmWrites) (Pull go-ethereum up to 26aea736 (7 Feb 2019) #3 ∩ Pull from go-ethereum up to 26aea73 (7 Feb 2019) #1 = MakeWriteSet on the validated worker's IBS, pruned to the keys the tx touched — no stale speculative fields). Replace txResult.writes = normalizeWriteSet(...) with that.
• commitment fold = txResult.writes.TouchUpdates(updates) (Fix eth tests #8). Drop calcState (Rename LDBDatabase to BoltDatabase, fix some compile errors in tests #7).
• delete Pull from go-ethereum up to 2f24e25 (6 Mar 2019) #2, remove AscendGreaterOrEqual1 #6, Rename LDBDatabase to BoltDatabase, fix some compile errors in tests #7; merge Fix for the p2p deadlock when remapping the database #5 into the same path so finalize and per-tx use one collector.
• keep Pull from go-ethereum up to 26aea73 (7 Feb 2019) #1 for BlockSTM dependency tracking only — stop using it as the apply/commitment payload.

The rule that stops the refix cycle

Fix any parallel-vs-serial divergence by making MakeWriteSet/updateAccount correct — never by adding another branch to normalizeWriteSet or calcState. Those two are on the chopping block.

Long-term: extract the serial flow (MakeWriteSet → commitment fold → domain write → BAL) to run post-validation as the single path; IBS becomes a pure per-tx transaction-state.

[mh0lt]

TestEIP7708BurnLogWhenCoinbaseSelfDestructs is fixed in #21153 (commit 445f97e) and the t.Skip removed.

Root cause turned out to be different from the diagnosis above. The parallel-exec finalize already puts the SelfDestructPath write into TxOut (via the worker IBS during exec), and ApplyVersionedWrites already applies it to the post-apply IBS in finalizeTxSimple — so the coinbase ends up correctly marked selfdestructed on that IBS, no ExecutionResult.SelfDestructedAddresses field needed. The actual gap was downstream:

1. The parallel worker runs with shouldDelayFeeCalc=true (ApplyMessageNoFeeBurnOrTip), so the priority fee is accumulated onto the version-map base in finalizeTxSimple but never lands on the post-apply IBS — which therefore handed LogSelfDestructedAccounts a selfdestructed coinbase with balance=0. GetRemovedAccountsWithBalance filters non-zero, so no Burn log was emitted.
2. Even when the Burn log would have been emitted, finalizeTxSimple never captured ibs.GetLogs(...) back onto result.Logs, so the log was stranded on the discarded post-apply IBS and the receipts root diverged from serial → BadBlock at validation.

Fix is two lines of logic in finalizeTxSimple (post-apply block):

• ibs.AddBalance(coinbase, FeeTipped, BalanceIncreaseRewardTransactionFee) + ibs.AddBalance(burnt, FeeBurnt, BalanceDecreaseGasBuy) after ApplyVersionedWrites. AddBalance leaves the selfdestruct flag intact (Selfdestruct only fires on the addr→clear transition, not on subsequent balance writes), so the account ends up selfdestructed with the priority fee as residual balance — exactly the state the serial path produces.
• result.Logs = append(result.Logs, ibs.GetLogs(...)...) after postApplyMessageFunc to surface the Burn log to the receipt.

Test passes in both serial and parallel modes. Other skipped tests in this issue (test_double_kill, dynamic_create2 collision, system_contract_deployment fork-transition, tipInsideBlock) are different bug classes and remain open.

[VBulikov]

yperbasis added Imp1 as Importance