stagedsync: Fix logIndex reset and missing websocket notifications in parallel execution

[Sahil-4555]

Issue

When Erigon is doing parallel block execution (EXEC3_PARALLEL=true), the state snapshots can end at step boundaries which fall mid-block. So when we restart the node or resume sync from these snapshots, the execution stage needs to start/resume execution from the middle of the block (running only the remaining transactions of that block).

While finalizing the transaction receipts for this partial block execution, the code in exec3_parallel.go was checking if txVersion.TxIndex > 0 && tx > 0 to fetch the previous transaction's receipt from the memory results map. But for the first transaction task of this new batch (where the local slice task index tx is 0, but the actual block-level index txVersion.TxIndex is > 0), this check was failing and prevReceipt remained nil. Because of this, it called CreateNextReceipt(nil) and the starting log index and cumulative gas for that mid-block transaction got reset to 0. This wrong log index got written directly to the database ReceiptDomain and caused inconsistent logIndex values when clients queried eth_getLogs.

Additionally, for partial blocks the websocket notification path was publishing tail-only receipts (only the resumed portion) via RecentReceipts.Add, which eth_subscribe log/receipt clients would receive as the complete block — diverging from eth_getLogs results.

Fix

The main fix is in the parallel executor's receipt finalization loop in exec3_parallel.go. We changed the finalize function signature to accept cumulativeGasUsed and firstLogIndex directly instead of a prevReceipt pointer, and it now calls CreateReceipt instead of CreateNextReceipt. This matches how the serial executor handles receipt creation. We also branch explicitly on tx == 0 (batch boundary) instead of checking prevReceipt == nil. When the first task in a batch has TxIndex > 0 (meaning we are resuming mid-block), we query the temporal database usingrawtemporaldb.ReceiptAsOf to fetch the previous transaction's cumulative gas used, cumulative blob gas used, and log index offset. We excluded IsBlockEnd tasks from this fallback since finalizeSystemTx never consumes prevReceipt anyway. Also added a two-step nil check on be.finalizedResults[tx-1] before accessing .Receipt since finalizedResults is a map and a missing key would return nil and cause a panic.

For the notification issue, we gated RecentReceipts.Add behind !applyResult.isPartial and moved it outside the block validation scope so that incomplete tail-only receipts are not published for partial blocks. We now use applyResult.Txs and applyResult.Header directly instead of fetching from the database. This is consistent with how the serial path gates behind startTxIndex == 0 in exec3_serial.go:421.

For the blob gas issue, we added a cumulativeBlobGasUsed field to execResult and snapshot be.blobGasUsed into it at finalization time. The publish loop now uses this per-result snapshot instead of reading be.blobGasUsed directly, which could have been overwritten by a later transaction. We also initialize be.blobGasUsed from the database cumBlobGasUsed value at the resume boundary.

Added TestParallelResumeBoundaryAndNotifications unit test to pin theresume boundary receipt reconstruction (first task with TxIndex > 0 at slice index 0 → correct offsets from DB) and the isPartial flag that controls notification skipping.

Closes: #22106

[Copilot]

Pull request overview

This PR fixes receipt log-index continuity and websocket log notification dispatch in the staged sync parallel execution path, addressing incorrect logIndex values observed by eth_getLogs when execution resumes mid-block across batches/restarts.

Changes:

• Fix receipt finalization when a batch starts mid-block by loading prior receipt state from ReceiptDomain via rawtemporaldb.ReceiptAsOf.
• Adjust receipt notification enqueue logic to (intended) allow websocket log subscribers to receive notifications when a block completes even if execution started in a partial batch.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

[sudeepdino008]

like when the batch limit is reached mid-block, or when we restart the node and resume sync

we don't "execute blocks incompletely". State snapshots can end at midblock however and so execution needs to resume from middle of the block.

[sudeepdino008]

one weird thing:

• prevReceipt is usually coming from current batch's execution, so it have logs.
• when we use prevReceipt to construct next receipt, we do currentReceipt.FirstLogIndexWithinBlock = prevReceipt.FirstLogIndexWithinBlock + len(logs)
• in edge case when we retrieve from db, receipt has no logs (rcache has it, receipt doesn't). Also, prevReceipt created with no logs.
• Also, in this case, logIndexAfterTx retrieved from db is actually the index after adding logs...

but it works out in end
prevReceipt.FirstLogIndexWithinBlock = logIndexAfterTx. <-wrong assignment
but
currentReceipt.FirstLogIndexWithinBlock = prevReceipt.FirstLogIndexWithinBlock + len(logs)

logs is empty, so currentReceipt.FirstLogIndexWithinBlock gets set with the right value..

I'd suggest adding a comment about this

---

• exec3_serial did it properly, by re-executing the prev tx and getting logs etc. But seems we don't need it.
• I wanna get rid of the receipt version check, but bloatnet still has it. I think bloatnet is dead, but if it isn't - i'll regen the receipt and get rid of this check.

[Sahil-4555]

You are totally right that prevReceipt.FirstLogIndexWithinBlock = logIndexAfterTx isn’t technically the previous starting index, but since we set Logs = nil (so len(Logs) == 0) the calculation logIndexAfterTx + 0 gives the starting index for the next transaction. This allows us to reuse CreateNextReceipt directly, without extra DB fetching or re-executing the previous transaction.

I’ve added a clear code comment explaining this mathematical trick. Thanks for catching this!

[yperbasis]

The receipt-index reconstruction itself checks out, but a few things need fixing before merge:

1. Partial blocks now notify with tail-only receipts (exec3_parallel.go:579). For an isPartial block, blockResult.Receipts contains only the resumed tail (built from be.results; the task producer skips the prefix), while Txs/Header are the full block's. RecentReceipts.Add stores this as the block's complete set, so eth_subscribe log/receipt clients get the tail presented as the whole block — diverging from eth_getLogs, replaying Removed on unwind for the tail only, and bypassing the !isPartial-gated receipts-root/bloom validation. Please either reconstruct the prefix (receipts.DerivePriorReceipts, as the serial path does for Finalize) or keep skipping partial blocks like serial does (startTxIndex == 0, exec3_serial.go:421); publishing an incomplete set is worse than either.

2. The resume-at-block-end offset still gets zero notifications (exec3_parallel.go:2505). If the boundary falls between the last user tx and the block-end txNum, the resumed executor's only task is block-end: Receipts is empty, Add early-returns, and the completed block emits newHeads but no logs/receipts — the #22106 symptom remains. The fallback also runs for block-end tasks even though finalizeSystemTx never consumes prevReceipt; please exclude IsBlockEnd tasks.

3. The legacy-schema check can never fire (exec3_parallel.go:2506). ReceiptStoresFirstLogIdx tests CurrentDomainVersion < v1.1, but the exec stage only ever sees {3,0} (schema default) or {1,1} (AdjustReceiptCurrentVersionIfNeeded pins legacy datadirs to exactly v1.1), so the guard is dead code — while 3.0-written data tails still pass through and get exactly the shifted indexes the error message claims to prevent. The underlying problem is the predicate threshold (arguably Less(V2_0); dates to #16677) — better to fix that than ship a check that suggests protection it doesn't provide.

4. be.blobGasUsed = cumBlobGasUsed seeds from a slot the parallel executor writes per-tx, not cumulative (exec3_parallel.go:2525): the publish loop passes tx.GetBlobGas() as ApplyTxIndexes' cummulativeBlobGas arg (exec3_parallel.go:2740), unlike serial which passes the running cumulative (exec3_serial.go:556). On parallel-synced datadirs the seed is the previous tx's own blob gas (0 for non-blob txs). Latent today because the validator is skipped for partial blocks, but wrong — fix the write side to store the running cumulative, or drop the seed.

5. Shape: branch explicitly on tx == 0 instead of prevReceipt == nil (the nil case is otherwise impossible; if that invariant ever breaks, this silently reconstructs from pre-batch DB state instead of failing loudly), and consider passing the two scalars through finalize into the existing CreateReceipt like serial does (exec3_serial.go:452-461) — finalize has a single call site — rather than a synthetic receipt whose FirstLogIndexWithinBlock holds the next tx's index. That also removes the need for the 7-line comment (which restates CreateNextReceipt's math and already drifts from it); the // Initialize cumulative blob gas... comment just restates the assignment and can go.

6. Per repo guidelines (CLAUDE.md, TDD for bug fixes), please add a test pinning the resume boundary (first task with TxIndex > 0 at slice index 0 → reconstructed receipt values) and the notification behavior — nothing currently prevents a regression of the #22106 corruption.

[Sahil-4555]

Hi @yperbasis,

Thanks for the detailed feedback. I have gone through all your points and done the needful changes. Please find the updates below:

1. Partial block notifications: We are now skipping notifications for partial blocks completely, same like how the serial execution path is doing. We check !applyResult.isPartial before calling RecentReceipts.Add now.
2. Excluding block-end: Excluded the IsBlockEnd tasks from the fallback database query since finalize system tx does not need it anyway.
3. Dead code removal: Removed the legacy-schema check completely as it was dead code and not firing.
4. Blob gas seed: We initialized be.blobGasUsed from the db cumBlobGasUsed as per the comment.
5. Code shape: We changed the branching logic to check tx > 0 directly instead of checking if receipt is nil. Also, we are passing the offset values directly into CreateReceipt via finalize now, and cleaned up those long comments.
6. Unit test: Added TestParallelResumeBoundaryAndNotifications unit test to verify this boundary fallback and partial block notification skipping. All tests are passing fine.

Please review and let me know if it looks good now.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 5 comments.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.