flaky test: `TestExecutePayloadSuccess`

[sebastianst]

Flaky Test

Test: TestExecutePayloadSuccess
Package: rust/op-reth/tests/proofs/core
CI Job: op-reth-e2e-sysgo-tests

CI Runs

• https://app.circleci.com/pipelines/gh/ethereum-optimism/optimism/121652/workflows/c7a2af02-affc-4d0d-bd43-461f58d6a244/jobs/4731572
• Failed 2 times in a row with the same error

Error

execute_payload_test.go:61: no state found for block number 1

Context

The test sets up a system, funds an account (tx confirmed at block 1), then immediately queries op-reth for the state of block 1. op-reth doesn't have the state ready yet — likely a race condition between block production and state availability in the execution client.

Two other proof tests in the same package pass consistently (TestL2MultipleTransactionsInDifferentBlocks, TestL2MultipleTransactionsInSingleBlock).

Encountered while working on #19881.

🤖 Generated by Claude Code

[sebastianst]

🤖 Claude says:

Root Cause Analysis

The debug_executePayload RPC on op-reth uses OpStateProviderFactory, which reads from a proofs ExEx store — an async execution extension that processes ChainCommitted notifications after block commit. The test queries the validator immediately after funding completes on the sequencer, but the validator's proofs ExEx may not have indexed block 1 yet, producing "no state found for block number 1".

The race window is between the validator updating its unsafe head and the ExEx processing the ChainCommitted notification. Normally sub-millisecond, but CI resource contention widens it.

Fix

PR: #19899 — adds WaitForBlockNumber synchronization between sequencer and validator, matching the pattern used by other non-flaky tests in the same package (TestDebugExecutionWitness, TestStorageProofUsingSimpleStorageContract).

🤖 Generated by Claude Code

[wwared]

Claude: Reopening — this flake has resurfaced.

CI run: https://app.circleci.com/pipelines/github/ethereum-optimism/optimism/122137/workflows/ee0edca5-4492-454a-a36f-63a6af363381/jobs/4767278
Job: op-reth-e2e-sysgo-tests
Date: 2026-04-08
Error: execute_payload_test.go:70: no state found for block number 1
Branch: gh-readonly-queue/develop/pr-19953

This was encountered while working on #19953.

[sebastianst]

🤖 Claude says:

Updated Root Cause Analysis

The previous fix (PR #19899, WaitForBlockNumber) was insufficient. WaitForBlockNumber only checks the validator's EL head, but debug_executePayload reads from the proofs ExEx store (OpStateProviderFactory), which is populated asynchronously via ChainCommitted notifications.

CI evidence from the 2026-04-08 failure confirms: block reached at 02:09:14.715, test failed at 02:09:14.721 — the EL head advanced but the ExEx store hadn't indexed the block yet.

New Fix

PR #19986 adds utils.WaitForProofsStoreBlock which polls debug_proofsSyncStatus (reads from the same Arc<MdbxProofsStorage> backing OpStateProviderFactory) until the store has indexed the target block. This completely eliminates the race.

🤖 Generated by Claude Code