Found on main (e44109d) while writing integration tests for #21860 (see PR referencing this issue).
Edited: the original report blamed the txpool ("reorg processing re-injects unwound transactions into the pending pool"). Instrumenting the repro disproved that — the pool stays empty and consistent throughout. The real problem is a stale cached pending block in the RPC layer; the description below has been updated accordingly.
What happens
A node builds a payload once (block 1 during test setup) and then follows the chain purely via newPayload/forkchoiceUpdated — in the test it is bounced between two competing forks, but the bounce is incidental. The head state is fully settled — eth_getStorageAt/eth_call reads at latest are all correct.
Submitting a new transaction through the standard bind flow (nonce from the backend, i.e. pending eth_getTransactionCount) then fails:
INTERNAL_ERROR: nonce too low
and keeps failing on every retry for 30+ seconds — this is not the usual transient post-reorg lag; the pending-nonce view stays durably inconsistent with the head state.
Root cause
rpchelper.Filters cache the last pending block broadcast by the block builder (HandlePendingBlock) and never invalidate it. eth_getTransactionCount(pending) first asks the txpool; when the pool has no entries for the sender it falls through to a state read at the block resolved by _GetBlockNumber(pending), which returns filters.LastPendingBlock() — the payload built long ago:
DBG GetTransactionCount pending: pool miss, falling through to state read
DBG pending->LastPendingBlock num=1 plainState=14
DBG GetTransactionCount state read nonce=0
DBG validateTx NonceTooLow senderNonce=13 txnNonce=0
So the pending nonce is read from block 1's state (nonce 0) while the head is at 14 (nonce 13). The submission built on that nonce is correctly rejected by the pool with nonce too low, and the retry loop keeps re-reading the same stale value. The wedge is durable because the cached pending block only refreshes when this node builds another payload — which, in the bind flow, first requires a successful submission.
The txpool behaves correctly in this scenario: the unwound fork's transactions are delivered on each reorg and discarded as NonceTooLow against the post-reorg state (both forks confirm the same nonces), every OnNewBlock completes without error, and the pool ends empty — which is exactly why the pending-nonce query falls through to the stale pending-block read.
Beyond the test, this hits any node that builds a payload occasionally — e.g. an EL behind a solo-staking validator: after its proposal slot, every pending-tagged RPC read (eth_getTransactionCount, eth_call, eth_getBlockByNumber, gas oracle) stays pinned to that block's height until the next proposal.
Reproduction
TestEngineApiForkBounceStateChurn in execution/engineapi/engine_api_state_churn_reorg_test.go works around this by pinning the nonce. Remove the workaround in churnAndAssert:
nonce, err := eat.RpcApiClient.GetTransactionCount(coinbaseAddr, rpc.LatestBlock)
require.NoError(t, err)
transactOpts.Nonce = nonce
(delete these lines so bind falls back to the backend-provided pending nonce) and the test times out after 30s of retries: poke submission did not settle.
Fix
#22326 invalidates the cached pending block in Filters.onNewHeader once the chain moves past it (a header at or above its height, or a competing block replacing its parent); pending reads then fall back to latest-executed state, matching what other clients serve when no block is being built.
Workaround
Set the nonce explicitly from eth_getTransactionCount(latest) instead of relying on the pending nonce.
Found on
main(e44109d) while writing integration tests for #21860 (see PR referencing this issue).Edited: the original report blamed the txpool ("reorg processing re-injects unwound transactions into the pending pool"). Instrumenting the repro disproved that — the pool stays empty and consistent throughout. The real problem is a stale cached pending block in the RPC layer; the description below has been updated accordingly.
What happens
A node builds a payload once (block 1 during test setup) and then follows the chain purely via
newPayload/forkchoiceUpdated— in the test it is bounced between two competing forks, but the bounce is incidental. The head state is fully settled —eth_getStorageAt/eth_callreads atlatestare all correct.Submitting a new transaction through the standard bind flow (nonce from the backend, i.e. pending
eth_getTransactionCount) then fails:and keeps failing on every retry for 30+ seconds — this is not the usual transient post-reorg lag; the pending-nonce view stays durably inconsistent with the head state.
Root cause
rpchelper.Filterscache the last pending block broadcast by the block builder (HandlePendingBlock) and never invalidate it.eth_getTransactionCount(pending)first asks the txpool; when the pool has no entries for the sender it falls through to a state read at the block resolved by_GetBlockNumber(pending), which returnsfilters.LastPendingBlock()— the payload built long ago:So the pending nonce is read from block 1's state (nonce 0) while the head is at 14 (nonce 13). The submission built on that nonce is correctly rejected by the pool with
nonce too low, and the retry loop keeps re-reading the same stale value. The wedge is durable because the cached pending block only refreshes when this node builds another payload — which, in the bind flow, first requires a successful submission.The txpool behaves correctly in this scenario: the unwound fork's transactions are delivered on each reorg and discarded as
NonceTooLowagainst the post-reorg state (both forks confirm the same nonces), everyOnNewBlockcompletes without error, and the pool ends empty — which is exactly why the pending-nonce query falls through to the stale pending-block read.Beyond the test, this hits any node that builds a payload occasionally — e.g. an EL behind a solo-staking validator: after its proposal slot, every
pending-tagged RPC read (eth_getTransactionCount,eth_call,eth_getBlockByNumber, gas oracle) stays pinned to that block's height until the next proposal.Reproduction
TestEngineApiForkBounceStateChurninexecution/engineapi/engine_api_state_churn_reorg_test.goworks around this by pinning the nonce. Remove the workaround inchurnAndAssert:(delete these lines so bind falls back to the backend-provided pending nonce) and the test times out after 30s of retries:
poke submission did not settle.Fix
#22326 invalidates the cached pending block in
Filters.onNewHeaderonce the chain moves past it (a header at or above its height, or a competing block replacing its parent); pending reads then fall back to latest-executed state, matching what other clients serve when no block is being built.Workaround
Set the nonce explicitly from
eth_getTransactionCount(latest)instead of relying on the pending nonce.