rpc: finish atomic read-view acquisition and propagation

[yperbasis]

Context

#22533 defines the RPC read-view policy: block-only, head-sensitive reads may use one selected overlay generation, while reads that combine blocks with temporal state or history stay on one exact committed transaction. It also makes nested overlay wrapping idempotent.

#22987 adds the general pin carried by a transaction, atomic (database snapshot, overlay) acquisition, an explicit nil pin when no overlay is published, and propagation to gas-oracle worker transactions. It applies that mechanism to the fee-history and related fee/head endpoints.

After both PRs merge, the policy and primitive exist, but two adoption gaps remain:

1. Some overlay-intended handlers still open a database transaction and resolve the live overlay afterwards with WithOverlay, WithTemporalOverlay, LatestOverlay, or LatestSD. A publish, unpublish, or commit between those operations means the pair was not acquired atomically.
2. Composite requests can select a view and then lose it when a nested public handler or resolver opens another transaction. The response can combine different overlay generations or different canonical database snapshots.
3. A remote rpcdaemon without local block snapshots passes a database transaction to RemoteBlockReader, but transaction, canonical-hash, header-number, and block lookups issue independent ETHBACKEND requests. The Bor bridge lookup is independent as well. A commit or reorg can therefore make remote block identity disagree with temporal state read through the caller's transaction.

Remaining work

Audit and finish atomic acquisition

Inventory every production call to LatestSD, LatestOverlay, WithOverlay, and WithTemporalOverlay after rebasing both PRs. Classify each call site by policy rather than mechanically migrating all of them:

• An overlay-backed top-level handler should acquire its transaction through BeginTemporalRoWithOverlay (or an equivalent atomic helper) at the request boundary. Existing wrapper helpers may still accept a transaction, but must preserve its carrier and never re-resolve a live overlay.
• A committed state/history handler should keep its exact transaction and use the ...InView helpers from rpc: separate overlay-backed and committed read paths #22533. It must not be upgraded to an overlay merely because the new acquisition helper exists.
• A state-dependent latest handler that needs the in-flight state belongs to the coordinated SD-aware consumer work in Build SD-aware temporal view to remove FcuBackgroundCommit RPC plain-tx reverts #21314. Do not add SharedDomains to the generic database transaction layer here.

The audit should include the remaining block/transaction, Bor, debug, Erigon, receipt, and txpool paths, not only the fee endpoints migrated by #22987.

Known mixed-view consumers include:

• block, transaction-by-index, uncle, GraphQL, and Otterscan getters that resolve through one live overlay selection and fetch through another;
• blockAccessListBytes, which resolves through the overlay but reads the header and BAL bytes from the raw transaction;
• overlay_callConstructor, which independently selects views for transaction lookup, block fetch, the execution gate, and the state reader;
• receipt generator methods, which can replace the caller-selected transaction with the live temporal overlay and bind replay to the process-wide state cache. A receipt read from another generation may then receive the block identity supplied by the caller and enter an LRU keyed by transaction number or block hash;
• eth_callBundle, which resolves its state block through the overlay but reads cached or historical state from the raw transaction and reacquires the parent header separately;
• eth_createAccessList, whose latest header may come from the overlay while its cached state view is bound to the raw transaction;
• erigon_getBalanceChangesInBlock, which resolves and gates through the overlay but reads TxNums, account history, and state from the raw transaction;
• eth_callMany, which independently resolves its selector, execution gate, block, and state reader. Malformed and pending state-context validation is tracked separately by rpc: validate eth_callMany and debug_traceCallMany state contexts #23444.

Propagate the selected view through composite requests

Known cases:

• eth_fillTransaction pins its header and fee-default calculation in rpc: pin gas-oracle requests to one overlay view #22987, but GetTransactionCount and EstimateGas open their own views. The database nonce fallback, gas estimate, fee defaults, and final header can therefore come from different heads. Pending txpool data is intentionally live, but its database fallback still needs an explicit relationship to the request view.
• GraphQL blocks(from:, to: omitted) resolves the upper bound in one transaction, then calls GetBlockDetails once per height and opens a new transaction each time. A reorg during the loop can assemble a range from different canonical branches. This is also why rpc: separate overlay-backed and committed read paths #22533 keeps the GraphQL upper bound committed instead of exposing the overlay head.
• Any internal call from one public RPC method to another has the same failure mode if the callee performs its own acquisition.

Make the public handler or top-level resolver the acquisition boundary. Prefer internal ...InView operations that accept the selected transaction or pin. Use one transaction for sequential composite reads when possible. If another transaction is required, carry the same overlay or explicit nil pin and verify that its committed canonical snapshot is compatible, as the gas-oracle fork path in #22987 does. Retry or fall back to a single-view path when compatibility cannot be established.

Reuse OverlayViewCarrier, PinToOverlay, and the exact-view helpers from the two prerequisite PRs. Add another request-view abstraction only if it removes real duplication without hiding whether a path is overlay-backed or committed.

Bind remote block and transaction lookups to the request view

A remote rpcdaemon without local block snapshots cannot gain consistency merely by passing its kv.Tx to RemoteBlockReader: TxnLookup, CanonicalHash, HeaderNumber, and BlockWithSenders currently ignore that transaction and issue independent ETHBACKEND requests. The Bor bridge lookup is independent as well. This affects committed paths even when their RPC helpers use ...InView.

Define a cross-process contract that either carries a snapshot or version identity to backend reads, or validates returned transaction and block identities against canonical data from the caller's transaction before consuming state or history. Fetching immutable block content by a hash already validated in the caller's view may remain remote. Avoid call-site-specific validation that leaves other remote reader methods live.

Tests must open the rpcdaemon transaction, force a commit or reorg before the remote lookup, and verify that transaction lookup, canonical identity, block content, and the Bor fallback cannot advance past or switch away from the request view.

Acceptance criteria

• No overlay-policy top-level handler opens a raw transaction and then independently selects the live overlay; the transaction and overlay resolution are acquired atomically.
• Committed state/history handlers remain exact and cannot be upgraded by a nested helper.
• An already selected non-nil overlay generation remains stable through nested operations; an explicit nil pin remains committed if an overlay is published later.
• eth_fillTransaction uses one defined request view for its header, fee defaults, database nonce fallback, gas estimation, and transaction construction.
• GraphQL block ranges resolve their bound and every returned block from one canonical snapshot, or from separately opened views whose compatibility is verified.
• Internal RPC composition does not call another public handler that independently acquires a live view.
• Remote block and transaction lookups used by committed handlers are bound to, or validated against, the caller's canonical snapshot; the Bor fallback cannot select a different generation.
• Remote-mode tests force a commit or reorg between transaction acquisition and backend lookup and cover transaction lookup, canonical identity, block fetch after validation, and the Bor fallback.
• Deterministic tests force publish, unpublish, commit, and canonical changes between acquisition points and fail on the pre-fix behavior. Cover embedded and remote/no-overlay modes.
• The solution does not share an MDBX transaction across goroutines and does not add per-block acquisition or canonical-identity I/O to the normal GraphQL range path.

Dependencies

• rpc: separate overlay-backed and committed read paths #22533: endpoint policy, stable overlay reuse, and exact committed helpers.
• rpc: pin gas-oracle requests to one overlay view #22987: atomic acquisition, explicit nil pins, and cross-transaction pin propagation.
• Build SD-aware temporal view to remove FcuBackgroundCommit RPC plain-tx reverts #21314: coordinated SD-aware consumers for temporal state at the in-flight head.

Related issues

• execution/execmodule, node/shards: remove the global Events.LatestSD published-SD pointer (decouple readers via a publication id) #22494 proposes stable (db.sd) publication identities and explicit lifetimes; this work should remain compatible with that longer-term replacement for LatestSD.
• execution/execmodule: audit published-SD readers (Events.LatestSD, bg-commit) for shared-cache coherence #22214 audits cache coherence for readers using published SDs; pinning extends the lifetime over which that invariant matters.
• Investigate how much RPC↔exec state sharing to keep: STATE_CACHE_FILLS default, RPC cache reads, and the SharedDomains overlay #23082 evaluates whether embedded RPC should use the shared execution cache and overlay at all; its policy outcome may reduce or reshape this migration.
• debug_traceBlockByNumber panics on the tip: "assert: block.GasUsed() != gasUsed" — latest resolved on the overlay, replayed on the committed view #22969 is a direct mixed-view symptom in tracing and is addressed by rpc: separate overlay-backed and committed read paths #22533.
• race condition in new headers subscription #18138 is the historical newHeads to eth_feeHistory visibility race that rpc: pin gas-oracle requests to one overlay view #22987 addresses at the acquisition layer.

[VBulikov]

yperbasis added Imp2 as Importance

[VBulikov]

yperbasis changed Importance from Imp2 to Imp1