rpc/jsonrpc: eager in-memory cache for `debug_executionWitness` by awskii · Pull Request #22384 · erigontech/erigon

awskii · 2026-07-10T13:36:42Z

debug_executionWitness rebuilds the full witness on every call — it re-executes the block and re-folds the commitment trie, ~1–3s per witness on a mainnet-archive node (longer for deep history). Tip-following consumers (provers, stateless verifiers) request witnesses for recent blocks as they arrive, so the same block is rebuilt on every request.

An opt-in in-memory cache of recent witnesses makes a repeat request a verbatim serve. Measured on a mainnet-archive node, same ~15MB witness: cache hit ~1.2ms ttfb / ~4.7ms total, vs reth ~292ms / ~474ms and the on-demand ~2.8s.

Changes

Add an opt-in cache of the last N legacy-mode witnesses, keyed by canonical block hash (an lru.Cache). A node-side goroutine subscribes to new canonical headers and builds each block's witness right after it commits, through the same code path the RPC handler uses, so a cached witness is byte-identical to the on-demand one. A legacy request for a cached block is served from memory; canonical mode, a miss, or a reorged hash falls through to the existing path.
The builder marshals the witness JSON once, off the serve path, and stores it; a hit is served verbatim via rpc.fastJSONResult (MarshalFastJSON), skipping the per-hit struct marshal (~26ms for a 15MB witness).
Embedded-RPC only — the builder and the handler must share one process. Standalone rpcdaemon passes a nil cache and is unchanged. Requires --prune.experimental.include-commitment-history.
One flag: --witness.cache.blocks (0 = off, clamped to a max of 96). Reorg eviction is implicit: keying by hash means number-based lookups resolve to the current canonical hash and orphaned hashes age out of the LRU.

Pure refactor splitting the witness-building pipeline (buildAccessedState through the append-and-sort tail) into a shared DebugAPIImpl.buildWitnessResult method, so the on-demand handler and the upcoming eager cache builder produce byte-identical results. Adds a determinism-and-sort guard test.

…y hit

Background worker that eagerly builds legacy-mode debug_executionWitness results into the shared witnessCache as canonical headers arrive, reusing the buildWitnessResult seam so cached bytes are byte-identical to on-demand. - shouldBuild: pure tip-gate (single-block advance to the freshest unbuilt tip) - decideCommittedHead / waitCommittedHead: Fork-1 safe-commit gate that polls the committed head via a fresh temporal RO tx per attempt, matching the hash before building and treating a mismatch as a reorg-away - RunWitnessCacheBuilder: coalesce-to-latest loop that reconciles the cache on every batch and builds only the newest tip-gated header

…uilder Add witness.cache.blocks (default 0, capped 96) and witness.cache.maxmb (default 1024) flags, thread them through HttpCfg, and wire the eager witness-cache builder into the embedded node. The cache is embedded-RPC only and gated on the DB-persisted commitment-history flag; standalone rpcdaemon and mcp pass nil. APIList gains the shared *witnessCache param; the builder-owned DebugAPIImpl shares that same pointer.

rpc/jsonrpc: add witness_cache_* counters (hit/miss, build_ok, build_fail_verify/other, evict, coalesce_drop), bytes/entries_resident gauges, and a build_duration histogram, wired at the serve, build, and eviction sites. Classify verify failures via an errWitnessVerifyFailed sentinel wrapped around the shared build seam. Enrich the witness cache flag help text with the raw-size cap context.

rpc/jsonrpc: builder resolves the block by the canonical hash it validated, not by number, so the built witness is provably keyed under (num, hash). Tests: fix canonical-bypass subtest that could never fail (discarded error + nil-safe assertion); assert hit/miss counter deltas on serve; cover newWitnessCache clamp/boundary and decodeHeaderRefs/processHeaderBatch newest-selection; join the builder goroutine before DB teardown in builder tests to avoid a use-after-close race under -race. docs/plans: correct RunWitnessCacheBuilder signature (drop notifications arg).

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Store the marshaled JSON once in the builder and serve a cache hit through MarshalFastJSON, skipping the per-hit struct marshal (~26ms for a 15MB witness -> ~11ns, zero allocs). Byte-identical to the on-demand response.

Copilot

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 3 comments.

Replace the hand-rolled number-keyed witness cache with the same hashicorp lru.Cache used for the block cache, keyed by block hash. Hash keying makes reorgs self-evicting — a reorged hash is never requested again and ages out — so the reconcile path is gone. Memory is bounded by the block count; drops the byte cap and --witness.cache.maxmb.

…U library

Gate eager building on whether the tip's hash is already cached rather than a high-water block number, so a reorged head (a new hash at an already-built height) is rebuilt instead of falling through to on-demand forever. Removes the frozen high-water var.

Conflicts resolved: - node/eth/backend.go: combine the eager witness-cache setup (this branch) with the MCP streamable-HTTP refactor from main (#22624). Keep the witness-cache block and main's shared-APIList/apisForNamespaces + MCP server block; the single APIList call now passes both trailing params (testingEntry from main, witnessCache from this branch). - cmd/mcp/main.go: main rewrote the MCP path to serve via an in-process rpc.Server + DialInProc (#22624), superseding the ethAPI/erigonAPI/otsAPI extraction on this branch. Kept main's architecture and bumped the APIList call to the merged 14-arg signature (nil, nil). - rpc/jsonrpc/witness_cache_builder.go (non-conflict, merged clean but stale against main): db/services -> db/dbservices package rename, and NewBaseApi's positional args consolidated into NewBaseApiConfig(cfg); engine param widened to rules.Engine to match.

The startup log printed the raw --witness.cache.blocks value; the cache clamps it to witnessCacheMaxBlocks (96), so a larger flag value misreported the real capacity. Centralize the clamp in exported WitnessCacheCapacity and log that.

Copilot

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 1 comment.

Lint (after the db/services -> db/dbservices rename in the main merge): gofmt witness_cache_builder import ordering, modernize the clamp test loop to range-over-int. Metric fidelity (code-review): - record the build-duration histogram only on successful builds, not failures - don't count a cache miss when the request's block can't be resolved

ExecutionWitnessResult has no MarshalJSON, so encoding/json never dispatches to MarshalFastJSON — there is no recursion for executionWitnessResultView to guard against. Marshal the pointer directly, avoiding both the view type and the per-call struct copy.

Copilot

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

rpc/jsonrpc/witness_cache_builder.go:183

In the coalescing drain, freshest is updated using max(freshest, n2.num). During an unwind/reorg, header notifications can move the tip backwards (see execution/execmodule/notification_dispatcher.go:104-116 where notifyFrom can start from prevUnwindPoint+1). If an earlier queued batch had a higher block number than the later (reorged) batch, freshest stays at the earlier high-water value and shouldBuild(newest.num == freshest) becomes false, so the builder can skip eagerly caching the new canonical (lower) tip.

						freshest = max(freshest, n2.num)

Resolve APIList conflict from #22680 (rpc: extract api config pkg more): adopt main's config-object constructors NewPrivateDebugAPI(...DebugApiConfig) and NewTraceAPI(...TraceApiConfig), keeping debugImpl.witnessCache wiring. Update the witness builder + its tests to the new NewPrivateDebugAPI signature.

Copilot

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 1 comment.

#22384 ("rpc/jsonrpc: eager in-memory cache for debug_executionWitness") registered utils.WitnessCacheBlocksFlag in node/cli/default_flags.go earlier today, so the CLI reference was one flag behind again. It is operator-facing — it trades memory for debug_executionWitness latency on RPC nodes — so it belongs in the reference per the flag-coverage rule. Documented from the source Value/Usage: default 0 (cache disabled), clamped at 96, embedded RPC only, and dependent on commitment history. The Usage string names the --prune.experimental.include-commitment-history alias; the docs use the canonical --prune.include-commitment-history instead. After this, the Step-4c sweep on main reports 10 undocumented registered flags — the intentionally-undocumented internal/dev baseline shared with the other live branches. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

… flag, disk sizes (erigontech#22799) The `main` side of this week's documentation maintenance (w31). Carries the same stale-flag cleanup as erigontech#22793 (`release/3.5`) and erigontech#22798 (`release/3.6`) under the dual-commit rule, plus three findings that apply only to `main`. ## 1. Flags removed from the code but still documented Identical to erigontech#22793 — see that PR for the per-family provenance: * `--p2p.allowed-ports` — 8 occurrences / 4 files, removed by erigontech#21335 *"share one p2p.Server across all eth protocols"*; a single `--port` now suffices. Also fixes the `--P2P.allowed-ports` casing bug. * `--clique.checkpoint` / `.snapshots` / `.signatures` / `.datadir` — clique is gone from the tree. * `--diagnostics.*` — last present on `release/3.3`, already absent from `release/3.4`. * `--polygon.sync` — removed by erigontech#16035. * `--rpc.maxgetproofrewindblockcount.limit` — an internal config field with no CLI registration. ## 2. Wrong information corrected Also identical to erigontech#22793: Beacon API timeouts are bare integer seconds (`25`, not `25s` — they are `cli.Uint64Flag`, so `25s` does not parse); the pre-erigontech#21335 two-listener port model removed from `default-ports.md` and `multiple-instances.md`; `--p2p.protocol` default is `69, 70, 71`; `--maxpeers` default is 64; the Ethereum on ARM link now points at the canonical `EOA-Blockchain-Labs/ethereumonarm`; one unmatched parenthesis. ## 3. `--db.read.concurrency` semantics — `main` only erigontech#22408 *"node, commitment: fix parallel exec deadlock on many-core machines"* rewrote this flag's behaviour, and three pages said the opposite of what the code does. Each parallel-execution worker holds a long-lived read transaction, so a ceiling below the worker count would deadlock. The value is silently **raised** to the worker count, and lowering the flag does not reduce read concurrency at all — `--exec.workers` is the knob for that. The docs advised the reverse: *"Low values are fine for low read-concurrency nodes (for example, validators)"*. Verified against source rather than the commit message: the clamp is `cmd/utils/flags.go:2041` (`RoTxsLimit(c, cfg.ExecWorkerCount)`), and the existing *"HTTP/WebSocket fail fast with an overload response"* wording is **still accurate** (`rpc/http.go:241`, `httpOverloadedKey` / `kv.ErrReadTxLimitExceeded`), so it is kept rather than dropped along with the rest. Does not apply to `release/3.5` or `release/3.6`, which predate erigontech#22408. ## 4. `--witness.cache.blocks` — new today erigontech#22384 *"rpc/jsonrpc: eager in-memory cache for `debug_executionWitness`"* registered `utils.WitnessCacheBlocksFlag` earlier today, leaving the CLI reference one flag behind. It trades memory for `debug_executionWitness` latency, so it is operator-facing and in scope per the flag-coverage rule. Documented from the source `Value`/`Usage`: default `0` (disabled), clamped at `96`, embedded RPC only, requires commitment history. The source `Usage` names the `--prune.experimental.include-commitment-history` alias; the docs use the canonical `--prune.include-commitment-history`. ## 5. Disk sizes `disk-sizes.json` is brought in line with `release/3.5` (mainnet and gnosis, all three modes, measured 2026-07-19 / 2026-07-21) and the static markers in `hardware-requirements.mdx` re-rendered with `render-disk-sizes.py`, so the live branches carry one consistent set of numbers. ## Verification * `npm ci && npm run build` — green (`onBrokenLinks` and `onBrokenAnchors` both `throw`) * `generate-llms.py --check` — OK, 4 files, 74 pages * `render-disk-sizes.py --check` — OK * Step-4c bidirectional sweep — **233 registered, 10 undocumented**, i.e. the intentionally-undocumented internal/dev baseline, with no stale flags remaining * No `allowed-ports` / `diagnostics.` / `clique.` / `beacon.api.ide` / `polygon.sync` / `maxgetproofrewind` / `eth/68` references left in `docs/site/docs` --------- Co-authored-by: Bloxster <gianni.morselli@erigon.tech> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Alex Sharov <AskAlexSharov@gmail.com>

… on minimal nodes via head-capture (erigontech#22663) A `--prune.mode=minimal` node keeps account/storage/code history but no commitment history, so the eager witness cache and `debug_executionWitness` are both hard-gated off — a minimal node can't serve execution witnesses at all. This adds a head-capture mode that builds each head block's witness with commitment parent state read from an RO snapshot pinned at the parent (where commitment-latest already equals the parent trie) and plain state from the history a minimal node keeps. No commitment history is required or produced. ## Changes - `--witness.cache.head-capture`: build the last-N head witnesses against a rolling one-block-lag pinned parent snapshot, using a dual-tx reader (commitment-latest from the pinned parent tx and plain history from the committed tx). No code on the consensus commitment path. - Serve cache-only: typed out-of-window on miss (never a history recompute); by-hash requests are canonical-checked so a reorged-out orphan is never served as canonical. - Fail-closed: `witnessRoot==parent`, `computedRoot==block`, and stateless verify gate every cache insert. - Cache is now a struct with a resident-bytes cap (`--witness.cache.maxmb`). Tip-only and cache-only by design: after restart the cache is empty and re-warms forward, so the last ~N blocks are out-of-window until N new blocks pass. Stacked on erigontech#22384. --------- Co-authored-by: Alexey Sharov <askalexsharov@gmail.com>

awskii added 9 commits July 10, 2026 18:45

docs/plans: add eager debug_executionWitness witness-cache plan

06a9e12

rpc/jsonrpc: add bounded witnessCache type for eager witness cache

db45f2e

rpc/jsonrpc: serve debug_executionWitness from witness cache on legac…

1fe3e3c

…y hit

move completed plan: 20260710-witness-cache.md

bf99370

awskii requested review from AskAlexSharov, bloxster, lupin012 and yperbasis as code owners July 10, 2026 13:36

awskii requested a review from Copilot July 10, 2026 13:37

Copilot AI reviewed Jul 10, 2026

rpc/jsonrpc: serve cached witness verbatim via fastJSONResult

fbfd814

Store the marshaled JSON once in the builder and serve a cache hit through MarshalFastJSON, skipping the per-hit struct marshal (~26ms for a 15MB witness -> ~11ns, zero allocs). Byte-identical to the on-demand response.

yperbasis requested a review from Copilot July 10, 2026 15:03

yperbasis added the RPC label Jul 10, 2026

yperbasis added this to the 3.6.0 milestone Jul 10, 2026

Copilot started reviewing on behalf of yperbasis July 10, 2026 15:04 View session

Copilot AI reviewed Jul 10, 2026

View reviewed changes

Comment thread rpc/jsonrpc/witness_cache_builder.go Outdated

Comment thread docs/plans/completed/20260710-witness-cache.md Outdated

Comment thread rpc/jsonrpc/witness_cache.go Outdated

AskAlexSharov reviewed Jul 11, 2026

View reviewed changes

Comment thread rpc/jsonrpc/witness_cache.go Outdated

awskii added 4 commits July 12, 2026 01:40

docs/plans: remove completed witness-cache plan

8897ad6

rpc/jsonrpc: drop witness-cache unit tests that only exercised the LR…

2e82ec2

…U library

awskii mentioned this pull request Jul 12, 2026

rpc/jsonrpc: push fresh witnesses via debug_subscribe("executionWitnesses") #22407

Merged

AskAlexSharov requested a review from Copilot July 12, 2026 11:30

Copilot started reviewing on behalf of AskAlexSharov July 12, 2026 11:31 View session

AskAlexSharov approved these changes Jul 12, 2026

View reviewed changes

awskii mentioned this pull request Jul 22, 2026

rpc/jsonrpc, node, execution/commitment: serve debug_executionWitness on minimal nodes via head-capture #22663

Merged

AskAlexSharov approved these changes Jul 23, 2026

View reviewed changes

AskAlexSharov added this pull request to the merge queue Jul 23, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jul 23, 2026

AskAlexSharov added 2 commits July 24, 2026 14:12

AskAlexSharov requested a review from Copilot July 24, 2026 07:26

Copilot started reviewing on behalf of AskAlexSharov July 24, 2026 07:26 View session

Copilot AI reviewed Jul 24, 2026

View reviewed changes

Comment thread rpc/jsonrpc/debug_execution_witness.go Outdated

AskAlexSharov force-pushed the awskii/witness-cache branch from 169b5ab to b055c17 Compare July 24, 2026 07:36

AskAlexSharov requested a review from Copilot July 24, 2026 08:13

Copilot started reviewing on behalf of AskAlexSharov July 24, 2026 08:13 View session

Copilot AI reviewed Jul 24, 2026

View reviewed changes

AskAlexSharov requested a review from Copilot July 25, 2026 01:03

Copilot started reviewing on behalf of AskAlexSharov July 25, 2026 01:03 View session

Copilot AI reviewed Jul 25, 2026

View reviewed changes

Comment thread rpc/jsonrpc/witness_cache.go Outdated

awskii added 2 commits July 27, 2026 14:40

Merge remote-tracking branch 'origin/main' into awskii/witness-cache

4aca1d3

rpc/jsonrpc: correct witness-cache reorg-eviction comment

56d15a5

awskii changed the title ~~rpc/jsonrpc: eager in-memory cache for debug_executionWitness~~ rpc/jsonrpc: eager in-memory cache for debug_executionWitness Jul 27, 2026

AskAlexSharov added this pull request to the merge queue Jul 28, 2026

Merged via the queue into main with commit bc479d5 Jul 28, 2026
95 checks passed

AskAlexSharov deleted the awskii/witness-cache branch July 28, 2026 09:18

bloxster mentioned this pull request Jul 28, 2026

docs: w31 maintenance on main — stale flags, db.read.concurrency, new flag, disk sizes #22799

Merged

Navigation Menu

rpc/jsonrpc: eager in-memory cache for `debug_executionWitness` - #22384

rpc/jsonrpc: eager in-memory cache for `debug_executionWitness`#22384
AskAlexSharov merged 21 commits into
mainfrom
awskii/witness-cache

awskii commented Jul 10, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

awskii commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

awskii commented Jul 10, 2026 •

edited

Loading