debug: add debug_executionWitness spec

[MysticRyuujin]

Concrete spec proposal for #845, opened as a draft to anchor the discussion on the shape.

Specs debug_executionWitness(BlockNumberOrTagOrHash) returning the canonical execution witness, {state, codes, headers} with RLP-encoded headers. This matches the execution-specs stateless validation interface (stateless.py ExecutionWitness) and the SSZ container proposed in #773. The witness semantics are owned by execution-specs; this is the JSON-RPC binding.

Positions encoded here (rationale in #845):

• headers as RLP bytes; keys is optional. It is not part of the canonical format, but clients may include the accessed state keys for prover-side use (discussion). Semantics pinned per canepat's answers: reads and writes, one flat array of 20-byte addresses and 32-byte slot keys, deduplicated, and an empty set means omit the field.
• single method taking BlockNumberOrTagOrHash; no *ByHash variant, no mode param. A client that accepts a format-selection parameter MUST return the canonical witness when the caller omits it, so shipped mode params survive as extensions while the default flips to canonical.
• state/codes sorted lexicographically and deduplicated, headers ascending, per the host builder rules (build_execution_witness). Confirmed normative, which gives each block a single canonical witness and makes exact-match fixtures possible. The normative MUST is scoped to state/codes/headers; the optional keys field sits outside it.
• headers is never empty, because the guest takes the pre-state root from headers[-1].state_root. It holds at most 256 entries, since BLOCKHASH reaches back no further.
• genesis (block 0) and pending MUST error. Genesis has no parent header to prove the pre-state against, mirroring debug_traceBlock*; pending does not identify an executed block, and Erigon already rejects it. A hash naming a non-canonical block MAY be served when the client retains the required data.
• 4444 Pruned history unavailable reused; a distinct witness-unavailable code can ride the error-groups work (error-groups, tools, tests: add spec-mandated error-code fixtures for methods #784)

Update 2026-08-28: fixtures and the testgen generator are now included (the last commit). The two gaps the earlier draft named are now split cleanly:

1. Encoding: fixed in go-ethereum by eth: return canonical execution witness from debug_executionWitness go-ethereum#35606 (dedicated RPC result type with RLP headers and no keys field; pending rejected; the engine_newPayloadWithWitnessV5 blob is a separate encoding and untouched). Until that merges and the tools/go.mod pin advances, CI's make fill fails on the witness cases here, the same situation as eth_createAccessList: clarify gas-fee affordability when fee fields omitted #854.
2. Content: unchanged. Conformance with the canonical builder is the erigon-scale piece of work per client, and no client ships it yet. The fixtures therefore do not exact-match content: get-witness-by-number and get-witness-by-hash are speconly, so hive validates each client's response against this schema while the generator enforces structure at fill time (headers RLP-decode, ascend by block number, and end with the parent; state non-empty; state and codes sorted). Byte-exact fixtures would enshrine geth's legacy-semantics witness (bytecode on every read, sibling set from its trie iteration order); speconly avoids that while still rejecting wrong shapes.

get-witness-genesis, get-witness-pending, and get-witness-invalid-block cover the MUST-error inputs.

Verification: make fill reports zero failures against go-ethereum with ethereum/go-ethereum#35606 applied; speccheck passes, and injecting "keys": null into a fixture fails speccheck, which proves the schema is enforced. hive rpc-compat replays all five fixtures against geth built from that branch (Dockerfile.local, local fixtures, schema from this branch): 5/5 passed, failed=0, 2026-08-28.

The example values in the spec are real (generated from that geth against the test chain), but they illustrate the shape rather than a conformance baseline, per the content gap above.

[jsign]

Left some comments!

Mainly to be more precisely aligned with the specs.

[MysticRyuujin]

@jsign thanks — all four applied in 6b650d0. Two questions left that only the specs side can answer:

1. Is the sorting/dedup of state and codes normative, or incidental? The builder produces sorted(accessed_nodes.values()) and sorted(codes), and the descriptions here now assert it. If it's normative I can spec byte-exact responses and test them in rpc-compat; if it's just how build_execution_witness happens to be written, the RPC spec has to permit any order and conformance fixtures get much weaker. This is the single biggest factor in how testable this method is, so I'd rather pin it now.

2. Is the 256-header cap a protocol limit or guest-side defensiveness? validate_headers asserts len(encoded_headers) <= 256 and MAX_WITNESS_HEADERS fixes the same bound in the SSZ container. I've left it out of the schema for now rather than have the JSON-RPC layer invent a consensus bound — say if it should be stated.

[MysticRyuujin]

Open points that still need input from the client side, separate from the specs-side questions above.

keys semantics — @canepat, since you made the case for keeping it. It's optional in the schema, but the description is currently too vague to test against. To spec it I need what you actually consume in Zilkworm:

• reads only, or reads and writes?
• 20-byte account addresses and 32-byte storage slots in one flat array, or separated?
• is keys: [] distinguishable from omitting the field? geth returns keys: null today, which this schema rejects — I'd prefer clients omit rather than send null, but say if null is load-bearing.
• does anything depend on ordering, or is dedup sufficient?

One consequence worth stating plainly, since it's the cost of optionality: witness responses can't be byte-exact across clients if keys is optional — one conforming client includes it, another omits it, and both are correct. Conformance fixtures would have to compare the canonical {state, codes, headers} and tolerate the extension. I think that's an acceptable trade, but it should be a decision rather than a surprise when fixtures land.

Two input cases the draft doesn't define. I've specced the genesis case (no parent header to prove the pre-state against, so clients MUST error — mirroring debug_traceBlock*), but deliberately left these two open because a MUST here could spec shipped behavior into non-compliance:

• pending — the block param accepts the tag, and a witness for a block that hasn't been executed doesn't obviously mean anything. Do any of you serve it today?
• a block hash naming a non-canonical block — my inclination is to permit it when the client retains the block and the state it executed against, but there's no requireCanonical selector on this param. Correct me if that contradicts what you ship.

Three positions nobody has pushed back on yet, which I'd rather have confirmed than assumed: dropping mode (reth and erigon both ship it, both defaulting to legacy), a single method taking BlockNumberOrTagOrHash rather than reth's separate *ByBlockHash, and RLP-encoded headers.

That last one is the blocker for fixtures: geth currently returns headers as JSON objects, so nothing here can be conformance-tested until that changes. Is anyone on the geth side willing to own it? The unknown-block-number crash I hit while prototyping (BlockByNumberOrHash returning nil, nil and the handler dereferencing it) needs a nil guard independently of this spec.

[jsign]

@jsign thanks — all four applied in 6b650d0. Two questions left that only the specs side can answer:

1. Is the sorting/dedup of state and codes normative, or incidental? The builder produces sorted(accessed_nodes.values()) and sorted(codes), and the descriptions here now assert it. If it's normative I can spec byte-exact responses and test them in rpc-compat; if it's just how build_execution_witness happens to be written, the RPC spec has to permit any order and conformance fixtures get much weaker. This is the single biggest factor in how testable this method is, so I'd rather pin it now.
2. Is the 256-header cap a protocol limit or guest-side defensiveness? validate_headers asserts len(encoded_headers) <= 256 and MAX_WITNESS_HEADERS fixes the same bound in the SSZ container. I've left it out of the schema for now rather than have the JSON-RPC layer invent a consensus bound — say if it should be stated.

1. For now let's consider it normative, although this this might change in the future. I think it is better debug_executionWitness is in full alignment on how the specs construct the witness. This sorting/deduping was done to have canonical witness.
2. A cap protocol, since BLOCKHASH opcode can ask for up to an 256 old block hash -- so comes from there. As in, it shouldn't make sense we have 300 items in that entry.

[canepat]

keys semantics — @canepat, since you made the case for keeping it. It's optional in the schema, but the description is currently too vague to test against. To spec it I need what you actually consume in Zilkworm:

• reads only, or reads and writes?

Both read-only and read/write accesses.

• 20-byte account addresses and 32-byte storage slots in one flat array, or separated?

One flat array.

• is keys: [] distinguishable from omitting the field? geth returns keys: null today, which this schema rejects — I'd prefer clients omit rather than send null, but say if null is load-bearing.

Empty array, omitted and null are all treated as empty preimage set, so I would say it's ok asking clients to omit rather than send null.

• does anything depend on ordering, or is dedup sufficient?

Ordering does not matter for us on the prover side and also dedup happens automatically. I guess we may lean towards asking for dedup just to keep keys as small as possible.

One consequence worth stating plainly, since it's the cost of optionality: witness responses can't be byte-exact across clients if keys is optional — one conforming client includes it, another omits it, and both are correct. Conformance fixtures would have to compare the canonical {state, codes, headers} and tolerate the extension. I think that's an acceptable trade, but it should be a decision rather than a surprise when fixtures land.

As a small improvement over this, conformance fixtures might contain the keys in the expected response and compare it only if the client includes it.

Two input cases the draft doesn't define. I've specced the genesis case (no parent header to prove the pre-state against, so clients MUST error — mirroring debug_traceBlock*), but deliberately left these two open because a MUST here could spec shipped behavior into non-compliance:

• pending — the block param accepts the tag, and a witness for a block that hasn't been executed doesn't obviously mean anything. Do any of you serve it today?

Erigon doesn't and we have an open PR to explicitly return an error in case pending is passed.

• a block hash naming a non-canonical block — my inclination is to permit it when the client retains the block and the state it executed against, but there's no requireCanonical selector on this param. Correct me if that contradicts what you ship.

No objections.

Three positions nobody has pushed back on yet, which I'd rather have confirmed than assumed: dropping mode (reth and erigon both ship it, both defaulting to legacy), a single method taking BlockNumberOrTagOrHash rather than reth's separate *ByBlockHash, and RLP-encoded headers.

• dropping mode: this parameter is meant to help transitioning the witness representation from the legacy format to the canonical one. I think a smooth transition for witness consumers i.e. provers would be to first change the default and eventually remove the parameter. Perhaps we could make it optional in the spec to facilitate such transition, while avoiding burdening other clients with its implementation? Or just leave it out of the spec but don't rush the clients that support it to remove it. In any case, the main concern here is that the spec should precisely define the expected witness content and assumptions (e.g. empty bytecodes included or not, empty trie nodes included or not, the required order to apply inserts/updates/deletes to the trie...)
• single method taking BlockNumberOrTagOrHash: no objections
• RLP-encoded headers: no objections

[MysticRyuujin]

Thanks @canepat, all applied in 58eab5e.

• keys is now pinned to what you described: reads and writes, one flat array of 20-byte addresses and 32-byte slot keys, deduplicated. I kept the lexicographic sort even though ordering does not matter on the prover side; it keeps the field deterministic and consistent with state/codes. The schema now types each item as an address or a 32-byte key, and an empty set means omit the field, never null or [].
• pending now MUST error, next to the genesis rule. Erigon rejects it and geth's pending path crashes today, so no shipped behavior turns non-compliant.
• Non-canonical by hash is now an explicit MAY when the client retains the block and the parent state.
• mode stays out of the spec, but the transition you described is now in the method description: a client that accepts a format-selection parameter MUST return the canonical witness when the caller omits it. So reth and erigon keep mode as an extension, flip the default to canonical, and remove it on their own schedule.
• On precisely defining the witness content: the normative statement is now scoped to state/codes/headers matching the canonical builder (build_execution_witness). The content rules (empty bytecode excluded, node selection, ordering) live in execution-specs, and the spec references them rather than restating them, so the two cannot drift apart. The field descriptions summarize them for readers.
• Your fixture idea (expected responses carry keys, compared only when the client returns it) is how I plan to wire rpc-compat when fixtures land.

[MysticRyuujin]

@jsign @canepat

Mind taking a look at the latest YAML for the spec?

[MysticRyuujin]

Since the approvals: added the testgen generator and five fixtures as the last commit (8a670c3), so spec and tests review in one place. The two witness-returning cases are speconly; rationale and verification (fill, speccheck plus a keys:null negative test, hive 5/5 against a patched geth) are in the updated description. The geth encoding change is up as ethereum/go-ethereum#35606. CI's make fill stays red here until that merges and the tools/go.mod pin advances, the same situation as #854. If sign-off lands first, I drop the fixtures commit and this merges spec-only.

[MysticRyuujin]

Geth closed my PR so 🤷🏻‍♂️