Standardizing the Parity trace_* APIs and their output formats

[banteg]

Several execution clients implement the Parity trace_* namespace, but differences in behavior make it difficult for tooling to consume traces consistently.

While investigating evm-trace’s conversion of compressed vmTrace output into full execution traces, we found discrepancies that led to fixes proposed across multiple client implementations. This raised a broader question: would execution-apis be the right place to specify these methods and establish shared conformance tests?

The traditional namespace comprises:

• Execution and replay: trace_call, trace_callMany, trace_rawTransaction, trace_replayTransaction, trace_replayBlockTransactions.
• Historical call traces: trace_block, trace_transaction, trace_get, trace_filter.

Our source survey found all nine methods in Erigon, Reth, and Nethermind. Besu implements eight, excluding trace_replayTransaction. Geth exposes related functionality through debug_*, including flatCallTracer, but does not implement this namespace.

Method availability does not necessarily imply compatible behavior. For example, the implementations differ in how trace_get interprets its position argument and whether it returns an object or an array. The compressed vmTrace format also needs precise semantics and test vectors if consumers are to reconstruct execution state reliably.

There are two questions we would like to clarify before preparing a specification:

1. Is there interest in standardizing the existing trace_* methods and output formats here, including resolving implementation differences and adding cross-client tests?
2. Would Geth be interested in implementing an agreed specification? Alternatively, would this repository accept an optional tracing profile, with conformance required from clients that expose it?

The contributors guide describes specified APIs as canonical requirements for RPC compatibility, so we would like to understand whether an optional profile fits the repository’s intended scope.

It may also make sense to separate execution/replay from historical search. trace_filter does not inherently require an address index: some clients replay blocks, while Erigon uses indexes to narrow the work. A specification could define observable behavior, query limits, and unavailable-history errors without prescribing storage architecture or requiring archive retention.

We can contribute concrete discrepancy reports and regression cases from the evm-trace investigation. Feedback on scope and the appropriate standardization path would help guide that work.

[banteg]

Some relevant background from earlier discussions:

• In February 2024, s1na explicitly supported specifying optional APIs, using trace_ as the example: most clients implement it, Geth does not, and the shared behavior remains unspecified. In the preceding discussion, fjl noted that there was no official optional status. These were proposals and observations at the time, rather than an adopted conformance policy.
• In December 2025, Chase described preparatory work for adding methods such as trace_block and trace_transaction, alongside the separate debug_trace* standardization effort. The corresponding Hive change, ethereum/hive#1378, merged and enabled the trace namespace on supporting clients. That provides infrastructure groundwork, though it does not itself specify these methods or establish an optional profile.

Is there an existing plan or unresolved blocker from that work that this issue should align with? In particular, it would help to understand the intended conformance model for methods implemented by only some clients. We can contribute discrepancy reports and regression fixtures from the evm-trace investigation, including cases where compressed vmTrace output cannot reliably reconstruct execution state.

[bomanaps]

Thanks for raising this, adding it to the RPC Standards call agenda for next Monday.

[MysticRyuujin]

Hey @banteg - I'll be as transparent and blunt as I can be.

I support the standardization of the "Parity" trace_ namespace. Historically, the blocker has always been the go-ethereum team. They have refused to add the namespace and the methods to go-ethereum in the past. With that said, many of the team members who opposed it are not maintainers anymore and I have spoken to some other members of the team who are in favor of adding support.

If it was completely up to me, tracers would be removed from the debug namespace entirely. Mixing tracer methods with dangerous methods like debug_setHead is, and always was, a terrible design decision.

Putting that aside, there are a few way I could see this going.

First and foremost we need a spec, ideally that spec would be clean across the clients that already support the method: Nethermind, Erigon, Reth, etc. such that we have a rough consensus among those clients and a baseline to test go-ethereum with. This could come in the form of a PR to this repo without tests to accompany it.

I do not know if all the other clients are in 100% agreement and conformance with the trace methods today, but a simple hivechain cross client test could give us the answer. If they're not, someone will have to decide what is "correct" and what is not, and create the upstream fixes to get them in agreement.

Once that is done, then we either need to:

A. Implement it in go-ethereum and generate the tests for this repo via testgen
...or...
B. Bypass go-ethereum, bypass the test files, and merge it without tests

I'm fairly confident that if we can propose the spec, get all the client teams to thumbs up (why wouldn't they if they already pass the spec with working implementations?) that the go-ethereum team would come on board.

[banteg]

A small update ahead of the call: we now have reproducible cases showing that the remaining differences extend beyond method signatures.

On Reth v2.5.2, read-only simulations show EIP-7702 delegation changes reported as unchanged in stateDiff.code, and constructor/delegated execution returning missing or stale vmTrace.code. These are trace-output problems; the simulated execution produces the expected results. So far these reproductions are on Reth only, rather than a completed cross-client comparison.

We've also prepared a diagnostic fixture set covering selector/filter semantics and output fidelity, with observed behavior separated from proposed expectations. Happy to contribute those once the group agrees on scope and the conformance/testing approach.

[MysticRyuujin]

I think your best best in getting the tracers standardized and merged into this repo is to focus on what the majority of clients support today, if say 4 out of 6 clients support the method and more or less agree on the output the less we have to argue about the spec and the more likely it is we can include it as canonical here.

The easiest way to figure that out is to just run hive's rpc-compat tests with hand written test files and identifying the diffs across clients. I'm sure Claude or Codex could knock that out in like an hour.

hive now has ethereum/hive#1587 which was created for testing callTracer, it should help here in generating useful calls to test against.

[MysticRyuujin]

Also, I have an AI skill for this type of work, if you're interested in taking a stab at it:
https://github.com/MysticRyuujin/rpcwright

[banteg]

Following @MysticRyuujin's suggestion, we ran Hive's rpc-compat tests against Erigon, Nethermind, Reth and Besu on the same small generated chain. There is shared behavior to build on, but some basic contracts need agreement: trace_get with an empty path returns a root object, null or an array depending on the client, and default trace_filter matching across sender/recipient lists splits OR (Erigon/Reth) versus AND (Nethermind/Besu).

We also reproduced reporting bugs, including missing precompile return bytes in Besu's call-frame result and omitted EIP-7702 code changes in Reth's replay stateDiff. I've collected the findings and suggested resolutions here. The recommendations are provisional; runnable fixtures are ready to contribute once we agree on scope and testing approach.

[MysticRyuujin]

The findings are significant, in that there's a lot of them, all of which would have to get resolved in the clients.

The doc you produced has seemingly sensible recommendations and explanations for those decisions.

The next step would be to write the actual spec yaml, resolving as many of the discrepancies as possible. Followed by draft PRs (or issues) to the clients aligning them with the proposed spec.

This would allow each client team to respond to the spec and evaluate the necessary changes to comply.

[banteg]

An update on this: I've put together a draft specification and a cross-client test harness covering all nine trace_* methods, including trace, stateDiff and vmTrace.

Following the suggestions here, there are now per-client reports for Erigon, Reth, Nethermind and Besu. Each report explains the observed differences, proposed behavior, and what would need to change, with runnable cases and links to the relevant client code. The decision ledger collects the open questions and rationale. These are proposals for review; where a client disagrees with the draft, either side could need changing.

I've also implemented the namespace in an experimental Geth branch, tested against the same harness. This gives us a concrete implementation to discuss with the Geth team, though it isn't upstream support or endorsement. Some independent reporting fixes have already been submitted upstream; their status is tracked here.

The next step is to work through the disagreements with client maintainers and prepare manageable patches alongside any revisions to the spec. It would help to have one person from each client team own that review: start with your client's report, challenge the proposed choices, and help settle the behavior and compatibility implications.

I've set up a Telegram working group for coordination. Please join or point someone from your team to it if you'd like to work on this. We can keep decisions and their rationale here and in the spec review so they're publicly reviewable.

[lupin012]

For H03/H13, Erigon agrees with the direction on both. A few notes on each

H03

• AND as the default, and unrecognized mode values rejected.
• The AND default restores OpenEthereum's semantics: an empty list matches everything, and the two sides are combined with AND. In Erigon the AND case is already expressible with mode: "intersection" (Intersection mode for trace_filter erigontech/erigon#2119), so the change flips the default without removing anything: whoever wants union can keep asking for it explicitly.
• On compatibility: the only request whose behavior changes is one that supplies both lists and no mode. The case to document is {fromAddress: [X], toAddress: [X]}, written to mean "all activity involving X": under AND it becomes "calls from X to X", so it returns almost nothing, and no error. The migration is one field, mode: "union" — which is also why we ask that the
  extension stays.
• We also ask that mode: "intersection" remains a valid value, as a no-op under the new default: it is what users who already needed AND send today, and they would be the only ones to break if it became an unsupported value.

H13

• Agreed, but we would widen it to "apply the full admission rules at the selected state": the nonce is one case, and the same path also skips the balance check and EIP-3607.
• Some context: the check is absent in OpenEthereum as well as in Nethermind, Besu and Erigon — OE did not validate the nonce and topped up the sender's balance. We support the change anyway, because without the check the CREATE address and the stateDiff nonce are derived from the state nonce, so they describe a different transaction.
• One consequence to keep in mind if the optional blockNumber field that OpenEthereum had is ever restored: with the check in place, tracing a transaction at a past block becomes harder, because the caller has to know in advance the nonce the sender had at that block — and if it does not match there is no remedy, since the transaction is signed. The exception is pointing
  at the exact block the transaction belonged to, where the nonce matches on its own. The two decisions stay separate, but they interact.
• Please settle the error code along with the policy: rejections are -32000 here and -32000/-32003 in Reth.

[banteg]

An update since the draft went up.

The draft changed. A source review compared the draft with the Parity source, current Besu/Erigon/Nethermind/Reth code and the sibling eth_* methods, and its recommendations are now in #895. The larger changes:

• trace_filter range errors follow eth_getLogs (eth: specify getLogs error for block ranges beyond head #875): a bound beyond the head, a reversed range or pending returns -32602, with no clamping.
• Failed frames (H09): a REVERT frame carries result: {gasUsed, output}, as Erigon and Reth already emit; a failed CREATE reports no address or code; other halts may omit result. Failure labels are a fixed list.
• vmTrace mem (H20) is the post-operation contents of the range the opcode's operands designate, including MLOAD and the full CALL output window. This reverses the earlier "written bytes" narrowing and matches Parity, Erigon, Nethermind and Besu for MLOAD.
• Unsigned calls (H15) take their fee defaults and zero-fee BASEFEE rule from eth_call/eth_simulateV1, and gas above the RPC cap runs at the cap as eth_call does.
• Error codes reuse existing conventions: eth_simulateV1 codes for trace_call/trace_callMany, and the eth_sendRawTransaction error groups for trace_rawTransaction.
• New rules for frame gas fields, calls that fail their precheck, callMany item boundaries, stateDiff existence and markers, and vmTrace gas at call boundaries, where the draft was previously silent.

@lupin012, on your H03/H13 notes: mode: "intersection" remains valid as the default, and the H13 codes are now settled together with the policy, using the error groups above with -32003 as the fallback. The block-argument interaction you describe is recorded under H12, which stays an extension.

Client progress. Eleven more fixes are open across Besu, Erigon, Nethermind, Reth and revm-inspectors, and Reth #27423 has merged. The decision index now marks 🛠️ where a submitted fix covers the whole measured difference, so the remaining ⚠️ cells are unaddressed work. H02, H07, H18 and H26 are converged through merged client fixes, alongside H03, H13 and H28. The experimental Geth fork follows the revised draft with no differences in the current matrix.

The reorg scenario also found two Engine API bugs outside the trace namespace: Erigon #24292 (a same-head forkchoice update drops safe/finalized, which now blocks valid reorgs) and Reth #27429 (a stale on-disk head is treated as canonical after a reorg).

Where input would help most:

• Client positions on the changed rules above, especially H09, H20 and the error codes. Each client report lists what would change for that client.
• Could a maintainer approve the pending workflows on feat(trace): Parity trace methods and output schemas #895, so the proposal files can be validated?