engine: add Rest-SSZ spec

[MariusVanDerWijden]

There have been multiple attempts at this already.
Moving away from JSON-RPC to REST-SSZ.
However most kept the engine api as is.
I think we have a good shot at refactoring the engine api with this change.

This Draft does that; the move and the refactoring.

Happy for any feedback I can get!

The core of the change is:

Old method	New endpoint	Notes
engine_newPayloadV{1..5}	POST /{fork}/payloads	parentBeaconBlockRoot and executionRequests folded into the SSZ envelope; expectedBlobVersionedHashes removed; INVALID_BLOCK_HASH removed from the status enum
engine_forkchoiceUpdatedV{1..4}	POST /{fork}/forkchoice	one atomic call; carries forkchoice state, optional payload_attributes, and (Amsterdam+) optional custody_columns
engine_getPayloadV{1..6}	GET /{fork}/payloads/{id}	poll-style, same semantics as today
engine_getPayloadBodiesByHashV{1,2}	POST /{fork}/bodies/hash	{fork} selects the response schema (not the era of requested blocks); POST because hash lists are too large for URLs
engine_getPayloadBodiesByRangeV{1,2}	GET /{fork}/bodies?from=...&count=...	{fork} selects the response schema
engine_getBlobsV1	POST /blobs/v1	independently versioned; legacy version numbers carry forward
engine_getBlobsV2	POST /blobs/v2	all-or-nothing cell proofs
engine_getBlobsV3	POST /blobs/v3	partial-response cell proofs
engine_getBlobsV4	POST /blobs/v4	cell-range selection
engine_getClientVersionV1	GET /identity + X-Engine-Client-Version request header	unscoped
engine_exchangeCapabilities	GET /capabilities	unscoped
engine_exchangeTransitionConfigurationV1	removed	already deprecated since Cancun

[arnetheduck]

One thing that I think would make sense would be to reuse the base structure of the beacon api (https://github.com/ethereum/beacon-APIs/) - this includes several things:

• primitives and their json/ssz encoding - ie in general, where possible, reuse the types from https://github.com/ethereum/beacon-APIs/blob/master/types/primitive.yaml so that we don't end up with pointless minor differences in how for example a number is string-encoded (0x0 vs 0x and the like)
• execution payloads and other (consensus) spec types - the "shape" of objects in the beacon api generally follows the shape of things as they travel on the gossip network and their SSZ encoding - by reusing these types, we would reduce the maintenance overhead of having to pointlessly reorder and rename the exact same fields from the beacon api/consensus spec just to send the same info to the execution api in a slightly different shape
• use of the canonical ssz/json encodings specified here: https://github.com/ethereum/consensus-specs/blob/master/ssz/simple-serialize.md#json-mapping - this aids debugging and removes the need to double-specify things
• explicit encoding of fork in the http headers -> we can then upgrade to an new hard fork "automatically" without having to come up with V2, V3 etc

[arnetheduck]

The core of the change is:

For top-up sync we also need "current block number", similar to eth_blockNumber but limited to latest and with a well-defined behavior for when the EL does not have a state.

[developeruche]

I opened #773 a few weeks ago with a narrower scope: adding a single new endpoint (POST /new-payload-with-witness) that combines engine_newPayload and debug_executionWitness into one call and returns the witness SSZ-encoded over HTTP. The motivation was to unblock zkVM provers and zkAttestors from having to follow the chain one block behind.

Since #793 is now doing a full Engine API refactor with the same REST+SSZ foundation, I think the witness endpoint fits naturally into this design. A few thoughts:

The witness endpoint should be added to this new spec. The existing two-call flow (engine_newPayload → debug_executionWitness) has real-world latency problems at a ~500 MB witness, the JSON-RPC + JSON approach takes ~8s just to return the witness. With HTTP + SSZ and the EL pipeline optimizations I profiled (moving trie writes off the critical path, parallelizing storage trie updates), this drops to ~932ms total EL time. That's comfortably within the 8s newPayload timeout even for worst-case blocks.

Suggested endpoint: POST /{fork}/payloads/with-witness (or folded directly into POST /{fork}/payloads as an optional response field when requested via a query param or Accept header). The response would carry the PayloadStatus + ExecutionWitness SSZ-encoded, consistent with the rest of the new spec.

Benchmark data (ethrex, 203 txs, 36 Mgas, ~502 MB SSZ witness):

Approach	EL Total	Wire Size
JSON-RPC + JSON	8,131 ms	~502 MB
HTTP + SSZ	1232 ms	502 MB

Happy to close #773 in support of this PR I think it's a better, more seamless flow. Would love to discuss where the witness endpoint fits best in the new endpoint table.

cc: @MariusVanDerWijden

[flcl42]

dropping /engine/v2 prefix misleads a bit

[arnetheduck]

+1 - the beacon api lives under /eth/vX/beacon - would be good to see this api under /eth/vX/engine to unify the approaches

[flcl42]

What does v2 mean? Was there v1? Why is fork necessary in the URL? Fork name seems to be a way to describe minor API version, but on other side blobs endpoints have it after resource name, not before and it's a number

[flcl42]

Also if payload did not change across forks, does it mean we will need same endpoint under different urls? Just single /vN/ looked simpler

[arnetheduck]

the consensus REST spec does not have this requirement - we use 1.1 throughout and going to 2.0 would not be viable short-term for Nimbus.

[kasey]

I've spoken with several people about developing streaming extensions for the engine API, some examples: registering to receive all txs from a particular account; receiving all cells for a set of kzg commitments; streaming EL events. For most language/library stacks http/2 is the simplest and most effective tool for this job. A lot of flexibility is introduced by giving the client/server native multiplexing, customizable framing. This could be powerful for some of the syncing options we've discussed like EL being fed blocks by CL.

As a compromise we could define these streaming methods as optional while clients work on introducing http/2 support. In practice http/2 capable libraries also support http/1.1. This is usually negotiated as ALPN during the TLS handshake, but the fallback option defined for plaintext protocols (like the engine api) is an upgrade header: Upgrade: h2c. So when the CL first connects to the EL to determine capabilities, it can make an http/1.1 request with an upgrade header (to confirm http/2 support if the EL server gives the expected upgrade response preamble). These streaming extensions would presumably be themselves represented as capabilities, potentially with a prefix that indicates the L7 protocol (http2/streamBlobsForCommitments).

[arnetheduck]

during the TLS handshake,

which tls handshake? we're talking about a private el/cl connection that in many cases does not have tls enabled (because enabling it would be an absolute mess of managing certificates etc or disabling verification which is unusual).

multiplexing

just make 2 connections in these cases?

upgrade

sure - nothing prevents an el/cl to talk http/2 - that said, this is also not a public interface serving thousands of clients but rather a private connection for driving one part of the client with another. For the public rpc, it makes sense - for the engine api that sees a few messages per second, it seems overengineered.

[kasey]

which tls handshake? we're talking about a private el/cl connection that in many cases does not have tls enabled (because enabling it would be an absolute mess of managing certificates etc or disabling verification which is unusual).

I might have confused things with too much superfluous info. I'm not saying TLS is involved, I'm saying it is not involved, which is ok because there is a cleartext upgrade path. More succinctly -- there are clear standards for a single server to support both, so we can define the ssz flavored existing methods as http/1.1 while allowing http/2 capable servers to upgrade their connections to http/2 and provide additional optional methods.

For the public rpc, it makes sense - for the engine api that sees a few messages per second, it seems overengineered

The point isn't about scale/concurrency - I'm not suggesting http/3 here. The appeal is here is actually the simplicity that comes from the shape of the protocol fitting the use case, vs requiring devs to layer on additional complexity in order to compensate for the fact that http/1.1 framing and encoding was built for a different kind of request/response cycle and has acquired technical debt.

[MariusVanDerWijden]

Updated the spec according to the results from the discussion

[MariusVanDerWijden]

In todays ssz call, we discussed that there are some axuiliary apis also exposed on the engine api. See https://github.com/ethereum/execution-apis/blob/main/src/engine/common.md#underlying-protocol

[LukaszRozmej]

Feedback from implementing this in Nethermind (our REST+SSZ surface tracks this draft; we've just re-synced against d39e9a27). Five things, roughly in order of how much they cost an implementer.

1. d39e9a27 renumbered the base path from /engine/v2/ to /engine/v1/ — was that intended?

The commit is titled "move fork into header out of path", and that's the change everyone reviewed. But it also changed /engine/v2/{fork}/... → /engine/v1/... throughout. We picked up the header move and kept v2, which means a CL following the current draft gets a 404 from us on every REST URL — and per the transition-window section that reads as "this EL has no REST surface, fall back to JSON-RPC", so the divergence is silent rather than loud.

Two things suggest the renumber may have been incidental rather than deliberate:

• refactor-ssz.md is still titled "Engine API v2 -- SSZ Container Sketches".
• refactor.md's own goals section still says "The new API puts the fork in the URL (/engine/v1/...)", and the PR description table still shows POST /{fork}/payloads.

If /engine/v1 is intended, the stale fork-in-URL text is worth a cleanup pass so implementers don't split the difference the way we did. refactor-ssz.md also still links #get-forkpayloadspayloadid, which no longer resolves.

2. The worked byte example contradicts the status enum

refactor-ssz.md pins VALID = 0 and its Example A correctly shows status: 0x00. But refactor.md § "Example: submit a payload" describes the same 41-byte response as:

status (1 byte = 0x01, VALID)

Since the whole point of that section is a byte-exact example, this one is likely to get copied into a test vector.

3. payload_id TTL contradicts the polling model

These two paragraphs sit a few lines apart under "Payload retrieval":

The EL keeps optimising the payload until the slot deadline, so successive GETs against the same {payloadId} may return different bytes.

Token TTL. A payloadId is valid until either the payload was retrieved by GET /payloads/{payloadId} or another payload was built via a forkchoice with payload attributes.

Under the second rule the first GET invalidates the token, so the polling described in the first is impossible. We implement the polling reading (a token stays valid until the build is superseded or expires). Worth pinning explicitly, since "MAY stop the build after serving a call" is a third, weaker statement in the same section.

4. 413 request-too-large vs 400 ssz-decode-error are indistinguishable for the count limits

The error table says 413 is for a body that "exceeds an advertised limits.* value". But bodies.max_count (32) and blobs.max_versioned_hashes (128) are also the SSZ MAX_* constants on the request containers — so a 33-hash BodiesByHashRequest is simultaneously an over-limit request (413) and a schema violation that fails to decode against List[Hash32, 32] (400 ssz-decode-error). Any implementation that decodes before dispatching will produce 400, and a length-limit check can't run before the decoder without duplicating SSZ layout knowledge in the handler.

Suggestion: either say that 413 applies only when an operator has advertised a value below the SSZ MAX_* (which makes 400 correct in the default configuration), or say 413 wins and note that implementations need a decoder that distinguishes limit overflow from structural failure. We currently return 400 here.

5. Minor: MAX_BAL_BYTES exceeds the request body cap

MAX_BAL_BYTES = MAX_BYTES_PER_TX = 2**30 (1 GiB) while MAX_REQUEST_BODY_SIZE = 2**26 (64 MiB), so the BAL bound is 16× the largest body that can carry it. Listed as an open question already, just noting the interaction when it gets tightened.

---

Two things that were unambiguous and worth keeping as-is, since they're easy to get wrong and the draft calls both out explicitly: the Optional[String] nesting (List[List[byte, 1024], 1], with the inner offset) and execution_requests preceding should_override_builder. The worked byte examples for the former are genuinely useful — we match them.

[MariusVanDerWijden]

We decided its time to get this merged upstream and keep iterating on it on main if necessary

[naoifelcnc]

src/engine/refactor-ssz.md

[naoifelcnc]

src/engine/refactor-ssz.md