engine: add payload witness endpoint to REST + SSZ proposal

[jsign]

Note: this PR is based on previous work from @developeruche, plus some extra refinements

This PR adds POST /engine/v1/payloads/witness to the REST + SSZ proposal. It lets zkVM provers and stateless validators submit a payload and receive its validation result, execution witness, and transaction sender public keys in one call, avoiding a separate witness request and hex-JSON encoding.

The endpoint is optional from Amsterdam onward and advertised through capabilities. It accepts the same request as /payloads. Every VALID response must include a complete witness, including for already-known payloads; other statuses must omit it.

Responses also include public_keys, allowing proving hosts to populate StatelessInput.public_keys without recovering keys themselves. For VALID payloads, including already-known payloads, the list contains one 65-byte uncompressed sender public key per transaction, in transaction order; for other statuses, it is empty. Stateless validators must still verify the supplied keys against transaction signatures and recovery IDs.

The proposal defines witness contents, public-key semantics, SSZ containers, and size bounds. Behavioral requirements live in refactor.md; encoding definitions live in refactor-ssz.md.

[developeruche]

This looks good to me!..

But currently, this capability is not being advertised.

I guess /engine/v1/capabilities could return an extra field "optional_endpoints": ["payloads/witness"], the overall response would be something like this;

{
  "supported_forks":          ["paris", "shanghai", "cancun", "prague", "osaka", "amsterdam"],
  "optional_endpoints":       ["payloads/witness"],
  "fork_scoped_endpoints":    ["payloads", "forkchoice", "bodies"],
  "independently_versioned":  { "blobs": ["v1", "v2", "v3", "v4"] },
  "unscoped_endpoints":       ["capabilities", "identity"],
  "limits": {
    "bodies.max_count":           32,
    "blobs.max_versioned_hashes": 128,
    "payload.max_bytes":          67108864
  }
}

What do you think?

also made a PR adding an example for this route here

[jsign]

This looks good to me!..

But currently, this capability is not being advertised.

I guess /engine/v1/capabilities could return an extra field "optional_endpoints": ["payloads/witness"], the overall response would be something like this;

{
  "supported_forks":          ["paris", "shanghai", "cancun", "prague", "osaka", "amsterdam"],
  "optional_endpoints":       ["payloads/witness"],
  "fork_scoped_endpoints":    ["payloads", "forkchoice", "bodies"],
  "independently_versioned":  { "blobs": ["v1", "v2", "v3", "v4"] },
  "unscoped_endpoints":       ["capabilities", "identity"],
  "limits": {
    "bodies.max_count":           32,
    "blobs.max_versioned_hashes": 128,
    "payload.max_bytes":          67108864
  }
}

What do you think?

Uhm, I'm not sure this is necessary, since in this PR we already say that this optional endpoint is advertised in fork_scoped_endpoints here:

ELs that implement it advertise payloads/witness in the fork_scoped_endpoints list of GET /capabilities

So if it is advertised there, I'm not sure we need a new optional_endpoints field.
Curious to get @MariusVanDerWijden opinion on this.

Reg the example PR, I'll take a look soon! Merged!

[jsign]

@MariusVanDerWijden opened for review!

[MariusVanDerWijden]

Generally LGTM, theres one change that I couldn't understand

[MariusVanDerWijden]

Whats up with this?

[jsign]

Ah, this just fixes a typo in an existing example. The existing PayloadStatus enum defines VALID = 0 and INVALID = 1, so the byte here should be 0x00.

[jsign]

@MysticRyuujin could this be merged?

[MysticRyuujin]

@jsign - I usually don't merge engine changes since it's the prevue of the CL devs. Was is approved in an ACD call or something? I can merge it if there's consensus on it though. I only see @MariusVanDerWijden here and he reviewed but did not actually approve yet

[MysticRyuujin]

Has it been implemented in any of the clients?

Fable:

[jsign]

Was is approved in an ACD call or something? I can merge it if there's consensus on it though.

I think this was agreed to happen in a previous breakout call where @MariusVanDerWijden and @developeruche participated.

I only see @MariusVanDerWijden here and he reviewed but did not actually approve yet

Ah, true. Maybe @MariusVanDerWijden can explicitly approve if it looks good. I kind of assumed this was okay reg his latest message but agree a green check sounds good.

Has it been implemented in any of the clients?

@MysticRyuujin, I don't think so -- is this a requirement?
If so, I can talk with some ELs about implementing it if it's required to merge the PR. Is this the case? Should this happen in a main branch in an EL? (Or was it just a general question?)

(Recall this is PR express the endpoint as an optional one)

[MysticRyuujin]

Personally, I like to see a majority of clients at least thumbs up OR have implemented the proposal (or at least have drafts/PRs) to know that if we merge it, it matches or will soon match reality.

It's not a hard rule for the repo but its how I see it and try to manage it on the json-rpc execution APIs side.

[jsign]

Personally, I like to see a majority of clients at least thumbs up OR have implemented the proposal (or at least have drafts/PRs) to know that if we merge it, it matches or will soon match reality.

It's not a hard rule for the repo but its how I see it and try to manage it on the json-rpc execution APIs side.

Sounds good to me! I'll see to have a draft PR in an EL client to provide more confidence. Can ping back when I've something to reference!