Specify subscription API

[s1na]

Most EL clients implement the subscription API documented here: https://geth.ethereum.org/docs/interacting-with-geth/rpc/pubsub. These are widely adopted but not yet included in the specs. It's easy to specify eth_subscribe and eth_unsubscribe. I'm unsure how to specify the concrete methods (newHeads, newPendingTransactions, etc.) as they are passed as parameters to eth_subscribe. Plus when we push data, we're effectively sending an eth_subscription request from server to the client, and the data for that request depends on the concrete method.

After the current state is done I'd like to propose the following modifications:

• Adding an optional parameter to newPendingTransactions named fullTx. It defaults to false. If true the server returns the full tx object instead of the hash. This is already merged in geth.
• Adding an optional parameter to newHeads named verbosity. Default value of 0 returns only the hash. If value is 1, we return the complete head object. If value is 2, we return the complete block object. If value is 3 we return the block object + receipts.
• Add optional parameter to newPendingTransactions named filter. The server uses the filter to narrow down the list of txes to return.

[s1na]

Open-RPC cannot express the semantics of the eth_subscribe. It is one method that can have a different set of params based on the event type. E.g.:

• If type is logs next param can be a filter { address: .., topics: ..}
• The other types don't have at the moment further parameters.

The same thing happens in the notification requests, i.e. eth_subscription. Each event returns a different object. Because the parameters of this request are addressed by key, we could define a schema with keys for all possible event types. But that's less than ideal.

There exists an issue open-rpc/spec#381 which if fixed would unblock this item.

[shanejonas]

I wouldn't call this completely blocked. You can do a looser definition without tying the params together like was done here in the meantime:

https://github.com/MetaMask/api-specs/blob/f5e94e75b3434c284501810e8043484b4e697c03/openrpc.yaml#L1041-L1134

But I also ran into inconsistencies where geth implements eth_subscribe with params newPendingTransactions: string, and txFull: boolean but hardhat does not.

[s1na]

@shanejonas my plan was to specify the vanilla eth_subscribe and then propose the different arguments that each of those topics could have, like the txFull one you mentioned, but never got anywhere. Feel free to pick this up.

[fubuloubu]

Random aside reading this now, and also noting this would take a significant amount of client work to support, but as someone who's had to deal with streaming RPC data over websockets (and their limitations) for a while now I keep wondering why there can't be something more simple streaming over HTTP, perhaps leveraging SSE or JSON-RPC "Notifications" (or similar technology)

High level, create new JSON RPC methods using this type of HTTP streaming interface that mimic the same dynamics as the eth_subscribe methods: eth_streamNewHead and eth_streamLogs (probably want to skip pending transactions for now)

The two methods should have a SSE-like "restart" functionality where you can optionally give startBlock (as an integer value not too far from head, and maybe "finalized"/"latest") and get the stream of events since that block (to easily get up to date on reconnects). In both cases, when the client submits these requests to the node, the node will just keep streaming back new data until the client (or the node) disconnects. Having them be two separate RPC requests alleviates the problem of their specification, and having it be supported over HTTP (likely HTTP/2 over SSE, similar to how MCP works) would make it a lot easier to support because we won't need to have a separate transport just for subscriptions.

Sorry for this shot in the dark on a 2+ yr old issue, but it just keeps sticking in my head that it could solve a lot of problems with eth_subscribe