Add EIP: Max blob flag for local builders

[kevaundray]

No description provided.

[kevaundray]

default is up for debate here

[smartprogrammer93]

default should be null, and only if a value is configured it is then applied

[kevaundray]

Potuz brought up the valid point that if I am using mev-boost with min-bid and I have low bandwidth, then I will fallback to the protocol max if no value is assigned. What do you think?

[smartprogrammer93]

yes, and that should be the default. Operators need to be aware of their bandwidth limitations and set a proper value if the max is not suitable for their connection.

[rolfyone]

This could go very poorly if everyone were to set USER_CONFIGURED_MAX_BLOBS_PER_BLOCK =1 and all of a sudden we have no DA... This was given previously as a reason to not allow clients to configure a lower number of blobs from local building.

[kevaundray]

This affects the builder software so I guess even without this change, the sophisticated builders could also choose to not add blobs in their own builder code.

I did hear the censoring argument from @potuz and I disagree that having validators specify how many blobs they want to include in a block is censoring

[smartprogrammer93]

Having validators specify how many blobs in their blocks is not censoring. My block my rules. I can put a single transaction in it and that is okay as long as i follow consensus rules. That is my opinion. Now regardless, this limit should only affects the proposer in case of local building, so if MEV is used, the builder is the one who decides how many blobs to include.

[jimmygchen]

Agree with the above. Any concerns with a large builder deciding to not include blobs at all? 🤔
Not sure if there's any incentives to do so though.

[fradamt]

If builders decide not to include blobs because they are not paying enough, priority fees can be increased to compensate them.

[kevaundray]

Large/sophisticated builders are likely using custom builder software, so I think they could choose to include zero blobs without this change

[jflo]

how do we feel about a minimum value of 1 for this?

[kevaundray]

how do we feel about a minimum value of 1 for this?

1 as a default would be the "safest" -- do you think there is a large benefit to that over using the current pre-pectra value?

[smartprogrammer93]

@kevaundray. I assume what is meant by minimum here means the operator cannot set the maxBlobsPerBlock to 0. And if the user sets 0, it takes 1 as 1 is the minimum.

[kevaundray]

@kevaundray. I assume what is meant by minimum here means the operator cannot set the maxBlobsPerBlock to 0. And if the user sets 0, it takes 1 as 1 is the minimum.

Yep, I did not explicitly write that so I'll modify

[kevaundray]

pinging @MariusVanDerWijden for visibility

[MariusVanDerWijden]

Looks okay, not really an EIP, more an ERC in my opinion.
Again I worry about the instability in the blob base fee that it could cause, but others have already tried to convince me that this is not an issue. Another worry I have is that a transaction can still send up to max-size blobs, which I think we should change before we start work on this

[kevaundray]

Looks okay, not really an EIP, more an ERC in my opinion.

Again I worry about the instability in the blob base fee that it could cause, but others have already tried to convince me that this is not an issue. Another worry I have is that a transaction can still send up to max-size blobs, which I think we should change before we start work on this

I think a transaction sending max-size blobs is okay --
You would just not add it to your block right?

It seems the geth code keeps a running count of the blobs added to a block but currently it uses the Max-size to know when to stop -- I imagine that you could just use the user configured value?

[mteam88]

Kev mentioned this on a twitter thread about node requirements here: https://x.com/kevaundray/status/1886429037372801512

I think this provides some useful context.

[eth-bot]

✅ All reviewers have approved.

[jimmygchen]

Hi @kevaundray @fradamt

Is this (proposer including max blobs leading to a miss block) still a problem today with getBlobs already implemented?

Once we have PeerDAS, we'd really have to rely on distributed blob building (publishing):

• Without distributed blob building, if there's 64 blobs in a block, the proposer would send out 16MB worth of data to 8 peers, that's 128mb - and to send that under 500ms, the node would need 2 Gbps bandwidth!
• If we don't have a effective version of distributed blob building, then local block builders would only be able to safely include 4 blobs with the same bandwidth requirement (blob count for electra divided by 2 due to erasure coding).

IMHO we should really have a distributed blob building implementation before shipping PeerDAS - and even having enough clients (it could be 1-2) having a good implementation would be sufficient with helping blob propagation.

I fear that this flag would be misused, even if the validator's capacity allows - there's not much incentive to include blob txs, and this makes local block builder less useful for L2s, and hurts scalability of the DA layer.

[jimmygchen]

Proposer bandwidth is a problem we must solve with PeerDAS - and I think we will solve it with distributed blob building (which is mostly distributed blob publishing after #9378) , because local block builders will be pretty useless with propagating large number of blobs anyway - anything more than half of the electra max blobs (due to erasure coding) will require a higher bandwidth for them today - which is only 4.5 blobs under PeerDAS without distributed blob building.

[jimmygchen]

What's the reason to allow for even less than Electra max blob per block?

[kevaundray]

If you only have bandwidth for one blob for example (we don't allow this now, you would just miss the slot or have to use MEV-Boost) -- then you can set it one.

There was an argument being made to even allow zero because if local block builders think they are not being correctly incentivised by the priority fee then they can choose to not include any blobs

[0x00101010]

Would be great giving local block builders without MEV-boost a way to skip blob transactions so that we could continue to scale the network without compromising on decentralization (local builders forfeit a little bit blob fee in favor of higher chance of block inclusion)

[jimmygchen]

Here are some metrics to measure of effectiveness of getBlobs and Distributed Blob Publishing (posted in peerdas-testing but thought it's worth sharing here)

• the EL getBlobs hit rate is pretty high, we always get some blobs from the EL
• miss blobs per minute range from 0-8 blobs, we currently don't do retries on getBlobs, but a retry on getBlobs after some milliseconds could improve this further.
• the node is constantly publishing a small number of blobs (it only publishes what it hasn't seen on gossip)
• this is using geth, we know that the performance varies across EL, but geth is pretty fast.

This is metrics from mainnet today, which isn't too bad, and if we can bring down the missing blob count further (will be a focus in PeerDAS), local block builders may not have to publish blobs / columns at all - or alternatively we could do gradual blob publishing, which requires less bandwidth.

[kevaundray]

Thanks for the data!

I think intuitively, it will never be zero because if I receive a blob at time T, and I need to build a block at time T, then I would need to publish blobs because no-one else has seen them yet. But as time goes on, the probability that I need to publish blobs at the time of block building goes down.

Also I think this still is linked to bandwidth because if I receive N blobs at time T and bandwidth only allows me to publish N/2 of those before it is my time to publish, then I may need to publish the other N/2 right? N is likely very important here because as the blob target/limit increases, I imagine that there will be more blobs in the mempool. Perhaps low bandwidth nodes can instead order transactions by how long they have been in the mempool when restricting the number of blobs?

[fradamt]

Hi @kevaundray @fradamt

Is this (proposer including max blobs leading to a miss block) still a problem today with getBlobs already implemented?

Once we have PeerDAS, we'd really have to rely on distributed blob building (publishing):

• Without distributed blob building, if there's 64 blobs in a block, the proposer would send out 16MB worth of data to 8 peers, that's 128mb - and to send that under 500ms, the node would need 2 Gbps bandwidth!
• If we don't have a effective version of distributed blob building, then local block builders would only be able to safely include 4 blobs with the same bandwidth requirement (blob count for electra divided by 2 due to erasure coding).

IMHO we should really have a distributed blob building implementation before shipping PeerDAS - and even having enough clients (it could be 1-2) having a good implementation would be sufficient with helping blob propagation.

I fear that this flag would be misused, even if the validator's capacity allows - there's not much incentive to include blob txs, and this makes local block builder less useful for L2s, and hurts scalability of the DA layer.

Imo, this flag is the backup option. If we can quickly get distributed block publishing (+1 to the renaming) working such that anyone can propose a block with a high blob count, great. This is by far the best option and what we would be striving for, and I don't see why we can't do it. However, if it turns out that it needs more work, imo we should still go ahead with PeerDAS, though perhaps with a lower blob count than we would otherwise, and tell validators to either use mev-boost or this flag (or use mev-boost but still have the flag set in a reasonable way, in case they fallback to local block building).

Another small point is that the most important time is probably the time to publish each column to one peer, rather than to 8 of them, since at that point all pieces of data can be propagated without relying on your own bandwidth.

[kevaundray]

Hi @kevaundray @fradamt

Is this (proposer including max blobs leading to a miss block) still a problem today with getBlobs already implemented?

Once we have PeerDAS, we'd really have to rely on distributed blob building (publishing):

• Without distributed blob building, if there's 64 blobs in a block, the proposer would send out 16MB worth of data to 8 peers, that's 128mb - and to send that under 500ms, the node would need 2 Gbps bandwidth!

• If we don't have a effective version of distributed blob building, then local block builders would only be able to safely include 4 blobs with the same bandwidth requirement (blob count for electra divided by 2 due to erasure coding).

IMHO we should really have a distributed blob building implementation before shipping PeerDAS - and even having enough clients (it could be 1-2) having a good implementation would be sufficient with helping blob propagation.

I fear that this flag would be misused, even if the validator's capacity allows - there's not much incentive to include blob txs, and this makes local block builder less useful for L2s, and hurts scalability of the DA layer.

For local block building, a more reasonable time for publishing is 2 seconds right? For sophisticated builders playing timing games, I think 500ms or less is likely the sort of latency they would want. Something to note also here is that before 7623, the worse case block is about 2MB, so for 2 seconds this requires 64Mbps and after 7623 this effectively halves to 32Mbps, so bandwidth requirements (not for the average case) are already somewhat high

GetBlobs should definitely help, however quantifying its effect is dependent on the mempool implementation of each client, I imagine that it somewhat depends on your network topology too. In theory, it should require zero bandwidth to publish the blob in the happy case if it has been submitted to the network well in advance.

On the usefulness to L2s, local block building is roughly at 10% so it shouldn't have much of an effect on the L2s and DA scaling right? We asked the rollups in a separate call and there were no strong oppositions to it, where the main inconvenience would be the gas market.
(This is also assuming that all local block builders set their max blobs as low as possible)

In general, I think local block builders should be given the option to specify how their block should be built according to their constraints, as long as they follow the rules of the network and once we have full FOCIL, local block builders with low bandwidth could choose to use MeV-boost and instead contribute to censorship resistance via FOCIL

[jimmygchen]

Imo, this flag is the backup option. If we can quickly get distributed block publishing (+1 to the renaming) working such that anyone can propose a block with a high blob count, great. This is by far the best option and what we would be striving for, and I don't see why we can't do it. However, if it turns out that it needs more work, imo we should still go ahead with PeerDAS, though perhaps with a lower blob count than we would otherwise, and tell validators to either use mev-boost or this flag (or use mev-boost but still have the flag set in a reasonable way, in case they fallback to local block building).

💯 Agree.

Another small point is that the most important time is probably the time to publish each column to one peer, rather than to 8 of them, since at that point all pieces of data can be propagated without relying on your own bandwidth.

Yes that will help. Although for high blob count like 64 that's still a lot of data (16mb) that need to go out within a few hundred milliseconds 2 seconds (64 mbps required).

But overall I'm not against this EIP if the intention is using this as a backup option. I agree it's important that we have these local builders and making sure they can still produce blocks.

[jimmygchen]

GetBlobs should definitely help, however quantifying its effect is dependent on the mempool implementation of each client, I imagine that it somewhat depends on your network topology too. In theory, it should require zero bandwidth to publish the blob in the happy case if it has been submitted to the network well in advance.

Yes some implementation may be better than others, and the nodes using the better implementation (higher hit rate and low response time) would likely be the one contributing to blob publishing.

All nodes in the network that runs the latest spec-compliant client today already contribute the blob publishing, so if the proposer is only including public blob transactions, there should be some nodes (even just 1 would already help) on the network that already see the same blob and publish it to the network after receiving the block.

So in theory the proposer bandwidth just need to be sufficient to quickly publish the block. I'd like to see some metrics across multiple nodes to ensure all blobs for every block are published by one of the nodes - will reach out to pandaOps to see how we can make this work.

I'll stop here, as I'm going off topic :) and thanks for considering my input.

[kevaundray]

GetBlobs should definitely help, however quantifying its effect is dependent on the mempool implementation of each client, I imagine that it somewhat depends on your network topology too. In theory, it should require zero bandwidth to publish the blob in the happy case if it has been submitted to the network well in advance.

Yes some implementation may be better than others, and the nodes using the better implementation (higher hit rate and low response time) would likely be the one contributing to blob publishing.

All nodes in the network that runs the latest spec-compliant client today already contribute the blob publishing, so if the proposer is only including public blob transactions, there should be some nodes (even just 1 would already help) on the network that already see the same blob and publish it to the network after receiving the block.

So in theory the proposer bandwidth just need to be sufficient to quickly publish the block. I'd like to see some metrics across multiple nodes to ensure all blobs for every block are published by one of the nodes - will reach out to pandaOps to see how we can make this work.

I'll stop here, as I'm going off topic :) and thanks for considering my input.

imo this input is extremely valuable

[eth-bot]

All Reviewers Have Approved; Performing Automatic Merge...