Import expired blobs

[realbigsean]

Description

• initializing a new archive node that wishes to store all blobs back to the deneb fork will need a way to import blobs into the db
• syncing them over the network might place an undue burden on archive nodes because you'd ban all non-archive peers (unless we make peer scoring changes around this). This might not work well at all if there are too few archive nodes. It might work just by updating min_epochs_for_blob_sidecars_requests to u64::MAX
• a better option would be to add a --blob-provider flag that points to an archive node's beacon API
• we could also potentially support importing a blobs db from a file

[realbigsean]

we should try to include an option for this in the next release because I think it'll be an important feature once blobs start expiring on mainnet. I'm not sure which path is easier/more useful so would appreciate feedback. We could end up implementing both, but I think starting with one for the next release makes sense

RPC download

• if --blob-provider $URL is added, always download via beacon API up until the configured prune boundary
• could this run as a process mostly independent of sync? It would just be required to lag sync
  • how would this handle a scenario where blob data is invalid or unavailable - cause a crash?

DB Import export

When lighthouse is shut off:

• lighthouse db export-blobs --file $FILENAME
• lighthouse db import-blobs --file $FILENAME

On startup:

• should lighthouse identify the db changes and go back to verify old blobs if the node is synced?
• should this only be possible on a fresh sync?
• do we need to add a flag --archive-blobs ? Or should this automatically be identified
• LH will need to be aware of old blobs during sync so as to not duplicate download and writes, and to verify blocks older than the prune boundary against local blobs
• how should gaps in the db here be handled if they are older than the prune boundary?

[jonathanudd]

Any updates regarding this?

[michaelsproul]

@jonathanudd Not yet sorry. We've been blocked a bit on implementing PeerDAS, but now that the bulk of that code is in unstable we can set to work on a future-compatible design here.

One complication is that with PeerDAS most nodes will cease to store whole blobs, so only supernodes (nodes that opt to store all blobs) will be able to implement the HTTP API for fetching/exporting blobs. Every other node will just have fragments ("columns").

Another issue is partial blob storage. At the moment Lighthouse uses a single marker to track which blobs are available: the oldest_blob_slot. The simplest way to implement an archive mode without changing this would be to store every blob from oldest_blob_slot forwards, and just lower that slot when we import a batch. However a more useful mode for L2s is probably to archive just a subset of blobs that they're interested in. This would break the current oldest_blob_slot model, and require us to always consider the possibility of gaps. This may be fine, as we need to consider gaps even when importing the full history, as @realbigsean identified in his comment. Another complication related to partial blob storage is that we store lists of blobs in the DB indexed by block root, and maintain an invariant that we either have 0 blobs for a block, or all of them. This invariant would be harder to change, so the simplest form of partial blob storage would have to be on a per-block basis.

One way forward might be:

• Define an SSZ-based format for blobs similar to Era files. Something similar to Vec<BlobSidecar<E>> in slot-ascending order could be the starting point. We may want some metadata too like: file format version, network identifier, slot range (slot of oldest blob in the file, slot of newest blob in the file).
• We could assume for now that we only need to handle the Deneb case, or we could build in support for PeerDAS data columns in the file format. Using data columns would allow ordinary nodes (not supernodes) to import and export data. Alternatively, maybe the file format should be flexible enough to support both columns and full blobs post-PeerDAS?
• Implement lighthouse db import-blobs/export-blobs in terms of the above file format. To appease Lighthouse's DB invariants we could check that the sidecars for a given slot are complete, by comparing the list of blob sidecars to the blob KZG commitments from the block in the DB.
• Implement a consistency check in Lighthouse's HTTP API to guard against returning wrong data when there are gaps. This is related to: Will lighthouse return an empty list when fetching an expired blob? #5640. We should probably load the block header and check that the DB contains the correct (complete) number of blobs prior to returning. We can error out with a 404 if the blobs are unavailable.

Down the line we can:

• Encourage other clients to adopt the format we develop. This could be after some iteration on the design.
• Develop standalone tools for manipulating the archive files: e.g. filtering to achieve partial blob storage.