Add experimental complete-blob-backfill flag

[michaelsproul]

Issue Addressed

A different (and complementary) approach for:

• Import expired blobs #5391

Proposed Changes

This PR adds a flag to set the DA boundary to the Deneb fork. The effect of this change is that Lighthouse will try to backfill all blobs.

Most peers do not have this data, but I'm thinking that combined with trusted-peers this could be quite effective.

Additional Info

• We turn on --prune-blobs false when this flag is enabled, and override any setting of --prune-blobs true.
• Should we turn on genesis-backfill whenever complete-blob-backfill is turned on? I'm inclined to say no, because users may want to backfill blobs only to e.g. 5 months (standard block backfill distance).

[chong-he]

I have done some testing on this together with @michaelsproul

To backfill the blobs beyond the 18-day window, we supply to the node that we want to backfill the following flags:
--libp2p-addresses
--trusted-peer

Example: lighthouse bn --libp2p-addresses /ip4/ip_address/tcp/9000 --trusted-peer 16Ulongstring --some-other-flags

It is also recommended to supply the same flag to the source node (the node with the blobs data).

Some caveats/known issues:

1. From time to time the following logs will appear:

ERROR Backfill sync failed                          error: BatchDownloadFailed(Epoch(365247))
WARN  Backfill batch failed to download. Penalizing peers  score_adjustment: Low Tolerance Error, batch_epoch: 365329 

I believe this is expected as most of the peers do not have the blobs data outside the 18-day window. As the peers are penalized, this results in a fluctuating peer count, as evident from the connected peer below (note that the default peer is now 200 in the unstable branch):

2. The backfill of blocks+blobs further from the head could be slow, in the range of 2-4 slots/sec (this could also be partly due to hardware limitations). So a complete backfill of blobs until the Deneb fork is estimated to take some time, could be as long as a few weeks.

   This could be improve if there are more nodes with blobs data are supplied - though this is not yet tested. In the testing, we supply only 1 node

3. The backfill could stuck - I have not investigated this. When it stalls, a restart of the beacon node will "resume" the backfill

Additional info: the node blobs data has backfilled to slot 10.2M from a fresh checkpoint sync:

"anchor": {
    "anchor_slot": "12452832",
    "oldest_block_slot": "10202176",
    "oldest_block_parent": "0xc91bf2f51b1ffdefe6dd791ba9c0465bf486cedea6f28d231b8f5742012be6d9",
    "state_upper_limit": "18446744073709551615",
    "state_lower_limit": "0"
  },
  "blob_info": {
    "oldest_blob_slot": "10202176",
    "blobs_db": true
  }
}

[jimmygchen]

Nice, looks good to me. Just need to disable the prune blobs behaviour when the flag is present.

[michaelsproul]

Done. I opted to just override --prune-blobs true if it is set, as this was simpler than re-parsing the same flag again, or fiddling with the order of parsing. I've added a test that confirms the behaviour, so we can always change it if we want to, but won't change it accidentally.

[jimmygchen]

Looks good, thanks!