PeerDAS: rate limit when syncing

[twoeths]

Describe the bug

Sometimes we can not sync a peer up to head, the #1 error of "Batch download error" is rate limit:

...
2025-07-03 20:34:01.640 | Jul-03 13:34:01.639[sync]          verbose: Batch download error id=Head, startEpoch=1240, status=Downloading, peer=16...2dKKZW, code=REQUEST_ERROR_INVALID_REQUEST, errorMessage=sNaPpYuV~2rate limited |  
-- | -- | --
  |   | 2025-07-03 20:34:01.639 | Jul-03 13:34:01.639[sync]          verbose: Batch download error id=Head, startEpoch=1242, status=Downloading, peer=16...2dKKZW, code=REQUEST_ERROR_INVALID_REQUEST, errorMessage=sNaPpYuV~2rate limited |  
  |   | 2025-07-03 20:34:01.638 | Jul-03 13:34:01.637[sync]          verbose: Batch download error id=Head, startEpoch=1241, status=Downloading, peer=16...2dKKZW, code=REQUEST_ERROR_INVALID_REQUEST, errorMessage=sNaPpYuV~2rate limited |  
  |   | 2025-07-03 20:34:01.627 | Jul-03 13:34:01.627[sync]          verbose: Batch download error id=Head, startEpoch=1236, status=Downloading, peer=16...2dKKZW, code=REQUEST_ERROR_INVALID_REQUEST, errorMessage=sNaPpYuV~2rate limited |  
  |   | 2025-07-03 20:34:01.551 | Jul-03 13:34:01.551[sync]          verbose: Batch download error id=Head, startEpoch=1239, status=Downloading, peer=16...2dKKZW, code=REQUEST_ERROR_INVALID_REQUEST, errorMessage=sNaPpYuV~2rate limited |  
  |   | 2025-07-03 20:34:01.551 | Jul-03 13:34:01.551[sync]          verbose: Batch download error id=Head, startEpoch=1237, status=Downloading, peer=16...2dKKZW, code=REQUEST_ERROR_INVALID_REQUEST, errorMessage=sNaPpYuV~2rate limited |  
  |   | 2025-07-03 20:34:01.550 | Jul-03 13:34:01.550[sync]          verbose: Batch download error id=Head, startEpoch=1235, status=Downloading, peer=16...2dKKZW, code=REQUEST_ERROR_INVALID_REQUEST, errorMessage=sNaPpYuV~2rate limited |  
  |   | 2025-07-03 20:34:01.549 | Jul-03 13:34:01.549[sync]          verbose: Batch download error id=Head, startEpoch=1238, status=Downloading, peer=16...2dKKZW, code=REQUEST_ERROR_INVALID_REQUEST, errorMessage=sNaPpYuV~2rate limited |  
  |   | 2025-07-03 20:34:01.549 | Jul-03 13:34:01.548[sync]          verbose: Batch download error id=Head, startEpoch=1233, status=Downloading, peer=16...2dKKZW, code=REQUEST_ERROR_INVALID_REQUEST, errorMessage=sNaPpYuV~2rate limited |  
  |   | 2025-07-03 20:34:01.548 | Jul-03 13:34:01.548[sync]          verbose: Batch download error id=Head, startEpoch=1232, status=Downloading, peer=16...2dKKZW, code=REQUEST_ERROR_INVALID_REQUEST, errorMessage=sNaPpYuV~2rate limited

...

there are are logs like:

2025-07-03 20:34:22.740	
Jul-03 13:34:22.740[sync]          verbose: Batch download error id=Head, startEpoch=1199, status=Downloading, peer=16...bA5Tw4, code=REQUEST_ERROR_UNKNOWN_ERROR_STATUS, errorMessage=HsNaPpYLX:Rate limited. There are already 2 active requests with the same protocol, status=139

Expected behavior

we should manage how many active requests in a peer
also it seems we download batches too aggressive, we should only download n epochs ahead

Steps to reproduce

No response

Additional context

No response

Operating system

Linux

Lodestar version or commit hash

peerDAS

[twoeths]

we also have this error:

2025-07-03 20:34:41.440 | Jul-03 13:34:41.439[sync]          verbose: Batch download error id=Head, startEpoch=1220, status=Downloading, peer=16...2dKKZW, code=REQUEST_ERROR_TTFB_TIMEOUT |  
-- | -- | --
  |   | 2025-07-03 20:34:41.439 | Jul-03 13:34:41.439[sync]          verbose: Batch download error id=Head, startEpoch=1228, status=Downloading, peer=16...2dKKZW, code=REQUEST_ERROR_TTFB_TIMEOUT |  
  |   | 2025-07-03 20:34:41.438 | Jul-03 13:34:41.437[sync]          verbose: Batch download error id=Head, startEpoch=1225, status=Downloading, peer=16...2dKKZW, code=REQUEST_ERROR_TTFB_TIMEOUT |  
  |   | 2025-07-03 20:34:41.435 | Jul-03 13:34:41.434[sync]          verbose: Batch download error id=Head, startEpoch=1221, status=Downloading, peer=16...2dKKZW, code=REQUEST_ERROR_TTFB_TIMEOUT |  
  |   | 2025-07-03 20:34:41.433 | Jul-03 13:34:41.433[sync]          verbose: Batch download error id=Head, startEpoch=1217, status=Downloading, peer=16...2dKKZW, code=REQUEST_ERROR_TTFB_TIMEOUT |  
  |   | 2025-07-03 20:34:41.106 | Jul-03 13:34:41.106[sync]          verbose: Batch download error id=Head, startEpoch=1218, status=Downloading, peer=16...2dKKZW, code=REQUEST_ERROR_TTFB_TIMEOUT |  
  |   | 2025-07-03 20:34:41.106 | Jul-03 13:34:41.106[sync]          verbose: Batch download error id=Head, startEpoch=1214, status=Downloading, peer=16...2dKKZW, code=REQUEST_ERROR_TTFB_TIMEOUT |  
  |   | 2025-07-03 20:34:41.105 | Jul-03 13:34:41.105[sync]          verbose: Batch download error id=Head, startEpoch=1213, status=Downloading, peer=16...2dKKZW, code=REQUEST_ERROR_TTFB_TIMEOUT

need to also implement https://github.com/ethereum/consensus-specs/pull/3767/files

[philknows]

Boosted priority on this one based on what we're seeing on the devnet-2 with syncing.

[wemeetagain]

don't forget to update https://github.com/ChainSafe/lodestar/pull/6353/files#diff-22c2f542800fffa1b6e11b7b5de8872d7510f8b960e51233cc8200796a8fc334R10
when resolving this issue

[wemeetagain]

welp... I have no idea what that link used to point to.... 🤷

[twoeths]

closed via #8185