feat: implement peerDAS on fulu#6353
Merged
Merged
Conversation
g11tech
force-pushed
the
electra-fork
branch
from
fc1990a to
9014c8c
Compare
g11tech
force-pushed
the
electra-fork
branch
from
488d853 to
4e98a4a
Compare
g11tech
force-pushed
the
electra-fork
branch
2 times, most recently
from
8a4f66e to
473a04f
Compare
Contributor
Performance Report✔️ no performance regression detected Full benchmark results
|
g11tech
force-pushed
the
peerDAS
branch
4 times, most recently
from
5fbbdb2 to
81aaeb5
Compare
matthewkeil
reviewed
Sep 17, 2024
commit e8bc729 Author: Matthew Keil <me@matthewkeil.com> Date: Tue Dec 3 10:08:30 2024 -0500 refactor: peerdas types (#7243) * refactor: organize peerDAS types * refactor: DataColumnsData * refactor: rename BlockInputBlobs BlockInputColumnData * refactor: split up and rename BlockInputData * refactor: clean up BlobsData * refactor: clean up CachedData types * refactor: change from interface to type and update enum values for grafana * chore: lint * fix: remove extraneous lint fix commit c8075d0 Author: Matthew Keil <me@matthewkeil.com> Date: Mon Nov 25 15:41:41 2024 +0800 feat: log peer disconnect info (#7231) * feat: log disconnect reason * feat: log peerScore update * fix: pretty print peerId * fix: use prettyPrintPeerId commit 8689c76 Author: Matthew Keil <me@matthewkeil.com> Date: Tue Oct 22 06:11:13 2024 -0400 feat: check for no commitments on block or column in sidecar validation (#7184) * feat: check for no commitments on block or column in sidecar validation * test: add sanity check for empty blob commitments in column validation * fix: test bug * fix: but in test passing commitments commit fccf9a2 Author: harkamal <gajinder@g11.in> Date: Tue Oct 8 12:44:11 2024 +0530 add prevdownload tracker commit 513bccc Author: harkamal <gajinder@g11.in> Date: Wed Oct 2 16:46:40 2024 +0530 fix commit 1c08ab3 Author: harkamal <gajinder@g11.in> Date: Wed Oct 2 16:40:40 2024 +0530 some fixing of beacon params commit 7c9a01c Author: harkamal <gajinder@g11.in> Date: Wed Oct 2 14:58:11 2024 +0530 add debug and fix datacolumns migration and improve log commit b04aaef Author: harkamal <gajinder@g11.in> Date: Wed Oct 2 12:52:24 2024 +0530 migrate datacolumns to finalized commit a0e0087 Author: harkamal <gajinder@g11.in> Date: Wed Oct 2 00:36:03 2024 +0530 more log commit 2736b8c Author: harkamal <gajinder@g11.in> Date: Tue Oct 1 19:43:59 2024 +0530 add enhance datacolumn serving logs commit cce193b Author: harkamal <gajinder@g11.in> Date: Tue Oct 1 17:36:14 2024 +0530 improve logging for debugging commit a3de70f Author: harkamal <gajinder@g11.in> Date: Thu Sep 26 14:43:49 2024 +0530 turn persisting network identity to default true commit cec27d6 Author: harkamal <gajinder@g11.in> Date: Sat Sep 21 22:07:58 2024 +0530 handle edge case commit 6a77828 Author: harkamal <gajinder@g11.in> Date: Sat Sep 21 20:33:53 2024 +0530 add debug console log commit 574837a Author: harkamal <gajinder@g11.in> Date: Sat Sep 21 18:39:36 2024 +0530 use sample subnets for data availability commit fee7c08 Author: harkamal <gajinder@g11.in> Date: Tue Sep 17 16:42:45 2024 +0530 validate inclusion proof commit 20ef4c6 Author: Matthew Keil <me@matthewkeil.com> Date: Tue Sep 17 05:33:40 2024 -0400 feat: validate data column sidecars (#7073) * feat: update c-kzg to final DAS version * refactor: use trusted-setup from c-kzg package * feat: implement validateDataColumnsSidecars * feat: check block and column commitments match * test: add unit test for validateDataColumnsSidecars * fix: invalid build and update validity condition of validateDataColumnsSidecars * fix: make error messages better * fix: electra vs peerdas type conflict commit b1940ee Author: Matthew Keil <me@matthewkeil.com> Date: Tue Sep 17 05:26:05 2024 -0400 fix: remove ckzg build script (#7089) * fix: remove unused ckzg build script * fix: remove unused rsync dep from Dockerfile commit bd4f7f9 Author: Matthew Keil <me@matthewkeil.com> Date: Mon Sep 16 06:31:52 2024 -0400 feat: update ckzg to final DAS version (#7050) * feat: update c-kzg to final DAS version * refactor: use trusted-setup from c-kzg package commit a33303f Author: Matthew Keil <me@matthewkeil.com> Date: Mon Sep 16 04:08:29 2024 -0400 feat: refactor and unit test getDataColumnSidecars (#7072) * refactor: getDataColumnSidecars * test: unit test getDataColumnSidecars with mocks from c-kzg library * refactor: use fromHex util * chore: update numbering on mocks * chore: update c-kzg to latest version * chore: fix type export syntax * test: add verification for cells from sidecars * test: add verification to DataColumnSidecars tests * refactor: getDataColumnSidecars for PR comments * feat: narrow type and remove unnecessary conditional * fix: getDataColumnSidecars param type * refactor: rename to computeDataColumnSidecars commit 4ec7aff Author: harkamal <gajinder@g11.in> Date: Fri Sep 13 18:19:45 2024 +0530 edge case optimization commit 3470076 Author: harkamal <gajinder@g11.in> Date: Fri Sep 13 17:31:52 2024 +0530 more debug log commit c4d04ee Author: harkamal <gajinder@g11.in> Date: Fri Sep 13 15:35:32 2024 +0530 fix the column id compute commit cdd9bae Author: harkamal <gajinder@g11.in> Date: Thu Sep 12 17:04:07 2024 +0530 update compute spec tests commit 2b10e4d Author: harkamal <gajinder@g11.in> Date: Thu Sep 12 15:25:00 2024 +0530 datacolumns retrival fix commit 56c8c6e Author: harkamal <gajinder@g11.in> Date: Thu Sep 12 13:57:51 2024 +0530 custodied column fetch debugging log commit af933fb Author: harkamal <gajinder@g11.in> Date: Wed Sep 11 22:11:16 2024 +0530 some fixes commit 74d8122 Author: harkamal <gajinder@g11.in> Date: Wed Sep 11 21:57:41 2024 +0530 add some log for debugging inbound data columns request commit f7571f4 Author: harkamal <gajinder@g11.in> Date: Wed Sep 11 17:07:52 2024 +0530 add some more loggig and availaibility tracking commit d35873e Author: harkamal <gajinder@g11.in> Date: Wed Sep 11 15:02:31 2024 +0530 further wait till cutoff for all data to be available commit 8c21168 Author: harkamal <gajinder@g11.in> Date: Wed Sep 11 00:02:09 2024 +0530 make pull a little less agressive commit de341b5 Author: harkamal <gajinder@g11.in> Date: Tue Sep 10 22:21:37 2024 +0530 add send more log commit bd84892 Author: harkamal <gajinder@g11.in> Date: Tue Sep 10 22:01:43 2024 +0530 more log commit d7721f8 Author: harkamal <gajinder@g11.in> Date: Tue Sep 10 21:19:11 2024 +0530 fix bug commit 5e1de6f Author: harkamal <gajinder@g11.in> Date: Tue Sep 10 20:45:05 2024 +0530 trying some fix commit 2bc1a0d Author: harkamal <gajinder@g11.in> Date: Tue Sep 10 20:11:05 2024 +0530 add cache tracking commit 387da88 Author: harkamal <gajinder@g11.in> Date: Tue Sep 10 19:27:05 2024 +0530 add more log commit aece0ab Author: harkamal <gajinder@g11.in> Date: Tue Sep 10 17:46:52 2024 +0530 fix add missing data availability resolutions commit 006e781 Author: harkamal <gajinder@g11.in> Date: Sat Sep 7 20:35:00 2024 +0530 add debug log commit bf08852 Author: harkamal <gajinder@g11.in> Date: Sat Sep 7 20:08:10 2024 +0530 resolve availability when datacolumns are downloaded and matched commit 2833ac0 Author: harkamal <gajinder@g11.in> Date: Thu Sep 5 20:17:04 2024 +0530 make the csc encoding updates as per latest spec commit 585165e Author: harkamal <gajinder@g11.in> Date: Wed Aug 28 17:35:31 2024 +0530 remove banning unknown block, addmore log commit a33a72f Author: harkamal <gajinder@g11.in> Date: Tue Aug 27 21:59:03 2024 +0530 subnet count 128 commit ae7678e Author: harkamal <gajinder@g11.in> Date: Tue Aug 27 21:29:50 2024 +0530 fx commit 4b6f167 Author: harkamal <gajinder@g11.in> Date: Tue Aug 27 19:37:37 2024 +0530 fix bug commit 180f7d8 Author: harkamal <gajinder@g11.in> Date: Tue Aug 27 18:58:26 2024 +0530 fix log commit 54579b0 Author: harkamal <gajinder@g11.in> Date: Tue Aug 27 17:21:07 2024 +0530 add more info for debugging commit a3533f8 Author: harkamal <gajinder@g11.in> Date: Tue Aug 27 17:13:41 2024 +0530 add supernode flag to configure node custody requirement and make it not required for validator commit e6c613f Author: harkamal <gajinder@g11.in> Date: Tue Aug 27 15:51:20 2024 +0530 rename electra fork to peerdas for rebase and make csc in metadata uint8 commit 81aaeb5 Author: harkamal <gajinder@g11.in> Date: Mon Aug 12 15:43:39 2024 +0530 feat: add and use metadatav3 for peer custody subnet fixes for metadata, working locally change the condition to update metadata csc change commit c7f6341 Author: harkamal <gajinder@g11.in> Date: Fri Aug 9 17:06:52 2024 +0530 fix the types/test rebase fixes commit a0c5d27 Author: harkamal <gajinder@g11.in> Date: Tue Jul 16 18:54:18 2024 +0530 fix: refactor to add and use nodeid computation and clear out nodeid tracking nodeid cleanup for network commit d423004 Author: harkamal <gajinder@g11.in> Date: Mon Jul 15 03:12:52 2024 +0530 feat: add the modifications to work with devnet2 some network options to control peering behavior allow setting node custody capability via --params use eip754 names for the peerdas config commit 47eedae Author: harkamal <gajinder@g11.in> Date: Sun Jul 14 17:56:59 2024 +0530 feat: get various sync mechanisms working with/without sharded data commit 156ef53 Author: matthewkeil <me@matthewkeil.com> Date: Fri Jun 21 17:09:44 2024 +0200 fix: docker build issue for c-kzg wip: REPLACE THIS COMMIT commit yarn lock rebase fixes fix: update c-zkg install workflow feat: add trustedSetupPrecompute cli flag fix: update trusted-setup for testing fix: update c-zkg install workflow to remove sudo fix: add rsync to apk deps commit 499d93c Author: harkamal <gajinder@g11.in> Date: Wed Jan 24 18:40:25 2024 +0530 feat: implement peerDAS on electra add some presets add further params and types add data column to types repo and network move to max request data columns to preset add the datacolumns data in blockinput and fix breaking errors in seen gossip blockinput handle data columns in gossip and the seengossip further propagate forkaware blockdata and resolve build/type issues further handle datacolumns sync by range by root and forkaware data handling fix issues chore: update c-kzg to peerDas version feat: add peerDas ckzg functions to interface fix the lookups handle the publishing flow various sync try fixes fixes compute blob side car various misl debuggings and fixes debug and apply fixes and get range and by root sync to work will full custody enable syncing with lower custody requirement use node peerid rather than a dummy string get and use the nodeid from enr and correctly compute subnets and column indexes filterout and connect to peers only matching out custody requiremnt try adding custody requirement add protection for subnet calc get the sync working with devnet 0 correctly set the enr with custody subnet info rebase fixes small refactor commit 4805a2e Author: harkamal <gajinder@g11.in> Date: Wed Jan 24 17:38:11 2024 +0530 feat: placeholder PR for electra add types stub and epoch config fix types
**Motivation** Addresses peerDAS PR comment #6353 (comment)
Merged
**Motivation** Addresses peerDAS PR comments #6353 (comment) #6353 (comment) #6353 (comment)
**Motivation** Addresses peerDAS PR comment #6353 (comment)
#8159) **Motivation** Addresses peerDAS PR comment #6353 (comment) --------- Co-authored-by: Nico Flaig <nflaig@protonmail.com>
**Motivation** Addresses peerDAS PR comments #6353 (comment) #6353 (comment) #6353 (comment) #6353 (comment) #6353 (comment) #6353 (comment) #6353 (comment) #6353 (comment) #6353 (comment) #6353 (comment) --------- Co-authored-by: Nico Flaig <nflaig@protonmail.com>
**Motivation** Addresses peerDAS PR comments #6353 (comment) #6353 (comment) #6353 (comment) --------- Co-authored-by: Nico Flaig <nflaig@protonmail.com>
**Motivation** Slight optimization from ethereum/consensus-specs#4459 to skip custody groups computation if we custody all groups. **Description** - throw error (instead of hidding a bug) if `custodyGroupCount > NUMBER_OF_CUSTODY_GROUPS` - skip computation if all groups are custodied
**Motivation** Addresses peerDAS PR comments #6353 (comment) #6353 (comment) #6353 (comment) #6353 (comment) #6353 (comment) --------- Co-authored-by: Nico Flaig <nflaig@protonmail.com>
**Motivation** Revert a commit adding a metric. Create a separate issue to fix more thoroughly
**Motivation** - some peers consistently sent mismatched DataColumnSidecars, we should gradually reduce peer score and eventually disconnect it **Description** - reduce peer score by using `network.reportPeer()` api Closes #8133 --------- Co-authored-by: Tuyen Nguyen <twoeths@users.noreply.github.com> Co-authored-by: Cayman <caymannava@gmail.com> Co-authored-by: Nico Flaig <nflaig@protonmail.com>
…oice spec test (#8170) Passes all tests in `v1.6.0-alpha.4`
**Motivation** Addresses peerDAS pr comments #6353 (comment)
**Motivation** - we got a lot of rate limited response from peers during syncing - we should be able to control which peers to connect to based on tracked active requests in PeerBalancer **Description** - enforce no more than MAX_CONCURRENT_REQUESTS per peer in sync - limit number of epochs downloaded ahead part of #8033 **Test results on fusaka-devnet-3** was able to sync `fusaka-devnet-3` 2 times using this branch along with #8150 <img width="851" height="346" alt="Screenshot 2025-08-09 at 14 45 32" src="https://github.com/user-attachments/assets/1f3afca0-13a6-4f7a-9c18-51d03cc34793" /> --------- Co-authored-by: Tuyen Nguyen <twoeths@users.noreply.github.com> Co-authored-by: Cayman <caymannava@gmail.com>
**Motivation** - sync chain was stuck because some peers incorrectly response 0 block for a batch when they don't have it, see #8147 - batch `n` has 0 block so chain does not process anything (and note that this is wrong response from peers, most likely peer started from a `checkpointSyncUrl` - batch `n+1` cannot process due to `UNKNOWN_BLOCK_PARENT` - then SyncChain stuck - sometimes other clients throw `ResourceUnavailable` instead, and that's expected in the spec **Description** - set `MAX_BATCH_PROCESSING_ATTEMPTS` to 0, ie if we cannot process a batch, we remove that sync chain and RangeSync will automatically add a new one - do not sync from peers without `earliestAvailableSlot` - throw error if peers return 0 blocks (peers should have returned `ResourceUnavailable` error in that case according to [the spec](https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/p2p-interface.md#responding-side), we handle just in case. For lodestar, I tracked that in #8149) Closes #8147 was able to sync `fusaka-devnet-3` 2 times using this branch along with #8166 <img width="851" height="346" alt="Screenshot 2025-08-09 at 14 45 32" src="https://github.com/user-attachments/assets/1f3afca0-13a6-4f7a-9c18-51d03cc34793" /> --------- Co-authored-by: Tuyen Nguyen <twoeths@users.noreply.github.com>
| const hasAllColumns = matchingSubnetsNum === sampleSubnets.length; | ||
| const clientAgent = peerData?.agentClient ?? ClientKind.Unknown; | ||
|
|
||
| this.logger.debug("onStatus", { |
Contributor
There was a problem hiding this comment.
Previously has been onStatus ${custodyGroupCount === undefined ? "undefined custody count assuming 4" : ""}, related to #8092
**Motivation** Addresses `peerDAS` PR comment #6353 (comment) Also clean up usage (emission and consumption) of the `peer_connected` event. Add helper function to `prettyPrintCustodyGroups`
**Motivation** - fix unit tests and e2e tests of `peerDAS` branch **Description** - when sending status, based on its body we set correct version, otherwise peers cannot deserialize the request body - at fulu fork transition, update local status cache so that it sends the correct version of status message - fix failed unit test as introduced by [PR-8116](#8116 (comment)) --------- Co-authored-by: Tuyen Nguyen <twoeths@users.noreply.github.com> Co-authored-by: Nico Flaig <nflaig@protonmail.com> Co-authored-by: Cayman <caymannava@gmail.com>
nflaig
approved these changes
Aug 11, 2025
wemeetagain
approved these changes
Aug 11, 2025
matthewkeil
approved these changes
Aug 11, 2025
matthewkeil
left a comment
matthewkeil
left a comment
Member
There was a problem hiding this comment.
no sleep for you!!!
and LGTM!!! 🚀
up up and away Ethereum!! Go zebra go!!
wemeetagain
added a commit
that referenced
this pull request
Aug 12, 2025
**Motivation** Addresses peerDAS PR comment #6353 (comment) --------- Co-authored-by: Cayman <caymannava@gmail.com>
wemeetagain
added a commit
that referenced
this pull request
Aug 15, 2025
**Motivation** - address #6353 (comment) **Description** Our block production and publishing data flow was messy. The data types and utility functions are jumbled, duplicate functions, differences from spec functions, etc. To that end, this refactors things that touch this proposal data flow. - Tweak the fork-agnostic `BlockContents` type to also include pre-deneb `BeaconBlock` - this allows us to remove `BlockOrContents` variables and types - Remove `Contents` (It isn't actually needed) - Collect all `produced*` caches into `blockProductionCache` - make blinded block reconstruction more typesafe - Separate blobs+proofs sanity validation from cells+proof sanity validation - Delete duplicated `BlobsBundle` type - Remove duplicate spec function `computeDataColumnSidecars` in favor of `getDataColumnSidecars*` functions - Refactor `computeBlobSidecars` into `getBlobSidecars` from the spec --------- Co-authored-by: Nico Flaig <nflaig@protonmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
13 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
implement :
lighthouse impl for reference: