Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement PeerDAS #14129

Draft
wants to merge 112 commits into
base: develop
Choose a base branch
from
Draft

Implement PeerDAS #14129

wants to merge 112 commits into from

Conversation

nalepae
Copy link
Contributor

@nalepae nalepae commented Jun 21, 2024 •
edited
Loading

Reading:

Peerdas from scratch

Remaining tasks:

  • Initial sync: Backfill data columns when starting from a checkpoint.
  • Validator custody: Implement, and specially implement the backfill. (In progress @nalepae)
  • Chaos testing: Indroduce a nasty peer flag that holds AND/OR (both are important) alterate data columns on RPC responses. (In progress 0xPC02)
  • Unit tests: Add them where they are not yet added.
  • KZG library: Replace the cKZG library by the goKZGone. (In progress @nalepae @kevaundray)
  • Metrics: Add PeerDAS specific metrics In progress (@KatyaRyazantseva)
  • Req/Resp: Ensure we Req / Resp (by root and by range) everywhere using not only super nodes. ( GetValidCustodyPeers)
  • prysmctl: Implement byRoot and byRange data columns requests
  • ENR (not peerDAS related) Find why sometimes ENR records are not correctly updated via discovery.
  • getBlobV1: Implement reconstructAndBroadcastBlobs for data columns.
  • Memory leak: Find it.

Remaining open questions:

  • Initial sync: Should we request max 64 columns and the reconstruct instead of requesting > 64 columns when needed?
  • ENR: Are we sure the sequence number cannot go backward after a reboot? (This trigs downscoring on Prysm.)
  • Subnet subscription (not peerDAS related): Should we subscribe to subnets / find peers 1 epoch in advance for getSubnetsToSubscribe and getSubnetsToFindPeersOnly?
  • Subnet unsubscription (not peerDAS related): Handle the same way dynamic (fork epoch) and non dynamic (fork epoch + 1) unsubscriptions.
  • Peer ban: What happens if a peer announces less custody groups via Metadata/ENR than the minimal requirement? Ban?

@nalepae nalepae added peerDAS Blocked Blocked by research or external factors labels Jun 21, 2024
@nalepae nalepae self-assigned this Jun 21, 2024
@nisdas nisdas force-pushed the peerDAS branch 3 times, most recently from dc39693 to 3fb1fa4 Compare August 20, 2024 09:41
@nisdas nisdas mentioned this pull request Aug 23, 2024
Merged
@nalepae nalepae force-pushed the peerDAS branch 2 times, most recently from f2dea1c to bd7ec3f Compare August 28, 2024 11:21
@KatyaRyazantseva KatyaRyazantseva mentioned this pull request Oct 3, 2024
Draft
@nalepae nalepae force-pushed the peerDAS branch 2 times, most recently from 1488044 to ddea2d7 Compare October 8, 2024 20:38
@KatyaRyazantseva KatyaRyazantseva mentioned this pull request Oct 23, 2024
Closed
3 tasks
@nalepae nalepae force-pushed the peerDAS branch 3 times, most recently from e92cde5 to 2dd4302 Compare November 25, 2024 14:06
nalepae and others added 11 commits November 27, 2024 10:11
@nalepae
Add in column sidecars protos (#13862)
c688c84
@nisdas @nalepae
add it (#13865)
3e23f6e
@nisdas @nalepae
Add Support For Discovery Of Column Subnets (#13883)
9ffc19d 
* Add Support For Discovery Of Column Subnets

* Lint for SubnetsPerNode

* Manu's Review

* Change to a better name
@nisdas @nalepae
Add Data Column Gossip Handlers (#13894)
d844026 
* Add Data Column Subscriber

* Add Data Column Vaidator

* Wire all Handlers In

* Fix Build

* Fix Test

* Fix IP in Test

* Fix IP in Test
@nisdas @nalepae
Add Request And Response RPC Methods For Data Columns (#13909)
3e722ea 
* Add RPC Handler

* Add Column Requests

* Update beacon-chain/db/filesystem/blob.go

Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>

* Update beacon-chain/p2p/rpc_topic_mappings.go

Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>

* Manu's Review

* Manu's Review

* Interface Fixes

* mock manager

---------

Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
@nalepae
Peer das core (#13877)
1bfbd39 
* Bump `c-kzg-4844` lib to the `das` branch.

* Implement `MerkleProofKZGCommitments`.

* Implement `das-core.md`.

* Use `peerdas.CustodyColumnSubnets` and `peerdas.CustodyColumns`.

* `CustodyColumnSubnets`: Include `i` in the for loop.

* Remove `computeSubscribedColumnSubnet`.

* Remove `peerdas.CustodyColumns` out of the for loop.
@nalepae
Implement custody_subnet_count ENR field. (#13915)
f503efc 
https://github.com/ethereum/consensus-specs/blob/dev/specs/_features/eip7594/p2p-interface.md#the-discovery-domain-discv5
@nisdas @nalepae
Update .bazelrc (#13931)
b78c348
@nalepae
Implement peer DAS proposer RPC (#13922)
1355178 
* Remove capital letter from error messages.

* `[4]byte` => `[fieldparams.VersionLength]byte`.

* Prometheus: Remove extra `committee`.

They are probably due to a bad copy/paste.

Note: The name of the probe itself is remaining,
to ensure backward compatibility.

* Implement Proposer RPC for data columns.

* Fix TestProposer_ProposeBlock_OK test.

* Remove default peerDAS activation.

* `validateDataColumn`: Workaround to return a `VerifiedRODataColumn`
@nisdas @nalepae
Add DA Check For Data Columns (#13938)
be0580e 
* Add new DA check

* Exit early in the event no commitments exist.

* Gazelle

* Fix Mock Broadcaster

* Fix Test Setup

* Update beacon-chain/blockchain/process_block.go

Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>

* Manu's Review

* Fix Build

---------

Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
@nalepae
Spectests (#13940)
dcbb543 
* Update `consensus_spec_version` to `v1.5.0-alpha.1`.

* `CustodyColumns`: Fix and implement spec tests.

* Make deepsource happy.

* `^uint64(0)` => `math.MaxUint64`.

* Fix `TestLoadConfigFile` test.
nalepae added 5 commits January 5, 2025 16:14
@nalepae
Prepare for future fork boilerplate.
8765197
@nalepae
Implement the Fulu fork boilerplate.
d8e09c3
@nalepae
Upgraded state to <fork> log: Move from debug to info.
bcfaff8 
Rationale:
This log is the only one notifying the user a new fork happened.
A new fork is always a little bit stressful for a node operator.
Having at least one log indicating the client switched fork is something useful.
@nalepae
Merge branch 'fulu-boilerplate' into peerDAS
a082d2a
@nalepae
Merge branch 'develop' into peerDAS
5f08559
@KatyaRyazantseva KatyaRyazantseva mentioned this pull request Jan 13, 2025
Merged
3 tasks
@KatyaRyazantseva KatyaRyazantseva mentioned this pull request Jan 15, 2025
Merged
3 tasks
@nalepae nalepae force-pushed the peerDAS branch 3 times, most recently from 30a32c4 to d7d370e Compare January 16, 2025 08:54
@KatyaRyazantseva KatyaRyazantseva mentioned this pull request Jan 16, 2025
Open
3 tasks
This was referenced Jan 20, 2025
Merged
Open
KatyaRyazantseva and others added 11 commits January 21, 2025 16:14
@KatyaRyazantseva
Add 'beacon' prefix to 'data_column_sidecar_computation' metric (#14790)
805ee1b
@KatyaRyazantseva
Add PeerDAS gossip verification metrics (#14796)
7faee5a
@KatyaRyazantseva
Add PeerDAS kzg and inclusion proof verification metrics (#14814)
19662da
@nalepae
Merge branch 'develop' into peerDAS
242c2b0
@nalepae
Merge branch 'develop' into peerDAS
632a060
@nalepae
reconstructAndBroadcastBlobs: Temporarily deactivate starting at Fulu.
ad11036
@nalepae
Sync: Integrate batch directly in buildBwbSlices. (#14843)
14f93b4 
Previously, `buildBwbSlices` were built, and then only to big requests were batched in `buildDataColumnSidecarsByRangeRequests`.

In some edge cases, this lead to requesting data columns to peers for blocks with no blobs.

Splitting by batch directly in `buildBwbSlices` fixes the issue.
@nalepae
Peer das sync empty requests (#14854)
01705d1 
* `TestBuildBwbSlices`: Add test case failing with the current implementation.

* Fix `buildBwbSlices` to comply with the new test case.

* `block_fetchers.go`: Improve logging and godoc.

* `DataColumnsRPCMinValidSlot`: Update to Fulu.
@nalepae
Merge branch 'develop' into peerDAS
a216cb4
@nalepae
Merge branch 'develop' into peerDAS
0923145
@nalepae
Avoid computing peerDAS info again and again. (#14893)
ac04246 
* `areDataColumnsAvailable`: `signed` ==> `signedBlock`.

* peerdas: Split `helpers.go` in multiple files respecting the specification.

* peerDAS: Implement `Info`.

* peerDAS: Use cached `Info` when possible.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@nalepae nalepae

Labels
Blocked Blocked by research or external factors peerDAS
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@nalepae @nisdas @jtraglia @0x00101010 @kevaundray @KatyaRyazantseva