Investigate whether libp2p should be used as the foundational network layer

[deblasis]

This discussion is meant to highlight pros/cons about using libp2p as a foundational network layer.

I'll personally research the topic, present my findings in here and open up the floor for discussion.

Origin

#275 (comment)

Research SPIKE: pokt-network/pocket-network-protocol#17 (comment)

[jessicadaugherty]

Documentation/past research from V1 P2P module initial scope: https://docs.google.com/document/d/10rHFr8HpUfpmHzXbyZWzq60aJDY1lYMR9kHNgdfgbNE/edit#heading=h.e1rj36wlfr45

[deblasis]

I have shared here a "light" version of my research. I will update this discussion with the final version as soon as it is completed but I am happy to answer any direct question in the meantime. Not because I am an expert but because my aim is to gain as much knowledge as possible especially about the parts that seem the least clear.

A couple of things I could say, citing the past research:

Libp2p was built with modularity in mind and as a main purpose, and was very much extracted out of ipfs’s stack, hence focuses on code modularity, and goes with an all round solution with no specific specialization or focus on a particular problem. Surely, there is a bias towards solutions adapted for distributed file systems, which is not obviously a good or not a so good fit for pocket.

Agreed, but we shouldn't forget that modularity allows us to swap implementations with something custom.

Libp2p is a big piece of software to own, and also given the addressing logic, any network using libp2p ends up becoming a subnetwork or the libp2p overlay, and there are numerous potential issues that could arise. (Parity has previously poisoned the ipfs’s network due to this)

I would say that the last part is incorrect. In the video what's mentioned is an "accidental DHT poisoning attack" caused by the fact that IPFS and substrate, accidentally merged, ending up having each other's values being propagated.

We already identified that the Kademlia DHT is not fit for purpose in our use-case, we are not yet even sure if we need a DHT at all depending on how we choose to implement our routing.

[Olshansk]

ending up having each other's values being propagated.

👍 Just wanted to call out "LibP2P - Consolidate Node/Peer ID with LibP2P Peer" (#348), as we'll need to make sure the ID in LibP2P, P2P, Consensus, etc... doesn't get mixed up. I think there is a lot of room for error here.

Simplifying the codebase will very much help resolve this too but I acknowledge ahead of time it's not an easy task.

We already identified that the Kademlia DHT

Can you point me to the reasoning here?

we are not yet even sure if we need a DHT at all depending on how we choose to implement our routing

Agreed. Wanted to point you to #339 where @jeremybeal11 asked some questions about RainTree and I responded with this:

[Olshansk]

tl;dr The core team decided to use LibP2P's network layer primitives, concepts and interfaces as a reference without directly integrating with LibP2P for the time being.

Reasons in favour of using LibP2P:

• Battle tested in production
• A lot of effort, time and contributions from experienced networking developers
• Recommendations from other projects suggesting their networking & security layer (not necessarily peer discovery & routing)
• Mature and active project; note that this can also be true in the case of low-quality projects

Reasons against using LibP2P:

• Library is large & complex as a result of many contributions and being made more general purpose
• It is difficult, if not impossible, for core Pocket team members to have the time to become experts in P2P
• Making pocket-specific changes by forking and maintaining a fork can get problematic. Merging changes upstream is also outside of our control.

Next steps:

1. Identify the primitives and interfaces in LibP2P's primitives related to networking routing and transport
2. Mirror the results of (1) in Pocket (i.e. copy-pasta)

Benefits of these steps:

1. Pocket developers become familiar with LibP2P interfaces & concepts
2. Pocket continues to own the whole stack end-to-end
3. If/when need be (e.g. TestNet timeline), it'll be easy to swap LibP2P in or out
4. Avoiding the import of a large dependency can avoid slowing things down in the short/mid-term by needing to onboard onto a new library

References

• Research spike: [SPIKE][P2P] Configuration - Research LibP2P pocket-network-protocol#17
• Implementation: [P2P] Integrate LibP2P #347