Add network agent architecture layer

[iduartgomez]

This is a broad issue to track any work related to the network layer. This pertains to:

• Specification of whatever communication protocol between agents.
• Implementation of such protocol.
• Set up of network topologies of agents.
• Peer discovery.
  etc.

[iduartgomez]

For communication a P2P network should be implemented.

The preferred transport protocol for it's attributes is QUIC, currently there are some implementations of QUIC in Rust:

• quinn, an async-friendly implementation on top of Tokio, quinn-protois executor agnostic and a custom event loop has to be provided.
• quiche, from Cloudflare
• neqo, from Mozilla

quinnis the highest level one, the proto subcrate would be on par with quiche as far as low level and can be used as a building block.

QUIC has some nice properties: efficient streams, native encryption, low latency and high throughput. This makes it ideal to build a RCP framework on-top of it which can be then used to handle communication between agents over a public network.

For peer routing Kademlia is the preferred data-structure/algorithm, libp2p.io has a working Rust implementation of it, as well as other useful building blocks. Unfortunately the QUIC protocol has not yet been added, although a PR is underway (libp2p/rust-libp2p#1334). Other building blocks that can be used from libp2p are the modules for peer discovery, streams, TLS. How could this be tied with quinn/quiche has to be explored (the undergoing PR uses quinn under the hood).

Other options: https://github.com/maidsafe/quic-p2p (does not seem mature enough?)

[iduartgomez]

Since transport layer is interchangable and libp2p transport protocols are implemented through traits could start working with tcp and move to QUIC later on.

[iduartgomez]

Some work done so far:

• Write the overarching architecture for handling configuration and building a network.
• Write the generic protocol to handle the different parts of connection, verification and messaging:
  • Write a protocol over libp2p to handle message passing between agents.
  • Delegate peer discovery to a kademlia DHT.
  • Custom generic serialization/deserialization of messages between peers. Messages can be optionally signed.

Ongoing: Groups, topic subscription and broadcasting.

The overall architecture is as follows:

• Following the patterns established in p2p the transport layer is separated, and handled entirely by it (so each peer can establish a connection via TCP or any other supported protocol; for now all testing is done directly through TCP. Security is also handled automatically; packets are encrypted by libp2p upgraded mutiplexed stream connection by default.
• There is a separate distinction between the network topology and the logical agent network topology.
• The DHT stores which peers own which agents and which agents belong to which groups. Both kinds of knowledge are shared as a resource via the DHT.
  • One peer node can own one or more agents, and those are unique at a given point to that node. There is no replication, at least initially, if a node is down the state and existence of the agent will be lost until it recovers.
  • Group membership discovery (and peer routing) is done via the DHT. The key is the identifier for the agent/group and the resource itself is the table of connections.
  • Once established any calls regarding group behavior will be handled by a second pub/sub kind of application layer protocol.

Groups are an abstract type which denote a set of agents which share a feature in common. Groups are owned by one or more agents. Agents can (request) join, create or drop groups.

• Groups have topic spans, anyone belonging to a group will be subscribed or publishing (depending on the kind of permits) to all the topics which are present in the span. In essence topics are MPMC channels, and some agents may only be listening, only writing or both. One topic can be present in different groups.
• Agents will automatically broadcast to all the topics in a group span; messages can be group delimited.