nnet: a fast, scalable, easy-to-use p2p network stack

GoDoc GitHub license Go Report Card Build Status PRs Welcome

Introduction

nnet is a fast, scalable, and developer-friendly network stack/layer for decentralized/distributed systems. It handles message delivery, topology maintenance, etc in a highly efficient and scalable way, while providing enough control and flexibility through developer-friendly messaging interface and powerful middleware architecture.

Features

Coming soon:

Usage

Requirements:

Install

go get -u -d github.com/nknorg/nnet

It's recommended to use Go Modules to ensure reproducibility of dependencies.

Basic

Assuming you have imported github.com/nknorg/nnet, The bare minimal way to create a nnet node is simply

nn, err := nnet.NewNNet(nil, nil)

This will create a nnet node with random ID and default configuration (listen to a random port, etc). Starting the node is as simple as

err = nn.Start(true) // or false if joining rather than creating

To join an existing network, simply call Join after node has been started

err = nn.Join("<protocol>://<ip>:<port>")

Put them together, a local nnet cluster with 10 nodes can be created by just a few lines of code.

nnets := make([]*nnet.NNet, 10)
for i := 0; i < len(nnets); i++ {
  nnets[i], _ = nnet.NewNNet(nil, nil) // error is omitted here for simplicity
  nnets[i].Start(i == 0) // error is omitted here for simplicity
  if i > 0 {
    nnets[i].Join(nnets[0].GetLocalNode().Addr) // error is omitted here for simplicity
  }
}

A complete basic example can be found at examples/basic/main.go. You can run it by

go run examples/basic/main.go

Middleware

nnet uses middleware as a powerful and flexible architecture to interact with node/network lifecycle and events such as topology change, routing, message sending and delivery, etc. A middleware is just a function that, once hooked, will be called when certain event occurs. Applying a middleware is as simple as calling ApplyMiddleware function (or MustApplyMiddleware which will panic if an error occurs). Let's apply a node.RemoteNodeReady middleware that will be called when a remote node is connected with the local node and exchanged node info

err = nn.ApplyMiddleware(node.RemoteNodeReady{func(remoteNode *node.RemoteNode) bool {
  fmt.Printf("Remote node ready: %v", remoteNode)
  return true
}, 0})

The number 0 after the function is the middleware priority, which is an int32 type number. Different middleware types take different arguments and have different return types, but they all share one thing in common: one of their returned value is a boolean indicating whether we should call the next middleware (of the same type). Multiple middleware of the same type will be called in the order from highest priority to lowest priority. Middleware with the same priority will be called in the order of being added. They form a pipeline such that each one can respond to the event with some data, modify data that will be passed to the rest middleware, or decides to stop the data flow. For example, we can randomly reject remote nodes

nn.MustApplyMiddleware(node.RemoteNodeConnected{func(remoteNode *node.RemoteNode) bool {
  if rand.Float64() < 0.23333 {
    remoteNode.Stop(errors.New("YOU ARE UNLUCKY"))
    return false
  }
  return true
}, 0})

nn.MustApplyMiddleware(node.RemoteNodeConnected{func(remoteNode *node.RemoteNode) bool {
  log.Infof("Only lucky remote node can get here :)")
  return true
}, 0})

nnet internally use middleware with 0 priority to hook up events, e.g. to add neighbor to overlay network when a remote node is ready. You can make your middleware to be called earlier or later by choosing higher/lower priority.

Middleware itself is stateless, but very likely you may need a stateful middleware for more complex logic. Stateful middleware can be created in a variety ways without introducing more complex API. One of the simplest ways is just calling a method in the middleware. As a conceptual example

coolObj := NewCoolObj()
nn.MustApplyMiddleware(node.BytesReceived{func(msg, msgID, srcID []byte, remoteNode *node.RemoteNode) ([]byte, bool) {
  coolObj.DoSomeCoolStuff(msg)
  return msg, true
}, 0})

Stateful middleware if very powerful and can give you almost full control of the system, and it can define its own middleware for other components to use. Actually the whole overlay network can be implemented as a stateful middleware that hooked into a few places like node.LocalNodeWillStart. We didn't choose to do it because overlay network is a top-level type in nnet, but there is nothing that prevent you to do it.

There are lots of middleware types that can be (and should be) used to listen to and control topology change, message routing and handling, etc. Some of them provide convenient shortcuts while some provide detailed low level control. Currently middleware type declaration are distributed in a few places:

Middleware architecture is very flexible and new type of middleware can be added easily without breaking existing code. Feel free to open an issue if you feel the need for new middleware type.

A complete middleware example can be found at examples/middleware/main.go. You can run it by

go run examples/middleware/main.go

Also, examples like examples/message/main.go and examples/efficient-broadcast/main.go use middleware to handle and count messages.

Sending and Receiving Messages

Sending message is top-level function in nnet. You can send arbitrary bytes to any other node/nodes by calling one of .SendXXX methods. Currently there are 3 types of user-defined message classified by routing types:

The broadcast message has a few subtypes:

nnet uses router architecture such that implementing a new routing algorithm is as simple as implementing a Router interface defined in routing/routing.go that computes the next hop using GetNodeToRoute method and register the routing type by calling localNode.RegisterRoutingType if needed.

For each routing types, there are 2 sending message APIs: async where send call is immediately returned if send success, and sync that will be blocked and wait for reply or timeout before return. Under the hood, sync mode creates a reply channel and waits for the message that replies to the original message ID so you can safely use it while sending/receiving other messages at the same time.

Sending an async message is straightforward. Let's send a relay message as an example:

success, err := nn.SendBytesRelayAsync([]byte("Hello world!"), destID)

You can choose to send the message in sync way such that the function call will only return after receiving the reply or timeout:

reply, senderID, err := nn.SendBytesRelaySync([]byte("Hello world!"), destID)

To handle received message and send back reply message, we can use the node.BytesReceived middleware together with SendBytesRelayReply method.

nn.MustApplyMiddleware(node.BytesReceived{func(msg, msgID, srcID []byte, remoteNode *node.RemoteNode) ([]byte, bool) {
  nn.SendBytesRelayReply(msgID, []byte("Well received!"), srcID) // error is omitted here for simplicity
  return msg, true
}, 0})

A complete message example can be found at examples/message/main.go. You can run it by

go run examples/message/main.go

There is another example examples/efficient-broadcast/main.go that counts and compares the received message count for push and tree broadcast message. You will see how tree message can reduce the bandwidth usage by an order of magnitude. You can run it by

go run examples/efficient-broadcast/main.go

Transport protocol

Transport layer is a separate layer that is part of a node in nnet. Each node can independently choose what transport protocol it listens to, and is able to talk to nodes that listen to different transport protocol as long as the corresponding transport layer is implemented. Each node address starts with the transport protocol that the node listens to, e.g. tcp://127.0.0.1:23333, such that other nodes know what protocol to use when talking to it.

Currently nnet have 2 transport protocol implemented: TCP and KCP (a reliable low-latency protocol based on UDP). In theory, any other reliable protocol can be easily integrated by implementing Dial and Listen interface. Feel free to open an issue if you feel the need for new transport protocol.

Changing transport protocol is as simple as changing the Transport value in config when creating nnet. A complete example can be found at examples/mixed-protocol/main.go. You can run it by

go run examples/mixed-protocol/main.go

NAT Traversal

If you are developing an application that is open to public, it is very likely that you need some sort of NAT traversal since lots of devices are behind NAT at the moment. NAT traversal can be set up in middleware such as overlay.NetworkWillStart and node.LocalNodeWillStart, or anytime before the network starts by calling the SetInternalPort method of the LocalNode. A complete NAT example that works for UPnP or NAT-PMP enabled routers can be found at examples/nat/main.go. You can run it by

go run examples/nat/main.go

Logger

Typically when you use nnet as the network layer of your application, you probably want it to share the logger with the rest of your application. You can do it by calling nnet.SetLogger(logger) method as long as logger implements the Logger interface defined in log/log.go. If you don't set it, nnet will use go-logging by default.

Benchmark

Throughput is a very important metric of the network stack. There are multiple potential bottlenecks when we consider throughput, e.g. network i/o, message serialization/deserialization, architecture and implementation efficiency. To measure the throughput, we wrote a simple local benchmark, which can be found at examples/message-benchmark/main.go. You can run it with default arguments (2 nodes, 1 KB message size) by

go run examples/message-benchmark/main.go

On a MacBook Pro 2018 this will give you around 75k message/s per node, far more than enough for most p2p applications.

When message size increase, the bottleneck will becomes bandwidth rather than message count. We can run the benchmark with 1 MB message by

go run examples/message-benchmark/main.go -m 1048576

This will give you around 900 MB/s per node on the same computer, again far more than enough in typical cases.

The same benchmark can be used to see how spanning tree broadcasting can greatly improve throughput. We first use the standard push method to broadcast in a 16-node network:

go run examples/message-benchmark/main.go -n 16 -b push

Each node can only receive around 300 unique message/s. The number is so low because: 1. we are running all 16 nodes on the same laptop with just 6 cores and more importantly 2. Gossip protocol has a high message redundancy, and in this particular example each node receives the same message 15 times! If we sum them up, the total messages being processed are 3001515, which is pretty close to the 75k we got in 2-node benchmark. Of course if the overlay topology is fully connected, we don't need Gossip protocol at all, but similar inefficiency will happen once the network size is large enough so that fully connected topology is unrealistic.

Now let's change to spanning tree broadcasting:

go run examples/message-benchmark/main.go -n 16 -b tree

Now each node can receive 3k unique messages/s, an order of magnitude boost! Actually if the benchmark is running on multiple computers with enough network resources, each node can again receive around 75k messages/s.

Who is using nnet

Welcome to open a pull request if you want your project using nnet to be listed here. The project needs to be maintained in order to be listed.

Contributing

Can I submit a bug, suggestion or feature request?

Yes. Please open an issue for that.

Can I contribute patches to NKN project?

Yes, we appreciate your help! To make contributions, please fork the repo, push your changes to the forked repo with signed-off commits, and open a pull request here.

Please sign off your commit by adding -s when committing:

git commit -s

This means git will automatically add a line "Signed-off-by: Name " at the end of each commit, indicating that you wrote the code and have the right to pass it on as an open source patch.

Community