Starting off easy

Let’s start single-threaded as we did in the last post: one thread that receives from one socket, sums one received bit (so the compiler won’t simply optimize away the whole program) and discard it.

This works surprisingly well: a single thread can receive and discard about 1.8 million packets per second from localhost on my machine. However, we need to distribute receiving onto multiple threads if we want to be able to efficiently distribute parsing and crypto on multiple threads as well. I further suppose that when using actual hardware, it will come in handy do have several threads splitting their time between waiting for the device and calculating stuff on the CPU.

One way to have multiple threads share the load, would be to bind each of them to a different socket address. This might work quite well but it complicates clientside logic, since the clients need to re-connect to different sockets when we change the amount of threads.

Fortunately, the SO_REUSERPORT from the last article also works for receiving messages. Before we go over how to make use of it, let me first show you what to avoid: Contesting on a single socket as I did in the tokio post.

What did I expect?

In retrospect, I seriously wonder, what I was thinking. Tokio distributes work across multiple tasks that might be executed in parallel or at least concurrently on a thread pool. This is great for connection-oriented protocols like TCP, when you have 1000 connenctions, they can easily be mapped to 8 threads that way, only costing resources when there is work available.

But this is not necessarily a good idea for UDP. The UDP protocol was designed for connection-less communication. If we want to implement connection-semantics on top of it, one has to do so in their own code, *after receiving messages. It therefore is utter nonsense to receive from one socket and distribute the receiving itself to a thread pool an exception would be if we cannot offload processing the received message to another thread for whatever reason.

After thinking about the approach in the tokio article, I am still surprised that it did in fact work so well.

Reusing the port

Now, let’s make use of SO_REUSERPORT. First the same note as in the send article: this won’t allow our code to run on windows or linux systems with ancient kernels. So what does SO_REUSERPORT actually mean for receiving Datagrams?

Let me quickly cite the docs here:

For UDP sockets, the use of this option can provide better distribution of incoming datagrams to multiple processes (or threads) as compared to the traditional technique of having multiple processes compete to receive datagrams on the same socket.

First thing to note: several threads are not competing for the incoming data and therefore are not churning on some lock or something. But how is this achieved? According to first result on google stating:

Incoming connections and datagrams are distributed to the server sockets using a hash based on the 4-tuple of the connection

Originally, I planned to dive into the net/ipv4/udp.c sources of the kernel here but I got scared off pretty quickly: I completely forgot that goto still exists. Perhaps one day I will work my way through the netcode. For now let’s just assume that the hashing works as one would expect.

For now, we need to

Using SO_REUSERPORT, have several threads that receive from their own share of connected hosts. This means that if one thread has no client assigned that is sending anything it would poll its socket and block until there is something new. However, polling might be slower than being notified. I therefore chose to use epoll for this cause: our code is woken up, whenever there is something interesting happening.

How to use epoll in Rust? One could use the syscall directly but that sounds quite taunting. Luckily there is mio, a crate that abstract all the nasty low-level things away and lets us directly register interests, for which, if fulfilled, we want to be notified.

Mio

Mio uses their own mio::net::UdpSocket, wich does not support setting the reuseport flag upon creation. Luckily mio can reuse std sockets and wrap them into their own constructs. So we can use a socket2 socket, set the reuseport flag, convert it to an std::net::UdpSocket and convert that one to a mio::net::UdpSocket.

let s2_socket = Socket::new(...);
s2_socket.set_reuse_port(true);
s2_socket.bind(&addr.into());
let std_socket = s2_socket.into_udp_socket();
let mio_socket = UdpSocket::from_std(std_socket);

Using Mio, we have a problem with measuring our performance: Using the std sockets, I simply could let them receive a million packets, return from the call and measure the execution duration and resume receiving. In mio however, things work quite a bit differently: We first register an interst with a Poll struct. That way, mio knows if we want to be notified when the socket is readable, writeable or both. We then poll on the Poll struct (not the socket) and get a list of events poll.poll(&mut events,...). We then have to process the events and hopefully have our token in it, stating that the socket is readable. Now the tricky part begins. Now we have to receive from the socket until it return the following Errot type: io::ErrorKind::WouldBlock. If we stop receiving before that, the socket might not fire the readable event again and we might never be able to read from it again.

One way to work around this is to re-register the interest. However, since I only want that code to measure things and not to win any beauty competitions, the receive loop of each thread measures the receive rates individually. This has another side-benefit: Having the rates of all threads, we can observe how incoming datagrams are distributed across the receiving threads.

The following table shows how well (or rather how bad) the clients (12) are distributed across the receivers (6).

thread      average pps
0           583666.7
1           186166.7
2           583972.2
3           384625.0
4           186250.0
5           384708.3

In a real-world scenario, we would most likely have more than a hundred clients, so the differences will be evened-out. Nonetheless, this observation is important: If we planned to deploy some traffic-aggregators in front of the actual game server to decrease the packet rate on the GS by combining multiple clients in one packet, we would observe the same inequality.

Summing up the averages in R reveals a combined packet-rate of 2.3 Mpps when sending with 2.4 Mpps at the receiver, so there is some packet loss but it is not yet the majority:

x <- read.csv("whatever.csv")
a <- aggregate(x,list(x$thread),mean)
> sum(a1$pps)
[1] 2309389

When sending without limits in the sender, the receiver can reach about 2.9 Mpps. However, the receivers with less associated senders will not live up to their full potential. A ‘saturated’ receiver is receiving about 1 Mpps on my machine.

Real Hardware

Now that we know what the software is capable of if running on its own, it is time to introduce a bit of problems: we need to send the traffic over real networks. Let me start with a bunch of hardware I have sitting at home, collecting dust.

My own hardware

We will repeat the same test as in the tokio article, and I expect to see the whole thing capped again at 500Kpps. In a second step, I will directly connect the two machines without any network hardware in between.

It turns out the assumption was correct, the packet rate was again capped at about 500 Kpps on the sending side. This is important, since we could probably increase the receive rate by sending from multiple devices. However, since I wanted to test without a switch in between, I directly connected the hosts with some CAT7 cable and tried again, observing the same packet rate.

Time to level up the hardware a bit, let’s make use of that fancy clouds.

Someone else’s hardware

To dampen your expectations in advance, this one was disappointing: At Digitalocean I ordered the fattest machines they have for me, 32 CPUs, loads of RAM and let them connect it to some VPC. They seem to be interconnected by a 1Gbps link according to iperf. Therefore I suspected we can observe results comparable to my home network or hopefully faster, these are state-of-the-art servers after all. Well, turns out virtualization is a problem: The benchmaks are capped at exactly 101000 pps, having a deviance of only 500 pps. This strongly indicates that the machines are rate-limited.

Renting a bit of metal

Luckily we one can rent bare-metal servers based on hourly billing. Since I had a bit of demo-credits left, I ordered two servers at packet.com for an hour to play with. Although being slightly overpowered (48 CPUs and 64Gb RAM), they will hopefully do their job well.

Both servers have two Intel x710, 10 Gbit/s NICs which are bonded in their default configuration.

So how much did that machines achieve? I ran 16 receivers, 32 senders with a packet size of 128 bytes. According to the previously used calculation method, adding up the average rates we total to 8 Mpps. However, this calculation method has its problems: it is only a rough estimate assuming all threads run for at their average speed the whole time. Since this is not the case, let’s have a few more details about the benchmark data.

Thread 12 did perform exceptionally well, showcasing the best-case scenario:

> filter(x, th==12)
   run th     pps
1    1 12 1037000
2    2 12 1046000
3    3 12 1046000
4    4 12 1059000
5    5 12 1054000
6    6 12 1050000
7    7 12 1054000
8    8 12 1059000
9    9 12 1059000
10  10 12 1054000
11  11 12 1059000
12  12 12 1054000
13  13 12 1054000

Whereas thread 3 had not enough work to do:

> filter(x, th==3)
  run th    pps
1   0  3 132000
2   1  3 145000

To summarize this catastrophic distribution, I want to show you the best boxplot I ever made:

It made a huge difference between using 16 receiver threads compared to only using 8 threads. On average we processed 576 Kpps per thread but, we might be able to process one million packets per second per thread, at least we did with thread 12. If it turns out we need to increase the total throughput we could try to tune the kernel, hardware or something in that line but I doubt that these rates won’t be enough for now.

Next steps

The next step is to develop some pseudo-reliable protocol on top of UDP and then find out how fast we can run it compensating packet loss. However, I first want to ensure that the other components of the networking stack are capable of keeping up with the speed of the socket, in particular deserialization and crypto.

The plan

As in my last post, I will start by maximizing the amount of packets per second the application can send, then the bandwidth can easily be increased by increasing the packet size.

Let’s start with some pseudocode (nearly executable python), to describe what needs to be done:

buffer=[...]
counter=0
timestamp()
while counter < 100_000_000:
  send()
  counter+=1
print(timediff())

In this first step, we simply don’t care if our packets reach their destination, so we can as well send them to somewhere they will be ignored like 127.0.0.1:42 (assuming nothing is listening on port 42). Note that this way we are not limited by the capabilities of our network card in this first step but expect to see results that we will most likely not achieve on real hardware.

The straight-forward approach: std::socket

Let’s start with the straight-forward approach: just use, what the std lib provides us. In this firt iteration’ this simply is a single thread, executing the rust-equivalent to the above pseudocode. This means we execute such a function, sending a lot of zeroes:

pub fn send_packets_to(&self, amount: u32, size: u32, to: SocketAddr) {
  let buf = vec![0 as u8; size as usize];
  for i in 0..amount {
    self.socket.send_to(&buf, &to).expect("Send failed");
  }
}

Note that we send the same buffer over-and-over again and do not re-allocate memory for it (at least not in our code).

So, how well does this work? Let’s mangle the results with a bit of R and see what we got:

Let me highlight some observations:

• The packet rate does not change considerably if we send messages of 2 bytes or 265 bytes.
• Up until 2^11 bytes we have quite stable performance of about 450K packets per second. Note however that this already equals 450000 * 2^11 * 8 = 7 Gbit/s which is probably more than you are prepared to pay for in a cloud-environment for prolonged times.
• We are sending to localhost here. This means in particular, that we do not involve any network cards or even such mundane things as ethernet MTUs that will limit us if we try to do this in a real scenario later on.

I conclude that it is not worth anything network-performance-wise to optimize package sizes below 256 bytes. In terms of traffic costs, this migtht be another story. We will later see how packet size influences parsing or cryprographic operations.

Now we can have a look at the average bandwidth and see if this is matches what the OS tells us while the benchmark is running.

As you can see, we reach impressive bandwidths, that are completely unrealistic if we write to an actual network device. So what is the next logical step? Of course:try to generate even more unrealisitc numbers: Currently we use only one thread but most of our machines have more than one core. So let the premature optimization begin. (You will later see why this might be not that stupid)

More threads with std::socket

The code in this section does not make that much sense in a server implementation: We will create multiple sockets that send to somewhere. So why write it? With such code we can later test our receive rates, since we will (most likely) accept packages from several clients at the same time.

As before, the packet rate does not decrease horribly until a packet size of 2^12 bytes.

Wait a second, are you telling me, that we satureate about 150 Gbits/s of bandwidth? Have a look yourself:

Going deeper: SO_REUSEPORT

In this section we will utilize multiple cores to send from one socket. If we used the std::socket implementation to bind on the same address from multiple threads it would crash and tell us that the address is already in use. Mhm. Luckily we can get around this: We can set some socket options to allow exactly that. But at what cost? The only thing we have to sacrifice is that we cannot run our code on windows servers anymore, but that is a sacrifice I am willing to make.

According to the docs, we need to set the flag SO_REUSEPORT. Unfortunately the rust std lib does not allow us to do this and we therefore need to use a crate that extends the capabilites of our socket creation: socket2.

With socket2 we can set the SO_REUSEPORT flag with the accoring methods:

let socket = Socket::new(
  Domain::ipv4(), 
  Type::dgram(), 
  Some(Protocol::udp())
  ).unwrap();

socket.set_reuse_port(true);
socket.set_nonblocking(true);

socket.bind(&addr.into());
let socket = socket.into_udp_socket();

Now we can bind multiple times to the same socket, e.g have 12 threads send from “127.0.0.1:9000”. Note however, that if we try to send faster than the system is able to handle, there might be no guarantees whether the kernel will drop packets and if so, from which thread they are.

The last thing to ask here is, can we gain a few more packets per sacket by optimizing the rust code itself? Perhaps even modify internals of a library? Luckily, we don’t need to. The amount of time spent in our code is so small, we cannot even see it in the Flamegraph below.

You can see that the code calls “__libc_sendto” internally and we therefore do not need to optimize any loops or such things in our rust code. However, there are still options we need to keep in mind for later: real-hardware might produce an entirely different picture but we will see when we come to that.

Next steps

Until now, we only sent traffic on out local host, what is quite boring. To get more realistic numbers, we need to send our traffic through real hardware. In a future article I will try to do exactly this, but my laptop has only a one gigabit port. Luckily you can rent hardware on a per-hour basis for affordable prices. As soon as the receiving side is ready, I will test send and receive speeds on real hardware.

Scenario

For my game server, I want to be able to receive many small UDP packets. Lots of them. Since typical updates in my case are only a few bytes (say x and y coordinates, a uid and timestamp totalling to say 32 bytes), a gameserver will reach its processing limit before even getting close to saturate the bandwidth of its uplink.

For this experiment, let’s assume we bind on one IPv4 UDP socket that receives all our game traffic. Further, let the received messages be sent to another task for processing (here: incrementing a counter and ignoring it).

Tokio

Tokio describes itself as an ‘asynchronous run-time’, in other words: it does provide means for running code annotated with those nice async keywords. Since tokio has deep roots in the network programming environment, it supports network primitives as first-class citizens. This allows us to use non-blocking code to read and write from the network.

So let’s use tokio by adding it to the cargo.toml with all features:

[dependencies]
tokio = { version = "0.2.10", features = ["full"] }

The client

Now to the client, its only job is to send lots of updates at a roughly fixed rate. You can find the full code in the github repo

Since we use tokio, we can directly tell it, that our main function will by executed asynchronous:

#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
  //...
}

Then we need a udp socket to send our messages with;

let mut socket = UdpSocket::bind(local_addr).await?;
socket.connect(&remote_addr).await?;

Now a bit of a payload, in this case a single i32 in a struct. Since we don’t care about those updates, we will send the same update over and over again:

let update = MovementUpdate{
    id: 7
};
let encoded = bincode::serialize(&update).unwrap();

Now, in a loop, we send the serialized update a few million times:

socket.send(&encoded).await?;

The server

The first server should perform the following steps:

• Accept a packet
• parse it
• increment the counter
• send it to the worker task for processing

Since the full code is in the repo, I will only highlight the interesting parts here.

We spawn our two async fns from the main function, that itself is not async. Spawning happens on a created runtime:

let (tx,rx) = mpsc::channel(100);

let handler = rt.spawn(rcv_pass_handler(opt.clone(),rx));
let receiver = rt.spawn(rcv_pass(opt.clone(),tx));
let join_handler = join_all(vec![handler,receiver]);

rt.block_on(join_handler);

The rx and tx channel parts are for communication between the tasks.

The rcv_pass function is looping over the following code to receive, parse and forward the datagrams.

let _res = socket.recv_from(&mut buf).await.unwrap();

let packet: MovementUpdate = bincode::deserialize(&buf[..]).unwrap();

tx.send(packet).await.unwrap();

(Sorry for the unwrap but here I don’t care)

The handler function is looping over the other end of the channel and incrementing a counter, printing its calue from time to time:

 while let Some(packet) = rx.recv().await {
    msg_ctr+=1;
    // print from time to time
}

And that’s it, now have a look at the results.

Results

That’s what I found when running the code, ymmw.

Same host

When running client and server on the same host, my machine could achieve about 700Kpps. This is good to know but has no relevance for any practical use cases except perhaps some IPC.

Different hosts

tl;dr: The maximum rate was 500Kpps.

To get closest to the use case, the server was run on a machine with decent power and the clients were run at several other hosts, everything connected via a gigabit ethernet switch.

Running any amount of clients with sufficient send rates capped at 500Kpps. But what was limiting throughüut here?

When running two independen isntances of the server, the total packet rate was, again, about 500Kpps. I therefore conclude that either the OS or the hardware is limiting the throughput.

If someone has a winows machine, it would be interesting to see if there is any difference in performance.

Conclusion

Rust and tokio are fast. Like, really fast. But more important, they are elegant, they allow you to things at the speed of C++ without having to care about all the tiny details. Combine that with the ergonomic features like the async functions, and you have ma favourite programming language.