Solarflare TCPDirect

 Copied from 

https://blog.emumba.com/a-beginners-guide-to-solarflare-tcpdirect-1c590759d18c

 

 

A beginner’s guide to Solarflare TCPDirect

For the past few months, our team has been working on reducing the overall latency of the TCP/IP stack. Solarflare’s OpenOnload based kernel bypass gives a significant performance boost over the kernel. Using that, we were able to achieve 1800ns UDP to TCP round trip with our application in it but this wasn’t enough. After some research, we found out that TCPDirect claims a further performance boost of around 300–350ns, so we decided to test it out. Although there is a manual available for TCPDirect, there isn’t any significant, easy to understand help available to port conventional sockets to TCPdirect “zockets”.

After a lot of exploration and experimentation, I have come out to a cheat sheet to convert a conventional socket-based application to TCPDirect’s “zockets”.

Initial Setup

Initial Setup

To get started with TCPDirect, you have to do the following initializations

1. Add the required libraries.
2. Call zf_init(). It initializes the TCPDirect.
3. Set the optional properties of TCP/IP stack in the “Attr” structure e.g buffer size or interface. This attribute will be used in the next step and later on while initializing the zockets.
4. Allocate the stack. One stack can handle a maximum of 64 zockets. If you want to open more zockets you have to create more stacks.

Unlike conventional UDP socket, which provides the leverage of sending and receiving UDP packets through the same socket, you must have two different zockets for sending and receiving in TCPDirect.

UDP Receive

UDP Receive

To set up a receive zocket, you need to perform the following steps.

1. Set up a UDP receive zocket using stack and attribute structures used in the initialization step.
2. Bind to the address you want to listen, here ai_local is addrinfo type structure.
3. Call zf_reactor_perform() to check if any packet is received.
4. Use zfur_zc_recv() to receive the packet, if the packet is available in the previous step.
5. As it is a zero-copy receive, free the sk-buff after utilizing the data.

UDP Send

UDP Send

To send a UDP packet, do the following steps

1. Set up a UDP send zocket using stack and attribute structures used in the initialization step. Give remote and local addresses in ai_local and ai_remote, which are addrinfo type structures.
2. Send packet using zfut_send_single().

TCP Send Receive

Unlike UDP, the same zocket can be used to send and receive packets.

1. Set up a TCP zocket using stack and attribute structures used in the initialization step.
2. Connect to the remote server using zft_connect, here ai_remote is addrinfo type structure.
3. Send a packet using zft_send_single().
4. Start listening to the port to receive packets.
5. Use zftl_accept() to check if a connection is accepted or not.
6. Frequently call zf-reactor_perform() to check for any received event or packet. This is a very critical function, as it also handles some non-user-visible events like sending the ACK or retransmitting the packet.
7. Use zft_zc_rect() to receive a packet.
8. As it is a zero-copy receive, free the sk-buff after utilizing the data.

Final words

TCPDirect APIs are a bit different from the kernel APIs but they can be easily mapped. A small effort in changing them can result in a significant performance boost of about 300–350ns in your latency-critical application. Hopefully, this guide will be helpful to get you started with TCPDirect.