eBPF Load balancer

This repo was created to reimplement and extend Liz Rice's eBPF-based load balancer example using Go user-space code. A big applause to Liz for her great work and for sharing it with the community. I've also enjoyed her book "Learning eBPF" and I highly recommend it to anyone interested in eBPF.

For user-space code I decided to use cilium's Go eBPF library. The eBPF program is written in C, and bpf2go is used to compile it and generate Go bindings. The user-space code is responsible for loading the eBPF program into the kernel, setting up the load balancer, and handling events from a ringbuffer. These events are then printed to the console.

Do not use this in production. This is a learning project and it's not optimized for performance or security.

Setup guides:

• Installations steps for Mac with Apple Silicon chip
• Installations steps for Ubuntu 22.04 (AMD64)

Installations steps for Mac with Apple Silicon chip

Initial setup

Install VMware Fusion on your M1 Mac. You can download it from VMware Fusion. Set up an Ubuntu 22.04 virtual machine within VMware Fusion. Ensure you configure the VM with OpenSSH. Once Ubuntu is installed, set up your SSH keys for secure access. Then use a terminal on your Mac to SSH into your new Ubuntu VM and install the following softwares:

• Go
• Docker
• clang and llvm

sudo apt update && sudo apt upgrade
sudo apt install clang llvm

• libbpf headers and linux kernel headers

Note: gcc-multilib is not currently available for ARM architectures on Ubuntu 22.04. Instead, I'm linking /usr/include/$(shell uname -m)-linux-gnu into the include path. See this thread for more info.

sudo apt install libbpf-dev
sudo ln -sf /usr/include/aarch64-linux-gnu/asm /usr/include/asm

• for using Makefile (and column)

sudo apt install build-essential bsdmainutils

• bpftool and ip (optional)

sudo apt install linux-tools-common linux-tools-generic iproute2

Clone the repo and initialize it

git clone https://github.com/kegliz/ebpf-lb.git
cd ebpf-lb
go get github.com/cilium/ebpf/cmd/bpf2go 

Build the eBPF program

make build

Create backend and client containers

for backends:

docker run -d --rm --name backend-A -h backend-A --env TERM=xterm-color nginxdemos/hello:plain-text
docker run -d --rm --name backend-B -h backend-B --env TERM=xterm-color nginxdemos/hello:plain-text

In a different terminal exec into one of the backends and install tcpdump if you want to see incoming traffic there.

docker exec -it backend-A /bin/sh 
apk add tcpdump
tcpdump -i eth0

Open a different terminal for the client and start its container:

docker run --rm -it -h client --name client --env TERM=xterm-color ubuntu:jammy
apt update && apt upgrade 
apt install curl

If these are the first docker containers you are running, there is a high chance that the IP addresses of the containers are as follows:

• backend-A: 172.17.0.2
• backend-B: 172.17.0.3
• client: 172.17.0.4

If not, then use the following commands to obtain the IP addresses and correct the program code accordingly

docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' backend-A
docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' backend-B
docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' client

Make sure the backends are running and available from the client so check it from the client container:

curl 172.17.0.2
curl 172.17.0.3

Start the load balancer

Running it as privileged gives it permissions to load eBPF programs:

docker run --rm -it -v ~/work/ebpf-lb:/lb --privileged -h lb --name lb --env TERM=xterm-color ubuntu:jammy
cd lb
./ebpf-lb

Test the load balancer

From the client container run the following command multiple times to see the load balancer in action:

curl 172.17.0.5

The answer should come from either backend-A or backend-B as the load balancer distributes the requests between them. You can also check the traffic on the backends with tcpdump. On the host, you can use the following command to see the debug output of the eBPF program:

sudo cat /sys/kernel/debug/tracing/trace_pipe

Installations steps for Ubuntu 22.04 (AMD64)

The only difference from the ARM-based installation is that there is no need to create a soft link to /usr/include/$(shell uname -m)-linux-gnu. Instead, simply install the gcc-multilib package:

sudo apt install gcc-multilib