Adding BPF target support to the Rust compiler

When I created this blog back in September my goal was to post at least once a month. It's December now and you're reading my second post, so I'm not exactly off to a great start. 🤔

Things have been busy on the Rust BPF front though! At the end of October I began working on a blog about the current state of things, exactly one year after I started getting involved. While doing that I finally started feeling inspired enough to try and add a BPF target to rustc, something that had been on my todo list for a very long time but never managed to find the time to work on. (Aah... if only someone wanted to sponsor all this work... wink wink!)

A couple of weeks ago I finally sent a pull request to get the new target(s) merged. The changes were pretty straightforward, with the only unexpected thing being that I ended up having to write - a partial linker needed to enable rustc to output BPF bytecode.

I'm going to tell you why I had to write the linker in a moment, but first, let's start with looking at how clang - the de facto standard BPF compiler - compiles C code to BPF.

How BPF projects are compiled with clang

BPF doesn't have things like shared libraries and executables. Programs are compiled as object files, then at run-time they're relocated and loaded in the kernel where they get JIT-ted and executed.

Because of that, and because for a long time function calls were not allowed so everything had to be inlined, BPF programs written in C are typically compiled as a single compilation unit, with library code written in header files and included with #include directives.

clang BPF compilation model

This compilation model is simple and effective: one compilation unit goes in, one object file comes out. Because BPF programs tend to be small, recompiling the whole source code on every change is generally not an issue. Since everything gets compiled together, there's no need for linking separate compilation artifacts. (You see where this is going?)

How Rust projects are compiled

Rust uses a different compilation model. Code is split into crates. Crates can't be lumped together with #include directives, they are always compiled independently as one or more compilation units.

Consider the following example:

alessandro@ubvm:~/src/app$ cargo tree
app v0.1.0 (/home/alessandro/src/app)
└── dep v0.1.0 (/home/alessandro/src/dep)

alessandro@ubvm:~/src/app$ cargo build
   Compiling dep v0.1.0 (/home/alessandro/src/dep)
   Compiling app v0.1.0 (/home/alessandro/src/app)
    Finished dev [unoptimized + debuginfo] target(s) in 0.19s

app is an application crate that depends on a library dep. When building, the following happens:

Rust compilation model

The rust compiler is invoked twice: first to compile the dep crate as a rust library, then to compile the app crate as an executable. When the app crate is compiled, the pre-compiled dep crate is provided as input to the compiler via the --extern option.

This compilation model always produces multiple object files, which then must be linked together to produce the final output. The rust compiler uses an internal linker abstraction, whose implementations spawn to external linkers like ld, lld, link.exe and others.

Therefore, to add a new target with this model we need a linker for the target. Since clang never links anything when targeting BPF though, it turns out that lld - the LLVM linker - can't link BPF at all. So I wrote a new linker.

A new (partial) BPF linker

bpf-linker takes LLVM bitcode as input, optionally applies target-specific optimizations, and outputs a single BPF object file. The inputs can be bitcode files (.bc), object files with embedded bitcode (eg .o files produced compiling with -C embed-bitcode=yes), or archive files (.a or .rlib).

The linker works with anything that can output LLVM bitcode, including clang. There are a couple of reasons for taking bitcode as input instead of object files.

Only a subset of Rust (just like only a subset of C) can be compiled to BPF bytecode. Therefore bpf-linker tries to push code generation as late as possible in the compilation process, after link-time optimizations have been applied and dead code has been eliminated. This avoids hitting potential failures generating bytecode for unsupported Rust code that is actually unused (eg, parts of the core crate that are never used in a BPF context).

Another reason is that the linker might need to apply extra optimizations like --unroll-loops and --ignore-inline-never when targeting older kernel versions that don't support loops and calls.

Not one but two BPF targets!

The rustc fork at includes two new targets, bpfel-unknown-none and bpfeb-unknown-none which generate little endian and big endian BPF respectively. The targets automatically invoke bpf-linker so with that fork, compiling a BPF project with Rust is finally as easy as:

alessandro@ubvm:~/src/app$ cargo build --target=bpfel-unknown-none
   Compiling dep v0.1.0 (/home/alessandro/src/dep)
   Compiling app v0.1.0 (/home/alessandro/src/app)
    Finished dev [unoptimized + debuginfo] target(s) in 1.98s
alessandro@ubvm:~/src/app$ file target/bpfel-unknown-none/debug/app
target/bpfel-unknown-none/debug/app: ELF 64-bit LSB relocatable, eBPF, version 1 (SYSV), not stripped

Getting the targets merged will probably take a while, but worry not! With a little trick, you can use the linker to compile BPF code with stable Rust already today!

I made bpf-linker implement a wasm-ld compatible command line. Since rustc already knows how to invoke wasm-ld when targeting webassembly, it can be made to use bpf-linker with the following options:

alessandro@ubvm:~/src/app$ cargo rustc -- \
        -C linker-flavor=wasm-ld \
        -C linker=bpf-linker \
        -C linker-plugin-lto 
   Compiling dep v0.1.0 (/home/alessandro/src/dep)
   Compiling app v0.1.0 (/home/alessandro/src/app)
    Finished dev [unoptimized + debuginfo] target(s) in 0.68s
alessandro@ubvm:~/src/app$ file target/debug/app
target/debug/app: ELF 64-bit LSB relocatable, eBPF, version 1 (SYSV), not stripped

Let's see what those options do:

And voilà! Go compile some BPF with stable rust now 🎉

What's next

bpf-linker is obviously new and needs more testing. Over the next few weeks I'm going to add more unit tests and try it on more Rust code. I'm also thinking of trying to link the whole Cilium BPF code with it just to test with a large, complex code base.

While working on the rustc target, at some point I went off on a bit of a tangent and ended up making some changes to LLVM and the kernel so I'm going to try and finish those off. They are needed to implement the llvm.trap intrinsic so panic!() can be implemented in a generic way, instead of having to resort to program-specific hacks like jumping to an empty program with bpf_tail_call(). I'll probably do a whole separate post about that.

Finally, after my last post I received some truly great feedback! I was especially pleased to hear from a couple of companies that are using BPF and that are considering using it with Rust.

I was equally pleased to see that there's a group of people developing BPF in C that feel strongly that I'm wasting my time and that Rust brings nothing over C, being BPF statically verified, not needing the borrow checker etc. They gave me inspiration for a post I'm hoping to publish soon, which will cover why I think that Rust has the potential to become as central to the BPF ecosystem as it is central to WebAssembly development today. Until next time!