Adding BPF target support to the Rust compiler
When I created this blog back in September my goal was to post at least once a month. It's December now and you're reading my second post, so I'm not exactly off to a great start. 🤔
Things have been busy on the Rust BPF front though! At the end of October I began working on a blog about the current state of things, exactly one year after I started getting involved. While doing that I finally started feeling inspired enough to try and add a BPF target to rustc, something that had been on my todo list for a very long time but never managed to find the time to work on. (Aah... if only someone wanted to sponsor all this work... wink wink!)
A couple of weeks ago I finally sent a pull request to get the new target(s) merged. The changes were pretty straightforward, with the only unexpected thing being that I ended up having to write https://github.com/alessandrod/bpf-linker - a partial linker needed to enable rustc to output BPF bytecode.
I'm going to tell you why I had to write the linker in a moment, but first, let's start with looking at how clang - the de facto standard BPF compiler - compiles C code to BPF.
How BPF projects are compiled with clang
BPF doesn't have things like shared libraries and executables. Programs are compiled as object files, then at run-time they're relocated and loaded in the kernel where they get JIT-ted and executed.
Because of that, and because for a long time function calls were not allowed so
everything had to be inlined, BPF programs written in C are typically compiled
as a single compilation unit, with library code written in header
files and included with
#include
directives.
This compilation model is simple and effective: one compilation unit goes in, one object file comes out. Because BPF programs tend to be small, recompiling the whole source code on every change is generally not an issue. Since everything gets compiled together, there's no need for linking separate compilation artifacts. (You see where this is going?)
How Rust projects are compiled
Rust uses a different compilation model. Code is split into crates. Crates
can't be lumped together with #include
directives, they are always compiled
independently as one or more compilation units.
Consider the following example:
alessandro@ubvm:~/src/app$ cargo tree
app v0.1.0 (/home/alessandro/src/app)
└── dep v0.1.0 (/home/alessandro/src/dep)
alessandro@ubvm:~/src/app$ cargo build
Compiling dep v0.1.0 (/home/alessandro/src/dep)
Compiling app v0.1.0 (/home/alessandro/src/app)
Finished dev [unoptimized + debuginfo] target(s) in 0.19s
app
is an application crate that depends on a library dep
. When building,
the following happens:
The rust compiler is invoked twice: first to compile the dep
crate as a rust
library, then to compile the app
crate as an executable. When the app
crate
is compiled, the pre-compiled dep
crate is provided as input to the
compiler via the --extern
option.
This compilation model always produces multiple object files, which then
must be linked together to produce the final output. The rust compiler uses an
internal linker abstraction, whose implementations spawn to external linkers
like ld
, lld
, link.exe
and
others.
Therefore, to add a new target with this model we need a linker for the target.
Since clang never links anything when targeting BPF though, it turns out that lld
-
the LLVM linker - can't link BPF at all. So I wrote a new linker.
A new (partial) BPF linker
bpf-linker takes LLVM bitcode as
input, optionally applies target-specific optimizations, and outputs a single
BPF object file. The inputs can be bitcode files (.bc
), object files
with embedded bitcode (eg .o
files produced compiling with
-C embed-bitcode=yes),
or archive files (.a
or .rlib
).
The linker works with anything that can output LLVM bitcode, including clang. There are a couple of reasons for taking bitcode as input instead of object files.
Only a subset of Rust (just like only a subset of C) can be compiled to BPF
bytecode. Therefore bpf-linker tries to push code generation as late as
possible in the compilation process, after link-time optimizations have been
applied and dead code has been eliminated. This avoids hitting potential
failures generating bytecode for unsupported Rust code that is actually
unused (eg, parts of the core
crate that are never used in a BPF context).
Another reason is that the linker might need to apply extra optimizations like --unroll-loops and --ignore-inline-never when targeting older kernel versions that don't support loops and calls.
Not one but two BPF targets!
The rustc fork at https://github.com/alessandrod/rust/tree/bpf includes two new
targets, bpfel-unknown-none
and bpfeb-unknown-none
which generate little
endian and big endian BPF respectively. The targets automatically invoke
bpf-linker so with that fork, compiling a BPF project with Rust is finally as
easy as:
alessandro@ubvm:~/src/app$ cargo build --target=bpfel-unknown-none
Compiling dep v0.1.0 (/home/alessandro/src/dep)
Compiling app v0.1.0 (/home/alessandro/src/app)
Finished dev [unoptimized + debuginfo] target(s) in 1.98s
alessandro@ubvm:~/src/app$ file target/bpfel-unknown-none/debug/app
target/bpfel-unknown-none/debug/app: ELF 64-bit LSB relocatable, eBPF, version 1 (SYSV), not stripped
Getting the targets merged will probably take a while, but worry not! With a little trick, you can use the linker to compile BPF code with stable Rust already today!
I made bpf-linker implement a wasm-ld
compatible command line. Since rustc
already knows how to invoke wasm-ld
when targeting webassembly, it can be
made to use bpf-linker
with the following options:
alessandro@ubvm:~/src/app$ cargo rustc -- \
-C linker-flavor=wasm-ld \
-C linker=bpf-linker \
-C linker-plugin-lto
Compiling dep v0.1.0 (/home/alessandro/src/dep)
Compiling app v0.1.0 (/home/alessandro/src/app)
Finished dev [unoptimized + debuginfo] target(s) in 0.68s
alessandro@ubvm:~/src/app$ file target/debug/app
target/debug/app: ELF 64-bit LSB relocatable, eBPF, version 1 (SYSV), not stripped
Let's see what those options do:
- -C linker-flavor=wasm-ld
tells the compiler that the linker supports the same command line options as
wasm-ld
- -C linker=bpf-linker configures bpf-linker as the linker to spawn
- -C linker-plugin-lto tells rustc to pass LLVM bitcode to the linker
And voilà ! Go compile some BPF with stable rust now 🎉
What's next
bpf-linker is obviously new and needs more testing. Over the next few weeks I'm going to add more unit tests and try it on more Rust code. I'm also thinking of trying to link the whole Cilium BPF code with it just to test with a large, complex code base.
While working on the rustc target, at some point I went off on a bit of a
tangent and ended up making some changes to LLVM and the kernel so I'm going to
try and finish those off. They are needed to implement the
llvm.trap intrinsic
so panic!()
can be implemented in a generic way, instead of having to resort
to program-specific hacks like jumping to an empty program with
bpf_tail_call()
. I'll probably do a whole separate post about that.
Finally, after my last post I received some truly great feedback! I was especially pleased to hear from a couple of companies that are using BPF and that are considering using it with Rust.
I was equally pleased to see that there's a group of people developing BPF in C that feel strongly that I'm wasting my time and that Rust brings nothing over C, being BPF statically verified, not needing the borrow checker etc. They gave me inspiration for a post I'm hoping to publish soon, which will cover why I think that Rust has the potential to become as central to the BPF ecosystem as it is central to WebAssembly development today. Until next time!