I've not tried using it on a Raspberry Pi.

I have not tried Buildroot but I'd expect Linux-libre to be able to function as a drop-in replacement for the one from kernel.org.