Completing eBPF JIT support for MIPS32

The MIPS64 architecture has had an eBPF JIT for quite some time, but
the kernel still lacks eBPF JIT support for MIPS32. In 2019 there was
an effort extend the MIPS64 JIT to also support 32-bit MIPS ISAs,
716850ab104d ("MIPS: eBPF: Initial eBPF support for MIPS32
architecture."), but the work was never completed. What would be
needed to bring eBPF JIT support to MIPS32?

According to Paul Burton, 36366e367ee9 ("MIPS: BPF: Restore MIPS32 cBPF JIT")

The eBPF JIT has a number of problems on MIPS32:

- Most notably various code paths still result in emission of MIPS64
  instructions which will cause reserved instruction exceptions & kernel
  panics when run on MIPS32 CPUs.

- The eBPF JIT doesn't account for differences between the O32 ABI used
  by MIPS32 kernels versus the N64 ABI used by MIPS64 kernels. Notably
  arguments beyond the first 4 are passed on the stack in O32, and this
  is entirely unhandled when JITing a BPF_CALL instruction. Stack space
  must be reserved for arguments even if they all fit in registers, and
  the callee is free to assume that stack space has been reserved for
  its use - with the eBPF JIT this is not the case, so calling any
  function can result in clobbering values on the stack & unpredictable
  behaviour. Function arguments in eBPF are always 64-bit values which
  is also entirely unhandled - the JIT still uses a single (32-bit)
  register per argument. As a result all function arguments are always
  passed incorrectly when JITing a BPF_CALL instruction, leading to
  kernel crashes or strange behavior.

- The JIT attempts to bail our on use of ALU64 instructions or 64-bit
  memory access instructions. The code doing this at the start of
  build_one_insn() incorrectly checks whether BPF_OP() equals BPF_DW,
  when it should really be checking BPF_SIZE() & only doing so when
  BPF_CLASS() is one of BPF_{LD,LDX,ST,STX}. This results in false
  positives that cause more bailouts than intended, and that in turns
  hides some of the problems described above.

- The kernel's cBPF->eBPF translation makes heavy use of 64-bit eBPF
  instructions that the MIPS32 eBPF JIT bails out on, leading to most
  cBPF programs not being JITed at all.

I can see two different ways to proceed from here.

1) Continue the work on the 64-bit eBPF JIT and complete the MIPS32
support there.

2) Use the 32-bit cBPF JIT as a fresh start for a new 32-bit eBPF JIT
implementation.

It depends of course on how much effort it would be to fix the
remaining issues with the current eBPF JIT. On the other hand, since
eBPF is a 64-bit architecture I would expect the bytecode to translate
nicely to a 64-bit target, and that a 32-bit JIT might differ a lot.
For other targets there are often separate 64-bit and 32-bit JIT
implementations. I also think the JIT support for MIPS32 should make
it possible to replace the cBPF JIT. That means it cannot fall back to
use the interpreter as I understand the current (32-bit)
implementation does to a large extent. Perhaps it would then be
cleaner to let the 32-bit version be separate implementation.

Any thoughts and suggestions? What do you think would be the best way forward?

Thanks,
Johan