[PATCH 4/9] powerpc/bpf: Handle large branch ranges with BPF_EXIT
From: Naveen N. Rao <hidden>
Date: 2021-10-01 21:15:58
Also in:
bpf
Subsystem:
bpf jit for powerpc (32-bit and 64-bit), bpf [general] (safe dynamic programs and tools), linux for powerpc (32-bit and 64-bit), the rest · Maintainers:
Hari Bathini, Christophe Leroy, Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko, Eduard Zingerman, Kumar Kartikeya Dwivedi, Madhavan Srinivasan, Michael Ellerman, Linus Torvalds
In some scenarios, it is possible that the program epilogue is outside the branch range for a BPF_EXIT instruction. Instead of rejecting such programs, emit an indirect branch. We track the size of the bpf program emitted after the initial run and do a second pass since BPF_EXIT can end up emitting different number of instructions depending on the program size. Suggested-by: Jordan Niethe <redacted> Signed-off-by: Naveen N. Rao <redacted> --- arch/powerpc/net/bpf_jit.h | 3 +++ arch/powerpc/net/bpf_jit_comp.c | 22 +++++++++++++++++++++- arch/powerpc/net/bpf_jit_comp32.c | 2 +- arch/powerpc/net/bpf_jit_comp64.c | 2 +- 4 files changed, 26 insertions(+), 3 deletions(-)
diff --git a/arch/powerpc/net/bpf_jit.h b/arch/powerpc/net/bpf_jit.h
index 89bd744c2bffd4..4023de1698b9f5 100644
--- a/arch/powerpc/net/bpf_jit.h
+++ b/arch/powerpc/net/bpf_jit.h@@ -126,6 +126,7 @@ #define SEEN_FUNC 0x20000000 /* might call external helpers */ #define SEEN_TAILCALL 0x40000000 /* uses tail calls */ +#define SEEN_BIG_PROG 0x80000000 /* large prog, >32MB */ #define SEEN_VREG_MASK 0x1ff80000 /* Volatile registers r3-r12 */ #define SEEN_NVREG_MASK 0x0003ffff /* Non volatile registers r14-r31 */
@@ -179,6 +180,8 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context * void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx); void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx); void bpf_jit_realloc_regs(struct codegen_context *ctx); +int bpf_jit_emit_exit_insn(u32 *image, struct codegen_context *ctx, + int tmp_reg, unsigned long exit_addr); #endif
diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/bpf_jit_comp.c
index fcbf7a917c566e..3204872fbf2738 100644
--- a/arch/powerpc/net/bpf_jit_comp.c
+++ b/arch/powerpc/net/bpf_jit_comp.c@@ -72,6 +72,21 @@ static int bpf_jit_fixup_subprog_calls(struct bpf_prog *fp, u32 *image, return 0; } +int bpf_jit_emit_exit_insn(u32 *image, struct codegen_context *ctx, + int tmp_reg, unsigned long exit_addr) +{ + if (!(ctx->seen & SEEN_BIG_PROG) && is_offset_in_branch_range(exit_addr)) { + PPC_JMP(exit_addr); + } else { + ctx->seen |= SEEN_BIG_PROG; + PPC_FUNC_ADDR(tmp_reg, (unsigned long)image + exit_addr); + EMIT(PPC_RAW_MTCTR(tmp_reg)); + EMIT(PPC_RAW_BCTR()); + } + + return 0; +} + struct powerpc64_jit_data { struct bpf_binary_header *header; u32 *addrs;
@@ -155,12 +170,17 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp) goto out_addrs; } + if (!is_offset_in_branch_range((long)cgctx.idx * 4)) + cgctx.seen |= SEEN_BIG_PROG; + /* * If we have seen a tail call, we need a second pass. * This is because bpf_jit_emit_common_epilogue() is called * from bpf_jit_emit_tail_call() with a not yet stable ctx->seen. + * We also need a second pass if we ended up with too large + * a program so as to fix branches. */ - if (cgctx.seen & SEEN_TAILCALL) { + if (cgctx.seen & (SEEN_TAILCALL | SEEN_BIG_PROG)) { cgctx.idx = 0; if (bpf_jit_build_body(fp, 0, &cgctx, addrs, false)) { fp = org_fp;
diff --git a/arch/powerpc/net/bpf_jit_comp32.c b/arch/powerpc/net/bpf_jit_comp32.c
index a74d52204f8da2..d2a67574a23066 100644
--- a/arch/powerpc/net/bpf_jit_comp32.c
+++ b/arch/powerpc/net/bpf_jit_comp32.c@@ -852,7 +852,7 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context * * we'll just fall through to the epilogue. */ if (i != flen - 1) - PPC_JMP(exit_addr); + bpf_jit_emit_exit_insn(image, ctx, tmp_reg, exit_addr); /* else fall through to the epilogue */ break;
diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
index f06c62089b1457..3351a866ef6207 100644
--- a/arch/powerpc/net/bpf_jit_comp64.c
+++ b/arch/powerpc/net/bpf_jit_comp64.c@@ -761,7 +761,7 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context * * we'll just fall through to the epilogue. */ if (i != flen - 1) - PPC_JMP(exit_addr); + bpf_jit_emit_exit_insn(image, ctx, b2p[TMP_REG_1], exit_addr); /* else fall through to the epilogue */ break;
--
2.33.0