On 5/24/21 12:24 PM, John Fastabend wrote:

Yonghong Song wrote:

quoted

On 5/24/21 10:23 AM, Andrii Nakryiko wrote:

quoted

On Sat, May 22, 2021 at 9:39 AM Yonghong Song [off-list ref] wrote:

quoted

LLVM upstream commit https://reviews.llvm.org/D102712
made some changes to bpf relocations to make them
llvm linker lld friendly. The scope of
existing relocations R_BPF_64_{64,32} is narrowed
and new relocations R_BPF_64_{ABS32,ABS64,NODYLD32}
are introduced.

Let us add some documentation about llvm bpf
relocations so people can understand how to resolve
them properly in their respective tools.

Cc: John Fastabend <john.fastabend@gmail.com>
Cc: Lorenz Bauer <redacted>
Signed-off-by: Yonghong Song <redacted>
---
   Documentation/bpf/index.rst      |   1 +
   Documentation/bpf/llvm_reloc.rst | 168 +++++++++++++++++++++++++++++++
   2 files changed, 169 insertions(+)
   create mode 100644 Documentation/bpf/llvm_reloc.rst

diff --git a/Documentation/bpf/index.rst b/Documentation/bpf/index.rst
index a702f67dd45f..93e8cf12a6d4 100644
--- a/Documentation/bpf/index.rst
+++ b/Documentation/bpf/index.rst

@@ -84,6 +84,7 @@ Other
      :maxdepth: 1

      ringbuf
+   llvm_reloc

Thanks Yonghong, I found this helpful. I still had to crack
open llvm code though to follow along. A couple small suggestions
below, may or may not be useful. Overall looks good.

quoted

quoted

quoted

   .. Links:
   .. _networking-filter: ../networking/filter.rst

diff --git a/Documentation/bpf/llvm_reloc.rst b/Documentation/bpf/llvm_reloc.rst
new file mode 100644
index 000000000000..bc62bce591b1
--- /dev/null
+++ b/Documentation/bpf/llvm_reloc.rst

@@ -0,0 +1,168 @@
+.. SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
+
+====================
+BPF LLVM Relocations
+====================
+
+This document describes LLVM BPF backend relocation types.
+
+Relocation Record
+=================
+
+LLVM BPF backend records each relocation with the following 16-byte
+ELF structure::
+
+  typedef struct
+  {
+    Elf64_Addr    r_offset;  // Offset from the beginning of section.
+    Elf64_Xword   r_info;    // Relocation type and symbol index.
+  } Elf64_Rel;
+
+For static function/variable references, the symbol often refers to
+the section itself which has a value of 0. To identify actual static
+function/variable, its section offset or some computation result
+based on section offset is written to the original insn/data buffer,
+which is called ``IA`` (implicit addend) below.  For global
+function/variables, the symbol refers to actual global and the implicit
+addend is 0.

Above was too terse for me to follow without looking into some clang
examples. Maybe an example right here would help not sure? Maybe expand
the text a bit? I don't have a really good suggestion.

Just send a new revision with an example. Hope it will make it easy to
understand the above ``IA`` concept.

quoted

quoted

quoted

+
+Different Relocation Types
+==========================
+
+Six relocation types are supported. The following is an overview and
+``S`` represents the value of the symbol in the symbol table::
+
+  Enum  ELF Reloc Type     Description      BitSize  Offset        Calculation
+  0     R_BPF_NONE         None
+  1     R_BPF_64_64        ld_imm64 insn    32       r_offset + 4  S + IA

There are cases where we set all 64-bits of ld_imm64 (e.g., extern
ksym, global variables). Or those will be a different relocation now
(R_BPF_64_ABS64?). If not, I think BitSize 64 is more correct here.

It is still R_BPF_64_64. In llvm, we have restriction that section
offset must be <= UINT32_MAX, and that is why only 32bit is used
to find the actual symbol in symbol table. 32bit permits 4GB section
which should enough in practice for a bpf program.

^^^ maybe add this note in the doc somewhere? I had similar questions.

Added in the new revision.

quoted

libbpf or tools can write to full 64bits of imm values of ld_imm64 insn.

The name is a little bit misleading, but it has become part of ABI
and lives in /usr/include/elf.h and we are not able to change it
any more.

quoted

Looking at LLVM diff I haven't found a test for global variables (at
least I didn't realize it was there), so double-checking here (and it
might be a good idea to have an explicit test for global variables?)

We have llvm/test/CodeGen/BPF/reloc.ll and
llvm/test/CodeGen/BPF/reloc-btf.ll covering R_BPF_64_ABS64. But I think
I can enhance
llvm/test/CodeGen/BPF/reloc-2.ll to cover an explicit global variable case.

^^^ maybe cross-reference llvm tests from kernel docs side? I often look at
these when I get something unexpected/unknown maybe others would find
it helpful, but not know where to look?

The llvm patch has not merged. We need to merge libbpf patch first. 
Otherwise, nightly libbpf CI will fail. But this doc includes a link
to the LLVM patch and you can just go to that llvm patch to find
examples!

quoted

quoted

quoted

+  2     R_BPF_64_ABS64     normal data      64       r_offset      S + IA
+  3     R_BPF_64_ABS32     normal data      32       r_offset      S + IA
+  4     R_BPF_64_NODYLD32  .BTF[.ext] data  32       r_offset      S + IA
+  10    R_BPF_64_32        call insn        32       r_offset + 4  (S + IA) / 8 - 1
+
+For example, ``R_BPF_64_64`` relocation type is used for ``ld_imm64`` instruction.
+The actual to-be-relocated data is stored at ``r_offset + 4`` and the read/write
+data bitsize is 32 (4 bytes). The relocation can be resolved with
+the symbol value plus implicit addend.
+
+In another case, ``R_BPF_64_ABS64`` relocation type is used for normal 64-bit data.
+The actual to-be-relocated data is stored at ``r_offset`` and the read/write data
+bitsize is 64 (8 bytes). The relocation can be resolved with
+the symbol value plus implicit addend.
+
+Both ``R_BPF_64_ABS32`` and ``R_BPF_64_NODYLD32`` types are for 32-bit data.
+But ``R_BPF_64_NODYLD32`` specifically refers to relocations in ``.BTF`` and
+``.BTF.ext`` sections. For cases like bcc where llvm ``ExecutionEngine RuntimeDyld``
+is involved, ``R_BPF_64_NODYLD32`` types of relocations should not be resolved
+to actual function/variable address. Otherwise, ``.BTF`` and ``.BTF.ext``
+become unusable by bcc and kernel.
+
+Type ``R_BPF_64_32`` is used for call instruction. The call target section
+offset is stored at ``r_offset + 4`` (32bit) and calculated as
+``(S + IA) / 8 - 1``.
+
+Examples
+========
+

I liked the examples.

Great. Just added one more in the new revision!