On Mon, Jul 26, 2021 at 10:35 AM Vincent Li [off-list ref] wrote:

On Sun, Jul 25, 2021 at 8:49 PM Vincent Li [off-list ref] wrote:

quoted

On Sun, Jul 25, 2021 at 7:39 PM Yonghong Song [off-list ref] wrote:

quoted

On 7/25/21 6:14 PM, Vincent Li wrote:

quoted

On Sun, Jul 25, 2021 at 6:01 PM Yonghong Song [off-list ref] wrote:

quoted

On 7/25/21 8:22 AM, Vincent Li wrote:

quoted

On Sat, 24 Jul 2021, Vincent Li wrote:

quoted

On Sat, 24 Jul 2021, Vincent Li wrote:

quoted

On Fri, Jul 23, 2021 at 7:17 PM Vincent Li [off-list ref] wrote:

quoted

Hi BPF experts,

I have a cilium PR https://github.com/cilium/cilium/pull/16916 that
failed to pass verifier in kernel 4.19, the error is like:

level=warning msg="Prog section '2/7' rejected: Argument list too long
(7)!" subsys=datapath-loader
level=warning msg=" - Type:         3" subsys=datapath-loader
level=warning msg=" - Attach Type:  0" subsys=datapath-loader
level=warning msg=" - Instructions: 4578 (482 over limit)"
subsys=datapath-loader
level=warning msg=" - License:      GPL" subsys=datapath-loader
level=warning subsys=datapath-loader
level=warning msg="Verifier analysis:" subsys=datapath-loader
level=warning subsys=datapath-loader
level=warning msg="Error filling program arrays!" subsys=datapath-loader
level=warning msg="Unable to load program" subsys=datapath-loader

then I tried to run the PR locally in my dev machine with custom upstream
kernel version, I narrowed the issue down to between upstream kernel
version 5.7 and 5.8, in 5.7, it failed with:

I further narrow it down to between 5.7 and 5.8-rc1 release, but still
no clue which commits in 5.8-rc1 resolved the issue

quoted

level=warning msg="processed 50 insns (limit 1000000) max_states_per_insn
0 total_states 1 peak_states 1 mark_read 1" subsys=datapath-loader
level=warning subsys=datapath-loader
level=warning msg="Log buffer too small to dump verifier log 16777215
bytes (9 tries)!" subsys=datapath-loader

The error message is "Log buffer too small to dump verifier log 16777215
bytes (9 tries)!".

Commit 6f8a57ccf8511724e6f48d732cb2940889789ab2 made the default log
much shorter. So it fixed the above log buffer too small issue.

Thank you for the confirmation, after I remove 'verbose' log, indeed
the problem went away for kernel 5.x- 5.8, but the
"Prog section '2/7' rejected: Argument list too long.." issue
persisted even after I remove the "verbose" logging
for kernel version 4.19, any clue on that?

No, I don't.

You need to have detailed verifier log. In verifier, there are quite
some places which returns -E2BIG.

I will do another round of bisect,  correct myself, the "The argument
list too long" occurred in 5.1, but not in 5.2

It looks to be this commit fixed the issue
commit c04c0d2b968ac45d6ef020316808ef6c82325a82 (HEAD)
Author: Alexei Starovoitov [off-list ref]
Date:   Mon Apr 1 21:27:45 2019 -0700
    bpf: increase complexity limit and maximum program size

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

level=warning msg="Error filling program arrays!" subsys=datapath-loader
level=warning msg="Unable to load program" subsys=datapath-loader

5.8 works fine.

What difference between 5.7 and 5.8 to cause this verifier problem, I
tried to git log v5.7..v5.8 kernel/bpf/verifier, I could not see commits
that would make the difference with my limited BPF knowledge. Any clue
would be appreciated!

I have git bisected to this commit:

[...]

Finally bring an end to my long time mystery issue of using
eth_store_daddr() result in
"invalid mem access", I need to initialize the mac address variable
with 0, uninitialized variable
also cause "invalid read from stack off" in kernel 4.9, but not
version above 4.9.
https://github.com/cilium/cilium/pull/17370#issuecomment-922396415