Re: kernel-selftests: make run_tests -C bpf cost 5 hours

On Thu, Dec 2, 2021 at 10:26 AM Daniel Borkmann [off-list ref] wrote:

On 12/1/21 7:54 AM, Zhou, Jie2X wrote:

quoted

ping

________________________________________
From: Zhou, Jie2X
Sent: Monday, November 29, 2021 3:36 PM
To: ast@kernel.org; daniel@iogearbox.net; andrii@kernel.org; kafai@fb.com; songliubraving@fb.com; yhs@fb.com; john.fastabend@gmail.com; kpsingh@kernel.org
Cc: netdev@vger.kernel.org; bpf@vger.kernel.org; linux-kernel@vger.kernel.org; Li, ZhijianX; Ma, XinjianX
Subject: kernel-selftests: make run_tests -C bpf cost 5 hours

hi,

    I have tested v5.16-rc1 kernel bpf function by make run_tests -C tools/testing/selftests/bpf.
    And found it cost above 5 hours.

    Check dmesg and found that lib/test_bpf.ko cost so much time.
    In tools/testing/selftests/bpf/test_kmod.sh insmod test_bpf.ko four times.
    It took 40 seconds for the first three times.

    When do 4th test among 1009 test cases from #812 ALU64_AND_K to  #936 JMP_JSET_K every test case cost above 1 min.
    Is it currently to cost so much time?

kern :info : [ 1127.985791] test_bpf: #811 ALU64_MOV_K: all immediate value magnitudes
kern :info : [ 1237.158485] test_bpf: #812 ALU64_AND_K: all immediate value magnitudes jited:1 127955 PASS
kern :info : [ 1341.638557] test_bpf: #813 ALU64_OR_K: all immediate value magnitudes jited:1 155039 PASS
kern :info : [ 1447.725483] test_bpf: #814 ALU64_XOR_K: all immediate value magnitudes jited:1 129621 PASS
kern :info : [ 1551.808683] test_bpf: #815 ALU64_ADD_K: all immediate value magnitudes jited:1 120428 PASS
kern :info : [ 1655.550594] test_bpf: #816 ALU64_SUB_K: all immediate value magnitudes jited:1 175712 PASS
......
kern :info : [16725.824950] test_bpf: #931 JMP32_JLE_X: all register value magnitudes jited:1 216508 PASS
kern :info : [16911.555675] test_bpf: #932 JMP32_JSGT_X: all register value magnitudes jited:1 178367 PASS
kern :info : [17101.466163] test_bpf: #933 JMP32_JSGE_X: all register value magnitudes jited:1 191436 PASS
kern :info : [17288.359154] test_bpf: #934 JMP32_JSLT_X: all register value magnitudes jited:1 165714 PASS
kern :info : [17480.615048] test_bpf: #935 JMP32_JSLE_X: all register value magnitudes jited:1 172846 PASS
kern :info : [17667.472140] test_bpf: #936 JMP_JSET_K: imm = 0 -> never taken jited:1 14 PASS

    test_bpf.ko dmesg output is attached.

On my side, I'm seeing:

# time ./test_kmod.sh
[ JIT enabled:0 hardened:0 ]
[  107.182567] test_bpf: Summary: 1009 PASSED, 0 FAILED, [0/997 JIT'ed]
[  107.200319] test_bpf: test_tail_calls: Summary: 8 PASSED, 0 FAILED, [0/8 JIT'ed]
[  107.200379] test_bpf: test_skb_segment: Summary: 2 PASSED, 0 FAILED
[ JIT enabled:1 hardened:0 ]
[  108.130568] test_bpf: Summary: 1009 PASSED, 0 FAILED, [997/997 JIT'ed]
[  108.143447] test_bpf: test_tail_calls: Summary: 8 PASSED, 0 FAILED, [8/8 JIT'ed]
[  108.143510] test_bpf: test_skb_segment: Summary: 2 PASSED, 0 FAILED
[ JIT enabled:1 hardened:1 ]
[  109.116727] test_bpf: Summary: 1009 PASSED, 0 FAILED, [997/997 JIT'ed]
[  109.129915] test_bpf: test_tail_calls: Summary: 8 PASSED, 0 FAILED, [8/8 JIT'ed]
[  109.129979] test_bpf: test_skb_segment: Summary: 2 PASSED, 0 FAILED
[ JIT enabled:1 hardened:2 ]
[ 6617.952848] test_bpf: Summary: 1009 PASSED, 0 FAILED, [948/997 JIT'ed]
[ 6617.965936] test_bpf: test_tail_calls: Summary: 8 PASSED, 0 FAILED, [8/8 JIT'ed]
[ 6617.966004] test_bpf: test_skb_segment: Summary: 2 PASSED, 0 FAILED

real    108m32.833s
user    0m0.031s
sys     108m17.939s

The hardened:2 run takes significantly longer due to excessive patching for the
jit constant blinding code.

I can confirm this too.

The slow tests are designed to check the result of all ALU and JMP
operations for as many different operand values as possible. It is not
feasible to test every combination of the two operand values, so the
space needs to be narrowed down. To exercise JIT code paths that
depend on the operand magnitude, values are chosen with different high
bit set (power-of-two magnitude) and some added small perturbations.
There is a quadratic pattern where both operands vary independently,
and a linear pattern were the MSBs are equal.

Maybe the test cases can be reduced for the latter,
otoh, it's good to know that they all pass as well.

The patterns described above can be tuned by changing the constants
PATTERN_BLOCK{1,2} in lib/test_bpf.c. According to my tests it seems
that the quadratic pattern takes most of the time, but the linear part
is not insignificant either. If I disable the quadratic pattern
completely and reduce the perturbation size for the linear pattern, I
get down to a more reasonable few seconds for each test case instead
of minutes.

The default test patterns may be a bit excessive, but they have also
found real bugs in some JITs. I would like to keep the settings as-is
for the non-hardened runs. I also think it is valuable to be able to
run the full tests during development. One solution could be to add a
module parameter for using a reduced pattern instead. Then the
test_kmod.sh script could set that parameter when running the 4:th
test, while still making it possible to run the full patterns with
hardening manually.

Thanks,
Johan