Re: selftests/net: udpgso: LTS kernels supportability ?

On 12/18/18 12:53 PM, shuah wrote:

On 12/18/18 4:37 AM, Rafael David Tinoco wrote:

quoted

On 12/17/18 4:42 PM, shuah wrote:

quoted

Hi Rafael,

On 12/17/18 10:53 AM, Rafael David Tinoco wrote:

quoted

Shuah,

I was recently investigating some errors coming out of our functional
tests and we, Dan and I, came up with a discussion that might not be
new
for you, but, interests us, in defining how to better use kselftests as
a regression mechanism/tool in our LKFT (https://lkft.linaro.org).

David / Willem,

I'm only using udpgso as an example for what I'd like to ask Shuah.
Feel
free to jump in in the discussion if you think its worth.

All,

Regarding: udpgso AND https://bugs.linaro.org/show_bug.cgi?id=3980

udpgso tests are failing in kernels bellow 4.18 because of 2 main
reasons:

1) udp4_ufo_fragment does not seem to demand the GSO SKB to be > than
the MTU for older kernels (4th test case in udpgso.c).

2) setsockopt(...UDP_SEGMENT) support is not present for older kernels.
(commits "udp: generate gso with UDP_SEGMENT" and its fixes seem to be
needed).

This case is easy right? Based on the test output below , I can see that
the failure is due to

./udpgso: setsockopt udp segment: Protocol not available. setsockopt()
is returning an error to clearly indicate that this options isn't
supported. This will be a test change to say test is a skip as opposed
to fail.

You referred to (2). (1) isn't that straightforward.

quoted

We have a solution for this - test should SKIP as opposed to FAIL.

quoted

With that explained, finally the question/discussion:

Shouldn't we enforce a versioning mechanism for tests that are testing
recently added features ? I mean, some of the tests inside udpgso
selftest are good enough for older kernels...

Right - we do have generic way to handle that by detecting if feature is
supported and skip instead of using Kernel version which is going to be
hard to maintain.

You can't distinguish case (1) failures between real failures OR older
kernel behaving differently then testcase expects.

quoted

quoted

But, because we have no control over "kernel features" and "supported
test cases", we, Linaro, have to end up blacklisting all selftests that
have new feature oriented tests, because one or two test cases only.

Can you share the blacklisted tests?

Sure!

The following yaml file:

https://github.com/Linaro/qa-reports-known-issues/blob/master/kselftests-production.yaml

Contains a list of all "known issues" for kselftests running in LKFT.

1 entry example (this particular one):

 - environments: *environments_all
    notes: >
      LKFT: net: udpgso.sh PASS on 4.18 and failed on 4.17 and below
    projects:
    - lkft/linux-stable-rc-4.14-oe
    - lkft/linux-stable-rc-4.9-oe
    - lkft/linux-stable-rc-4.4-oe
    - lkft/linaro-hikey-stable-rc-4.4-oe
    test_names:
    - kselftest/net_udpgso.sh
    - kselftest-vsyscall-mode-none/net_udpgso.sh
    - kselftest-vsyscall-mode-native/net_udpgso.sh
    url: https://bugs.linaro.org/show_bug.cgi?id=3980
    active: true
    intermittent: false

Tells us:

Its a known issue for all environments (arm, arm64, x86, x86_64, qemus,
etc) and kernels 4.14, 4.9 and 4.4. The "url" shows our internal bug
related to the known issue/investigation.

With that, we don't consider it as a regression when this test fails for
4.14, 4.9 and 4.4 in any existing environment (boards, qemu/kvm, hosts,
diff archs).

I hope it helps you,

Cheers o/
-- 
Rafael D. Tinoco
Linaro - Kernel Validation