Jakub Kicinski wrote:

On Wed, 28 Aug 2024 15:33:33 -0400 Willem de Bruijn wrote:

quoted

That could work.

Is reporting one KTAP and exitcode per directory vs per packetdrill
invocation good/bad/neither?

To me "neither", hopefully and unhelpfully I won't tell you anything
that's not common sense :) The balance is between:

 - having test cases which don't take too long (assuming debug kernel):
     <15min is good, 
     >1h I will start complaining,
     >1h30m is bad because we can't retry and still make the 3h
            deadline that NIPA has,
     >3h the test can't run in NIPA at all.

vs

 - flip side is having so many cases it's hard to keep track and render
   in the UI. JSON is relatively slow to process. If you have 150 cases,
   that's 300 per branch (debug and non-debug kernels), times 8
   branches a day => 2.4k results / day
   I think that's still fine-ish, but on the larger side for sure. For
   reference net and forwarding have ~100 tests each. FWIW we do have
   the ability to collect and display nested KTAP so the information is
   not lost (but it makes queries slower so we don't fetch it by default).

quoted

Three other issues if this is calling packetdrill directly is
- passing the non-trivial IP specific flags
- running twice, for IPv4 and IPv6
- chdir into the directory of the pkt file

That can be addressed by instead calling a small wrapper shell script.

That would do the test_func_builder part of packetdrill_ksft.py.
But without the need to handle netns, popen/cmd, etc, and thus the
ksft dependencies.

Right!

Implemented this, but hit a snag

Kselftest install does not preserve directories.

So all .pkt files are copied into net/packetdrill root. This is messy.
More fundamentally it breaks the includes in the files (e..g, `source
../common/defaults.sh`).

The output of having each test separate is also quite unreadable.

An alternative is to add each subdirectory as a separate TARGET.
But that seems similarly impractical. And does nothing to group
KTAP output.

In practice, most directories have a handful of .pkt scripts, and
each script is fast, so serialization is not a huge issue.

I found tools/testing/selftests/net/kselftest/ktap_helpers.sh,
which has helpful KTAP boilerplate for shell tests, including
KSFT exit codes. Am using that instead of python, to reduce the
external dependencies.