Re: distro support for CONFIG_KUNIT: [PATCH 0/3] bitmap: convert self-test to KUnit

On Mon, Feb 10, 2025 at 3:20 PM Yury Norov [off-list ref] wrote:

On Mon, Feb 10, 2025 at 11:35:48AM -0800, John Hubbard wrote:

quoted

On 2/9/25 11:54 PM, Geert Uytterhoeven wrote:

quoted

On Sat, 8 Feb 2025 at 18:53, Yury Norov [off-list ref] wrote:

quoted

On Fri, Feb 07, 2025 at 03:14:01PM -0500, Tamir Duberstein wrote:

quoted

On 7/27/24 12:35 AM, Shuah Khan wrote:

...

quoted

quoted

quoted

The crux of the argument seems to be that the config help text is taken
to describe the author's intent with the fragment "at boot". I think

IMO, "at boot" is a misnomer, as most tests can be either builtin
or modular.

Right.

quoted

quoted

KUNIT is disabled in defconfig, at least on x86_64. It is also disabled
on my Ubuntu 24.04 machine. If I take your patches, I'll be unable to

OK so I just bought a shiny new test machine, and installed one of the
big name distros on it, hoping they've moved ahead and bought into the kunit
story...

$ grep KUNIT /boot/config-6.8.0-52-generic
# CONFIG_KUNIT is not set

...gagghh! No such luck. One more data point, in support of Yuri's complaint. :)

quoted

I think distros should start setting CONFIG_KUNIT=m.

Yes they should! kunit really does have important advantages for many use
cases, including bitmaps here, and "CONFIG_KUNIT is not set" is the main
obstacle.

Hi John, Geert, Tamir,

Can you please explain in details which advantages KUNIT brings to
the test_bitmap.c, find_bit_benchmark.c and other low-level tests?

I can try, but I'm not the expert, and David Gow can probably elaborate further.

As I understand it the main benefit of KUnit is standardization and
speed (and standardization _is_ speed). KUnit makes it very easy for
me, a person who has not previously contributed to any of the bitmap
code, to run those tests, and it requires zero configuration, it all
just works. It's basically just `tools/testing/kunit/kunit.py run
bitmap`, and I get the test results in a human-readable format. The
same benefit applies on the author side: test facilities are
standardized, so once you get to know the tools, all the tests start
to look the same: you can jump in and contribute without having to
first learn the so-called local "testing framework".

The important part is that this all applies to ~all other tests
written in KUnit. I can even run them *all* trivially:
`tools/testing/kunit/kunit.py run`.

Anecdotally I've also noticed there are bots running those KUnit tests
e.g. see https://lore.kernel.org/all/20250207-blackholedev-kunit-convert-v1-1-8ef0dc1ff881@gmail.com/ (local)
where a test I converted was immediately flagged by a robot as having
dubious type coercion.

None of these are must-haves, they are just (to me) a nice way to make
the kernel more approachable for new contributors.

As for pure benchmarks like
find_bit_benchmark.c and
test_bitmap.c::`test_bitmap_{read,write}_perf` specifically: I believe
the benefits are super limited or even negative: AFAIK KUnit is
designed to generally suppress output (in the userspace reporter, not
in the kernel) unless a test fails, so I wouldn't hurry to use it for
these.

I'm not strongly against moving under KUNIT's hat, but I do:
 - respect commitment of my contributors, so I don't want to wipe git
   history for no serious reason;
 - respect time of my testers, so no extra dependencies;
 - respect time of reviewers.

These are valid concerns. Certainly the testing case is the most
compelling and folks are clearly interested in lowering those
barriers. I don't have any influence in this area, but I am grateful
to John for starting the conversation.

As I mentioned in the previous thread: I think we could keep
`test_bitmap_{read,write}_perf` in test_bitmap.c and get the best of
both worlds. WDYT?

Tamir,

If it comes to v2, can you please begin your series with an exhaustive
and clear answer to the following questions:
 - What do the tests miss now?
 - What do _you_ need from the tests? Describe your test scenario.
 - How exactly KUNIT helps you testing bitmaps and friends better?
 - Is there a way to meet your needs with a less invasive approach,
   particularly without run-time dependencies?

Hopefully I've answered these above. I can include some of it in a v2,
but perhaps the general pitch for KUnit is better placed in
documentation or slides from a conference?

Cheers.

Tamir