On Sat, May 7, 2022 at 11:03 PM Miguel Ojeda
[off-list ref] wrote:

Hi David,

On Sat, May 7, 2022 at 11:29 AM David Gow [off-list ref] wrote:

quoted

It's great to see some KUnit support here!

Thanks!

quoted

It's also possible to run these tests using the KUnit wrapper tool with:
$ ./tools/testing/kunit/kunit.py run --kconfig_add CONFIG_RUST=y
--make_options LLVM=1 --arch x86_64 'rust_kernel_doctests'

That also nicely formats the results.

Indeed!

    [16:55:52] ============ rust_kernel_doctests (70 subtests) ============
    [16:55:52] [PASSED] rust_kernel_doctest_build_assert_rs_12_0
    [16:55:52] [PASSED] rust_kernel_doctest_build_assert_rs_55_0
    ...
    [16:55:52] [PASSED] rust_kernel_doctest_types_rs_445_0
    [16:55:52] [PASSED] rust_kernel_doctest_types_rs_509_0
    [16:55:52] ============== [PASSED] rust_kernel_doctests ===============
    [16:55:52] ============================================================
    [16:55:52] Testing complete. Passed: 70, Failed: 0, Crashed: 0,
    Skipped: 0, Errors: 0

I've just sent out a pull request to get this working under UML as
well, which would simplify running these further:
https://github.com/Rust-for-Linux/linux/pull/766

quoted

That all being said, I can't say I'm thrilled with the test names
here: none of them are particularly descriptive, and they'll probably
not be static (which would make it difficult to track results /
regressions / etc between kernel versions). Neither of those are

Yeah, the names are not great and would change from time to time
across kernel versions.

We could ask example writers to give each example a name, but that
would make them fairly less convenient. For instance, sometimes they
may be very small snippets interleaved with docs' prose (where giving
a name may feel a bit of a burden, and people may end writing
`foo_example1`, `foo_example2` etc. for each small "step" of an
explanation). In other cases they may be very long, testing a wide API
surface (e.g. when describing a module or type), where it is also hard
to give non-generic names like `rbtree_doctest`. In those kind of
cases, I think we would end up with not much better names than
automatically generated ones.

The other aspect is that, given they are part of the documentation,
the prose or how things are explained/split may change, thus the
doctests as well. For instance, one may need to split a very long
`rbtree_doctest` in pieces, and then the name would need to change
anyway.

So I think we should avoid asking documentation writers to add a
manual name, even if that means a bit ugly test names. Also this way
they are consistently named. What do you think?

Yeah, these are all fair points: particularly for small doctests.

Maybe having an optional name, which more significant tests could use
to override the file:line names? That could be useful for a few of the
larger, more often referenced tests.

One idea could be giving them a name based on the hash of the content
and avoiding the line number, so that there is a higher chance for the
name to stay the same even when the file gets modified for other
reasons.

Ugh: it's a bit ugly either way. I suspect that file:line is still
probably better, if only because we need some way of looking up the
test in the code if it fails. I'd hate for people to be randomly
hashing bits of just to find out what test is failing.

quoted

necessarily deal breakers, though it might make sense to hide them
behind a kernel option (like all other KUnit tests) so that they can
easily be excluded where they would otherwise clutter up results. (And

Currently they are under `CONFIG_RUST_KERNEL_KUNIT_TEST` -- or do you
mean something else?

Oops: I missed that (one of the issues with testing this on a
different machine which had a rust toolchain). Looks good to me.

quoted

if there's a way to properly name them, or maybe even split them into
per-file or per-module suites, that would make them a bit easier to
deal.) Additionally, there are some plans to taint the kernel[1] when

Yeah, splitting them further is definitely possible. We are also
likely splitting the `kernel` crate into several, which would also
make the suites smaller etc. so perhaps further splits may not be
needed.

Ah: I didn't realise the plan was always to have crate-specific
suites, and possibly to split things up.

The KTAP output specification does actually support arbitrary nesting
(though KUnit itself doesn't at the moment), which would potentially
be an option if (e.g.) providing the complete module nesting made
sense. I'm not convinced that'd make things easier to read, though.

quoted

Regardless, this is very neat, and I'm looking forward to taking a
closer look at it.

Thanks again for taking a look and playing with it, I am glad you
liked it! (even if it is just a first approximation, and only supports
the `kernel` crate, etc.).

Cheers,
Miguel

Thanks,
-- David