On Thu, 26 Jun 2025 at 18:23, Sasha Levin [off-list ref] wrote:

On Thu, Jun 26, 2025 at 10:37:33AM +0200, Dmitry Vyukov wrote:

quoted

On Thu, 26 Jun 2025 at 10:32, Dmitry Vyukov [off-list ref] wrote:

quoted

On Wed, 25 Jun 2025 at 17:55, Sasha Levin [off-list ref] wrote:

quoted

On Wed, Jun 25, 2025 at 10:52:46AM +0200, Dmitry Vyukov wrote:

quoted

On Tue, 24 Jun 2025 at 22:04, Sasha Levin [off-list ref] wrote:

quoted

quoted

6. What's the goal of validation of the input arguments?
Kernel code must do this validation anyway, right.
Any non-trivial validation is hard, e.g. even for open the validation function
for file name would need to have access to flags and check file precense for
some flags combinations. That may add significant amount of non-trivial code
that duplicates main syscall logic, and that logic may also have bugs and
memory leaks.

Mostly to catch divergence from the spec: think of a scenario where
someone added a new param/flag/etc but forgot to update the spec - this
will help catch it.

How exactly is this supposed to work?
Even if we run with a unit test suite, a test suite may include some
incorrect inputs to check for error conditions. The framework will
report violations on these incorrect inputs. These are not bugs in the
API specifications, nor in the test suite (read false positives).

Right now it would be something along the lines of the test checking for
an expected failure message in dmesg, something along the lines of:

        https://github.com/linux-test-project/ltp/blob/0c99c7915f029d32de893b15b0a213ff3de210af/testcases/commands/sysctl/sysctl02.sh#L67

I'm not opposed to coming up with a better story...

If the goal of validation is just indirectly validating correctness of
the specification itself, then I would look for other ways of
validating correctness of the spec.
Either removing duplication between specification and actual code
(i.e. generating it from SYSCALL_DEFINE, or the other way around) ,
then spec is correct by construction. Or, cross-validating it with
info automatically extracted from the source (using
clang/dwarf/pahole).
This would be more scalable (O(1) work, rather than thousands more
manually written tests).

quoted

Oh, you mean special tests for this framework (rather than existing tests).
I don't think this is going to work in practice. Besides writing all
these specifications, we will also need to write dozens of tests per
each specification (e.g. for each fd arg one needs at least 3 tests:
-1, valid fd, inclid fd; an enum may need 5 various inputs of
something; let alone netlink specifications).

I didn't mean just for the framework: being able to specify the APIs in
machine readable format will enable us to automatically generate
exhaustive tests for each such API.

I've been playing with the kapi tool (see last patch) which already
supports different formatters. Right now it outputs human readable
output, but I have proof-of-concept code that outputs testcases for
specced APIs.

The dream here is to be able to automatically generate
hundreds/thousands of tests for each API in an automated fashion, and
verify the results with:

1. Simply checking expected return value.

2. Checking that the actual action happened (i.e. we called close(fd),
verify that `fd` is really closed).

3. Check for side effects (i.e. close(fd) isn't supposed to allocate
memory - verify that it didn't allocate memory).

4. Code coverage: our tests are supposed to cover 100% of the code in
that APIs call chain, do we have code that didn't run (missing/incorrect
specs).

This is all good. I was asking the argument verification part of the
framework. Is it required for any of this? How?