Re: [dpdk-dev] [PATCH v2] guides: add a testing guide for developing tests

On Tue, Mar 02, 2021 at 10:26:59AM -0500, Aaron Conole wrote:

Bruce Richardson [off-list ref] writes:

quoted

On Tue, Mar 02, 2021 at 10:07:26AM +0100, David Marchand wrote:

quoted

On Wed, Feb 10, 2021 at 3:56 PM Aaron Conole [off-list ref] wrote:

quoted

diff --git a/doc/guides/contributing/testing.rst b/doc/guides/contributing/testing.rst
new file mode 100644
index 0000000000..86ca24ce43
--- /dev/null
+++ b/doc/guides/contributing/testing.rst

@@ -0,0 +1,245 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright 2018 The DPDK contributors

2021?

Whoops, I forgot to update my time machine.

quoted

quoted

quoted

+
+.. _testing_guidelines:

I can't find a call to the testing_guidelines reference, so this can be removed.

done.

quoted

quoted

[snip]

quoted

+The suites can be selected by adding the ``--suite`` option to the
+``meson test`` command.  Ex: ``meson test --suite fast-tests``::
+
+  $ meson test -C build --suite fast-tests
+  ninja: Entering directory `/home/aconole/git/dpdk/build'
+  [2543/2543] Linking target app/test/dpdk-test.
+  1/60 DPDK:fast-tests / acl_autotest          OK       3.17 s
+  2/60 DPDK:fast-tests / bitops_autotest       OK       0.22 s
+  3/60 DPDK:fast-tests / byteorder_autotest    OK       0.22 s
+  4/60 DPDK:fast-tests / cmdline_autotest      OK       0.28 s
+  5/60 DPDK:fast-tests / common_autotest       OK       0.57 s
+  6/60 DPDK:fast-tests / cpuflags_autotest     OK       0.27 s
+  ...

Trying this in my build env, I get all tests failing.
This is because I run this as a normal user, but the system has
hugepages configured.
I figured this out quickly since I know the test framework (simply
added a echo 0; exit at the top of has-hugepages.sh).
But I am not sure a reader of this doc would be able to troubleshoot this.

Not sure if this is worth explaining here, or if we can enhance the
hugepage check (permissions maybe?).

I prefer to fix the hugepage check to make the tests SKIP when we don't
have hugepages accessible (so we need some kind of permission check in
there).  I will submit it separately.

quoted

quoted

[snip]

quoted

+Checking code coverage
+----------------------
+The meson build system supports generating a code coverage report
+via the `-Db_coverage=true` option, in conjunction with a package
+like **lcov**, to generate an HTML code coverage report.  Example::
+
+  $ meson covered -Db_coverage=true

At first, I read "covered" as a meson command :-).
I prefer an explicit "meson setup covered -Db_coverage=true", but well
this is more a matter of taste.

I also tend to prefer the build directory name at the end of the command,
so I'd suggest: "meson setup -Db_coverage=true covered".  Furthermore,
while I can understand the use of "covered" as a build directory name, I
think for consistency across all docs, we should just use "build" here as
the directory name, which again will reduce confusion.  "meson setup
-Db_coverage build"

Okay - I will paint this bikeshed like:

   meson setup build -Db_coverage=true

It's a little inconsistent everywhere - so I guess a good janitor
project would be to clean up all the places we have meson commands.

Otherwise, what I've found is that the options generally come after the
build directory / command is specified (ex: see octeontx, the arm64
cross build docs, etc.) so I'll keep that for consistency there.
Hopefully we will consistently become more consistent :)

That's some fine bikeshed painting! And I'm fine with all your suggestion.

/Bruce