Thread (11 messages) 11 messages, 2 authors, 2026-01-12

Re: [TEST] txtimestamp.sh pains after netdev foundation migration

From: Willem de Bruijn <willemdebruijn.kernel@gmail.com>
Date: 2026-01-12 03:28:41

Willem de Bruijn wrote:
quoted hunk ↗ jump to hunk
Willem de Bruijn wrote:
quoted
Jakub Kicinski wrote:
quoted
On Thu, 08 Jan 2026 14:02:15 -0500 Willem de Bruijn wrote:
quoted
Increasing tolerance should work.

The current values are pragmatic choices to be so low as to minimize
total test runtime, but high enough to avoid flakes. Well..

If increasing tolerance, we also need to increase the time the test
waits for all notifications to arrive, cfg_sleep_usec.
To be clear the theory is that we got scheduled out between taking the
USR timestamp and sending the packet. But once the packet is in the
kernel it seems to flow, so AFAIU cfg_sleep_usec can remain untouched.

Thinking about it more - maybe what blocks us is the print? Maybe under
vng there's a non-trivial chance that a print to stderr ends up
blocking on serial and schedules us out? I mean maybe we should:
diff --git a/tools/testing/selftests/net/txtimestamp.c b/tools/testing/selftests/net/txtimestamp.c
index abcec47ec2e6..e2273fdff495 100644
--- a/tools/testing/selftests/net/txtimestamp.c
+++ b/tools/testing/selftests/net/txtimestamp.c
@@ -207,12 +207,10 @@ static void __print_timestamp(const char *name, struct timespec *cur,
        fprintf(stderr, "\n");
 }
 
-static void print_timestamp_usr(void)
+static void record_timestamp_usr(void)
 {
        if (clock_gettime(CLOCK_REALTIME, &ts_usr))
                error(1, errno, "clock_gettime");
-
-       __print_timestamp("  USR", &ts_usr, 0, 0);
 }
 
 static void check_timestamp_usr(void)
@@ -636,8 +634,6 @@ static void do_test(int family, unsigned int report_opt)
                        fill_header_udp(buf + off, family == PF_INET);
                }
 
-               print_timestamp_usr();
-
                iov.iov_base = buf;
                iov.iov_len = total_len;
 
@@ -692,10 +688,14 @@ static void do_test(int family, unsigned int report_opt)
 
                }
 
+               record_timestamp_usr();
                val = sendmsg(fd, &msg, 0);
                if (val != total_len)
                        error(1, errno, "send");
 
+               /* Avoid I/O between taking ts_usr and sendmsg() */
+               __print_timestamp("  USR", &ts_usr, 0, 0);
+
                check_timestamp_usr();
 
                /* wait for all errors to be queued, else ACKs arrive OOO */
Definitely worth including.

Could it be helpful to schedule at RR or FIFO prio. Depends on the
reason for descheduling. And it only affects priority within the VM.

I'm having trouble reproducing it in vng both locally and on 
netdev-virt.

At this point, an initial obviously correct patch and observe how
much that mitigates the issue is likely the fastest way forward.
Instead of increasing tolerance, how about optionally allowing one
moderate timing error:
@@ -166,8 +167,15 @@ static void validate_timestamp(struct timespec *cur, int min_delay)
        if (cur64 < start64 + min_delay || cur64 > start64 + max_delay) {
                fprintf(stderr, "ERROR: %" PRId64 " us expected between %d and %d\n",
                                cur64 - start64, min_delay, max_delay);
-               if (!getenv("KSFT_MACHINE_SLOW"))
-                       test_failed = true;
+               if (!getenv("KSFT_MACHINE_SLOW")) {
+                       if (cfg_num_max_timing_failures &&
+                           (cur64 <= start64 + (max_delay * 2))) {
+                               cfg_num_max_timing_failures--;
+                               fprintf(stderr, "CONTINUE: ignore 1 timing failure\n");
+                       } else {
+                               test_failed = true;
+                       }
+               }
        }
 }
@@ -746,6 +755,10 @@ static void parse_opt(int argc, char **argv)
                case 'E':
                        cfg_use_epoll = true;
                        cfg_epollet = true;
+                       break;
+               case 'f':
+                       cfg_num_max_timing_failures = strtoul(optarg, NULL, 10);
+                       break;

+++ b/tools/testing/selftests/net/txtimestamp.sh
@@ -30,8 +30,8 @@ run_test_v4v6() {
        # wait for ACK to be queued
        local -r args="$@ -v 10000 -V 60000 -t 8000 -S 80000"
 
-       ./txtimestamp ${args} -4 -L 127.0.0.1
-       ./txtimestamp ${args} -6 -L ::1
+       ./txtimestamp ${args} -f 1 -4 -L 127.0.0.1
+       ./txtimestamp ${args} -f 1 -6 -L ::1
 }
and some boilerplate.

Can fold in the record_timestamp_usr() change too.

I can send this, your alternative with Suggested-by, or let me know if
you prefer to send that.

It's tricky to reproduce, but evidently on some platforms this occurs,
so not unreasonable to give some leeway. A single UDP test runs 12
timing validations: 4 packets * {SND, ENQ, END + SND} setups. A single
TCP test runs additional {ACK, SND + ACK, ENQ + SND + ACK} cases. If
we consider 1/12 skips too high, we could increase packet count. 
That should say 16 validations: ENQ + SND validates both.
 
txtimestamp.sh runs 3 * 7 * 2 test variants. Alternatively we suppress
1 failure here, rather than in the individual tests.

Any of these approaches should significantly reduce the flake rate
reported on netdev.bots.
  
Keyboard shortcuts
hback out one level
jnext message in thread
kprevious message in thread
ldrill in
Escclose help / fold thread tree
?toggle this help