Jakub Kicinski wrote:

Hi Willem!

We discussed instability of txtimestamp.sh in the past but it has
gotten even worse after we migrated from AWS to netdev foundation
machines. Possibly because it's different HW. Possibly because we
now run much newer kernels (AWS Linux vs Fedora).

The test flakes a lot (we're talking about non-debug builds):
https://netdev.bots.linux.dev/contest.html?test=txtimestamp-sh

I tried a few things. The VM threads (vCPU, not IO) are now all pinned
to dedicated CPUs. I added this patch to avoid long idle periods:
https://github.com/linux-netdev/testing/commit/d468f582c617adece2a576788746a09d91e91574

These both help a little bit, but w still get 10+ flakes a week.
I believe you have access to netdev foundation machines so feel
free to poke if you have cycles..

From a first look at the most recent 20 flakes 
(ignoring two unrelated sockaddr failures).

17 out of 20 happen in the first SND-USR calculation.
One representative example:

    # 7.11 [+0.00] test SND
    # 7.11 [+0.00]     USR: 1767443466 s 155019 us (seq=0, len=0)
    # 7.19 [+0.08] ERROR: 18600 us expected between 10000 and 18000
    # 7.19 [+0.00]     SND: 1767443466 s 173619 us (seq=0, len=10)  (USR +18599 us)
    # 7.20 [+0.00]     USR: 1767443466 s 243683 us (seq=0, len=0)
    # 7.27 [+0.07]     SND: 1767443466 s 253690 us (seq=1, len=10)  (USR +10006 us)
    # 7.27 [+0.00]     USR: 1767443466 s 323746 us (seq=0, len=0)
    # 7.35 [+0.08]     SND: 1767443466 s 333752 us (seq=2, len=10)  (USR +10006 us)
    # 7.35 [+0.00]     USR: 1767443466 s 403811 us (seq=0, len=0)
    # 7.43 [+0.08]     SND: 1767443466 s 413817 us (seq=3, len=10)  (USR +10006 us)
    # 7.43 [+0.00]     USR-SND: count=4, avg=12154 us, min=10006 us, max=18599 us

These are just outside the bounds of 18000. So increasing the
tolerance in txtimestamp.sh will probably mitigate them. All 17
would have passed with the following change.

-        local -r args="$@ -v 10000 -V 60000 -t 8000 -S 80000"
+        local -r args="$@ -v 10000 -V 60000 -t 8000 -S 100000"

Admittedly a hacky workaround that will only reduce the rate.

It's interesting that

- every time it is the first of the four measurements that fails.
- it never seems to occur for TCP sockets.