Looking into this again.

Unfortunately, couldn't see how I would do bisection on the issue, as
it appears with that 5.4.118 commit implicated by the call stack,and
with tha tremoved,is obviously gone(that I tested, 5.4.135 with
b56ad4febe67b8c0647c0a3e427e935a76dedb59 reverted runs smoothly for
me, while the original 5.4.135 with that 5.4.118 time commit in, now
on a dozen machines, throws the WARNING.

I got email on the side of someone (Igor, on Cc) who sees the same
with DELL servers, a newer 5.10 kernel, for him running IPVS + he sees
actual operational impact there.

I just had a look at Linus mainlinetree, and see there is this
followup / further fix from Peter Zijlstra,
https://github.com/torvalds/linux/commit/3a7956e25e1d7b3c148569e78895e1f3178122a9
; now I'm much too incompetent to try and backport that, as it looks
more involved, but I imagine such a backport would be needed to fix
the WARNING (or IPVS breakage of Igor) we see.

best regards
  Patrick