Re: [PATCH v2] x86/csum: rewrite csum_partial()

On Fri, Nov 12, 2021 at 8:19 AM Eric Dumazet [off-list ref] wrote:

From: Eric Dumazet <edumazet@google.com>

With more NIC supporting CHECKSUM_COMPLETE, and IPv6 being widely used.
csum_partial() is heavily used with small amount of bytes,
and is consuming many cycles.

IPv6 header size for instance is 40 bytes.

Another thing to consider is that NET_IP_ALIGN is 0 on x86,
meaning that network headers are not word-aligned, unless
the driver forces this.

This means that csum_partial() fetches one u16
to 'align the buffer', then perform three u64 additions
with carry in a loop, then a remaining u32, then a remaining u16.

With this new version, we perform a loop only for the 64 bytes blocks,
then the remaining is bisected.

Tested on various cpus, all of them show a big reduction in
csum_partial() cost (by 50 to 80 %)

v3: - use "+r" (temp64) asm constraints (Andrew).
    - fold do_csum() in csum_partial(), as gcc does not inline it.
    - fix bug added in v2 for the "odd" case.
    - back using addcq, as Andrew pointed the clang bug that was adding
          a stall on my hosts.
      (separate patch to add32_with_carry() will follow)
    - use load_unaligned_zeropad() for final 1-7 bytes (Peter & Alexander).

v2: - removed the hard-coded switch(), as it was not RETPOLINE aware.
    - removed the final add32_with_carry() that we were doing
      in csum_partial(), we can simply pass @sum to do_csum().

Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Alexander Duyck <redacted>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Andrew Cooper <redacted>
---
 arch/x86/lib/csum-partial_64.c | 162 ++++++++++++++-------------------
 1 file changed, 67 insertions(+), 95 deletions(-)

Looks good to me.

Reviewed-by: Alexander Duyck <alexanderduyck@fb.com>