From: Eric Dumazet <hidden> Date: 2021-11-11 06:53:29
From: Eric Dumazet <edumazet@google.com>
With more NIC supporting CHECKSUM_COMPLETE, and IPv6 being widely used.
csum_partial() is heavily used with small amount of bytes,
and is consuming many cycles.
IPv6 header size for instance is 40 bytes.
Another thing to consider is that NET_IP_ALIGN is 0 on x86,
meaning that network headers in RX path are not word-aligned,
unless the driver forces this.
This means that csum_partial() fetches one u16
to 'align the buffer', then perform seven u64 additions
with carry in a loop, then a remaining u32, then a remaining u16.
With this new version, we perform 10 u32 adds with carry, to
avoid the expensive 64->32 transformation. Using 5 u64 adds
plus one add32_with_carry() is more expensive.
Also note that this avoids loops for less than ~60 bytes.
Tested on various cpus, all of them show a big reduction in
csum_partial() cost (by 30 to 75 %)
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Alexander Duyck <redacted>
---
arch/x86/lib/csum-partial_64.c | 146 +++++++++++++++++----------------
1 file changed, 77 insertions(+), 69 deletions(-)
From: Peter Zijlstra <peterz@infradead.org> Date: 2021-11-11 09:10:30
On Wed, Nov 10, 2021 at 10:53:22PM -0800, Eric Dumazet wrote:
+ /*
+ * This implements an optimized version of
+ * switch (dwords) {
+ * case 15: res = add_with_carry(res, buf32[14]); fallthrough;
+ * case 14: res = add_with_carry(res, buf32[13]); fallthrough;
+ * case 13: res = add_with_carry(res, buf32[12]); fallthrough;
+ * ...
+ * case 3: res = add_with_carry(res, buf32[2]); fallthrough;
+ * case 2: res = add_with_carry(res, buf32[1]); fallthrough;
+ * case 1: res = add_with_carry(res, buf32[0]); fallthrough;
+ * }
+ *
+ * "adcl 8byteoff(%reg1),%reg2" are using either 3 or 4 bytes.
+ */
+ asm(" call 1f\n"
+ "1: pop %[dest]\n"
That's terrible. I think on x86_64 we can do: lea (%%rip), %[dest], not
sure what would be the best way on i386.
That's an indirect branch, you can't do that these days. This would need
to use JMP_NOSPEC (except we don't have a !ASSEMBLER version of that.
But that would also completely and utterly destroy performance.
Also, objtool would complain about this if it hadn't tripped over that
first instruction:
arch/x86/lib/csum-partial_64.o: warning: objtool: do_csum()+0x84: indirect jump found in RETPOLINE build
I'm not sure what the best way is to unroll loops without using computed
gotos/jump-tables though :/
From: Peter Zijlstra <peterz@infradead.org> Date: 2021-11-11 09:44:30
On Thu, Nov 11, 2021 at 10:10:19AM +0100, Peter Zijlstra wrote:
On Wed, Nov 10, 2021 at 10:53:22PM -0800, Eric Dumazet wrote:
quoted
+ /*
+ * This implements an optimized version of
+ * switch (dwords) {
+ * case 15: res = add_with_carry(res, buf32[14]); fallthrough;
+ * case 14: res = add_with_carry(res, buf32[13]); fallthrough;
+ * case 13: res = add_with_carry(res, buf32[12]); fallthrough;
+ * ...
+ * case 3: res = add_with_carry(res, buf32[2]); fallthrough;
+ * case 2: res = add_with_carry(res, buf32[1]); fallthrough;
+ * case 1: res = add_with_carry(res, buf32[0]); fallthrough;
+ * }
+ *
+ * "adcl 8byteoff(%reg1),%reg2" are using either 3 or 4 bytes.
+ */
+ asm(" call 1f\n"
+ "1: pop %[dest]\n"
That's terrible. I think on x86_64 we can do: lea (%%rip), %[dest], not
sure what would be the best way on i386.
That's an indirect branch, you can't do that these days. This would need
to use JMP_NOSPEC (except we don't have a !ASSEMBLER version of that.
But that would also completely and utterly destroy performance.
Also, objtool would complain about this if it hadn't tripped over that
first instruction:
arch/x86/lib/csum-partial_64.o: warning: objtool: do_csum()+0x84: indirect jump found in RETPOLINE build
I'm not sure what the best way is to unroll loops without using computed
gotos/jump-tables though :/
From: Eric Dumazet <edumazet@google.com> Date: 2021-11-11 16:02:23
On Thu, Nov 11, 2021 at 1:44 AM Peter Zijlstra [off-list ref] wrote:
quoted hunk
On Thu, Nov 11, 2021 at 10:10:19AM +0100, Peter Zijlstra wrote:
quoted
On Wed, Nov 10, 2021 at 10:53:22PM -0800, Eric Dumazet wrote:
quoted
+ /*
+ * This implements an optimized version of
+ * switch (dwords) {
+ * case 15: res = add_with_carry(res, buf32[14]); fallthrough;
+ * case 14: res = add_with_carry(res, buf32[13]); fallthrough;
+ * case 13: res = add_with_carry(res, buf32[12]); fallthrough;
+ * ...
+ * case 3: res = add_with_carry(res, buf32[2]); fallthrough;
+ * case 2: res = add_with_carry(res, buf32[1]); fallthrough;
+ * case 1: res = add_with_carry(res, buf32[0]); fallthrough;
+ * }
+ *
+ * "adcl 8byteoff(%reg1),%reg2" are using either 3 or 4 bytes.
+ */
+ asm(" call 1f\n"
+ "1: pop %[dest]\n"
That's terrible. I think on x86_64 we can do: lea (%%rip), %[dest], not
sure what would be the best way on i386.
That's an indirect branch, you can't do that these days. This would need
to use JMP_NOSPEC (except we don't have a !ASSEMBLER version of that.
But that would also completely and utterly destroy performance.
Also, objtool would complain about this if it hadn't tripped over that
first instruction:
arch/x86/lib/csum-partial_64.o: warning: objtool: do_csum()+0x84: indirect jump found in RETPOLINE build
I'm not sure what the best way is to unroll loops without using computed
gotos/jump-tables though :/
Thanks Peter !
This is more or less the first version I wrote. (I was doing tests for
(len & 32), (len & 16) .. to not have to update len in these blocks.
Then, I tried to add an inline version, a la ip_fast_csum() but for IPv6.
Then I came up with the version I sent, for some reason my .config had
temporarily disabled CONFIG_RETPOLINE,
thanks for reminding me this !
I also missed this warning anyway :
arch/x86/lib/csum-partial_64.o: warning: objtool: csum_partial()+0x2f:
unannotated intra-function call
I will spend a bit more time on this before sending a V2, thanks again !
From: Alexander Duyck <hidden> Date: 2021-11-11 16:51:52
On Wed, Nov 10, 2021 at 10:53 PM Eric Dumazet [off-list ref] wrote:
quoted hunk
From: Eric Dumazet <edumazet@google.com>
With more NIC supporting CHECKSUM_COMPLETE, and IPv6 being widely used.
csum_partial() is heavily used with small amount of bytes,
and is consuming many cycles.
IPv6 header size for instance is 40 bytes.
Another thing to consider is that NET_IP_ALIGN is 0 on x86,
meaning that network headers in RX path are not word-aligned,
unless the driver forces this.
This means that csum_partial() fetches one u16
to 'align the buffer', then perform seven u64 additions
with carry in a loop, then a remaining u32, then a remaining u16.
With this new version, we perform 10 u32 adds with carry, to
avoid the expensive 64->32 transformation. Using 5 u64 adds
plus one add32_with_carry() is more expensive.
Also note that this avoids loops for less than ~60 bytes.
Tested on various cpus, all of them show a big reduction in
csum_partial() cost (by 30 to 75 %)
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Alexander Duyck <redacted>
---
arch/x86/lib/csum-partial_64.c | 146 +++++++++++++++++----------------
1 file changed, 77 insertions(+), 69 deletions(-)
@@ -21,97 +21,105 @@ static inline unsigned short from32to16(unsigned a)}/*-*Doa64-bitchecksumonanarbitrarymemoryarea.+*Doachecksumonanarbitrarymemoryarea.*Returnsa32bitchecksum.**Thisisn'tastimecriticalasitusedtobebecausemanyNICs*dohardwarechecksummingthesedays.-*-*Thingstriedandfoundtonotmakeitfaster:-*ManualPrefetching-*Unrollingtoan128bytesinnerloop.-*Usinginterleavingwithmoreregisterstobreakthecarrychains.+*+*Still,withCHECKSUM_COMPLETEthisiscalledtocompute+*checksumsonIPv6headers(40bytes)andothersmallparts.*/staticunsigneddo_csum(constunsignedchar*buff,unsignedlen){-unsignedodd,count;-unsignedlongresult=0;+unsignedlongdwords;+unsignedodd,result=0;-if(unlikely(len==0))-returnresult;odd=1&(unsignedlong)buff;if(unlikely(odd)){+if(unlikely(len==0))+returnresult;result=*buff<<8;len--;buff++;}-count=len>>1;/* nr of 16-bit words.. */-if(count){-if(2&(unsignedlong)buff){-result+=*(unsignedshort*)buff;-count--;-len-=2;-buff+=2;-}-count>>=1;/* nr of 32-bit words.. */-if(count){-unsignedlongzero;-unsignedcount64;-if(4&(unsignedlong)buff){-result+=*(unsignedint*)buff;-count--;-len-=4;-buff+=4;-}-count>>=1;/* nr of 64-bit words.. */
So for most cases getting rid of the alignment code should be fine.
However as I recall my main concern when dealing with something like
this was the case of a page straddling value. For most network packets
that shouldn't be the case but it may be something we want to add some
sort of debug check for where if we are unaligned, and the address
straddles pages, and the debugging is enabled we throw a warning at
least.
Otherwise as an alternative we might consider just performing one 8B
read at the start with us forcing the alignment via masking the
address and data and then just dropping the unused values in order get
us into 8B alignment so that we don't trigger the page straddling
read, and we can just rotate the result later to align it to the
start.
So in terms of this main loop it is probably as good as it gets for
anything pre-Haswell, but I am wondering if we might be able to
improve upon this for Haswell and newer architectures that have more
read throughput since I think those parts would be serialized on the
carry flag instead of being limited to one read per cycle.
+ dwords = len >> 2;
+ if (dwords) { /* dwords is in [1..15] */
+ unsigned long dest;
- /* last up to 7 8byte blocks */
- count %= 8;
- while (count) {
- asm("addq %1,%0\n\t"
- "adcq %2,%0\n"
- : "=r" (result)
- : "m" (*(unsigned long *)buff),
- "r" (zero), "0" (result));
- --count;
- buff += 8;
- }
- result = add32_with_carry(result>>32,
- result&0xffffffff);
+ /*
+ * This implements an optimized version of
+ * switch (dwords) {
+ * case 15: res = add_with_carry(res, buf32[14]); fallthrough;
+ * case 14: res = add_with_carry(res, buf32[13]); fallthrough;
+ * case 13: res = add_with_carry(res, buf32[12]); fallthrough;
+ * ...
+ * case 3: res = add_with_carry(res, buf32[2]); fallthrough;
+ * case 2: res = add_with_carry(res, buf32[1]); fallthrough;
+ * case 1: res = add_with_carry(res, buf32[0]); fallthrough;
+ * }
+ *
+ * "adcl 8byteoff(%reg1),%reg2" are using either 3 or 4 bytes.
+ */
+ asm(" call 1f\n"
+ "1: pop %[dest]\n"
+ " lea (2f-1b)(%[dest],%[skip],4),%[dest]\n"
+ " clc\n"
+ " jmp *%[dest]\n .align 4\n"
+ "2:\n"
+ " adcl 14*4(%[src]),%[res]\n .align 4\n"
+ " adcl 13*4(%[src]),%[res]\n .align 4\n"
+ " adcl 12*4(%[src]),%[res]\n .align 4\n"
+ " adcl 11*4(%[src]),%[res]\n .align 4\n"
+ " adcl 10*4(%[src]),%[res]\n .align 4\n"
+ " adcl 9*4(%[src]),%[res]\n .align 4\n"
+ " adcl 8*4(%[src]),%[res]\n .align 4\n"
+ " adcl 7*4(%[src]),%[res]\n .align 4\n"
+ " adcl 6*4(%[src]),%[res]\n .align 4\n"
+ " adcl 5*4(%[src]),%[res]\n .align 4\n"
+ " adcl 4*4(%[src]),%[res]\n .align 4\n"
+ " adcl 3*4(%[src]),%[res]\n .align 4\n"
+ " adcl 2*4(%[src]),%[res]\n .align 4\n"
+ " adcl 1*4(%[src]),%[res]\n .align 4\n"
+ " adcl 0*4(%[src]),%[res]\n"
+ " adcl $0,%[res]"
+ : [res] "=r" (result), [dest] "=&r" (dest)
+ : [src] "r" (buff), "[res]" (result),
+ [skip] "r" (dwords ^ 15)
+ : "memory");
+ }
I gave up on this after the whole specter/meltdown thing because it
was an indirect jump. Packing it down to a tight loop processing
single QWORDs may be in our favor as it can just replay the decoded
instructions for as many times as we need to.
- if (len & 4) {
- result += *(unsigned int *) buff;
- buff += 4;
- }
- }
+ if (len & 3U) {
+ buff += len & ~3U;
+ result = from32to16(result);
if (len & 2) {
result += *(unsigned short *) buff;
buff += 2;
}
+ if (len & 1)
+ result += *buff;
}
- if (len & 1)
- result += *buff;
This is another spot where I wonder if we can't get away with a single
8B read and then just mask the value based on the length remaining. I
would think it would save us a fair bit of time and a number of
cycles.
- result = add32_with_carry(result>>32, result & 0xffffffff);
if (unlikely(odd)) {
result = from32to16(result);
result = ((result >> 8) & 0xff) | ((result & 0xff) << 8);
Rather than doing the fold and rotate if odd would it maybe make sense
for us just to do a single 64b rotate based on the offset of the
original value. We could do that before the result is folded to a 32b
value and it would likely only cost us one cycle instead of the
several that are being spent here doing the test for zero, fold, and
rotate.
From: Eric Dumazet <edumazet@google.com> Date: 2021-11-11 16:53:07
On Thu, Nov 11, 2021 at 8:02 AM Eric Dumazet [off-list ref] wrote:
Thanks Peter !
This is more or less the first version I wrote. (I was doing tests for
(len & 32), (len & 16) .. to not have to update len in these blocks.
Then, I tried to add an inline version, a la ip_fast_csum() but for IPv6.
Then I came up with the version I sent, for some reason my .config had
temporarily disabled CONFIG_RETPOLINE,
thanks for reminding me this !
I also missed this warning anyway :
arch/x86/lib/csum-partial_64.o: warning: objtool: csum_partial()+0x2f:
unannotated intra-function call
I will spend a bit more time on this before sending a V2, thanks again !
BTW, I could not understand why :
result = add32_with_carry(result, *(u32 *)buff);
generates this code :
123: 41 8b 09 mov (%r9),%ecx
126: 89 4d f8 mov %ecx,-0x8(%rbp)
129: 03 45 f8 add -0x8(%rbp),%eax
12c: 83 d0 00 adc $0x0,%eax
Apparently add32_with_carry() forces the use of use of a temporary in memory
While
asm(" addl 0*4(%[src]),%[res]\n"
" adcl $0,%[res]\n"
: [res] "=r" (result)
: [src] "r" (buff), "[res]" (result)
: "memory");
gives
120: 41 03 01 add (%r9),%eax
123: 83 d0 00 adc $0x0,%eax
From: Andrew Cooper <hidden> Date: 2021-11-11 18:25:24
On 11/11/2021 16:52, Eric Dumazet wrote:
On Thu, Nov 11, 2021 at 8:02 AM Eric Dumazet [off-list ref] wrote:
quoted
Thanks Peter !
This is more or less the first version I wrote. (I was doing tests for
(len & 32), (len & 16) .. to not have to update len in these blocks.
Then, I tried to add an inline version, a la ip_fast_csum() but for IPv6.
Then I came up with the version I sent, for some reason my .config had
temporarily disabled CONFIG_RETPOLINE,
thanks for reminding me this !
I also missed this warning anyway :
arch/x86/lib/csum-partial_64.o: warning: objtool: csum_partial()+0x2f:
unannotated intra-function call
I will spend a bit more time on this before sending a V2, thanks again !
BTW, I could not understand why :
result = add32_with_carry(result, *(u32 *)buff);
generates this code :
123: 41 8b 09 mov (%r9),%ecx
126: 89 4d f8 mov %ecx,-0x8(%rbp)
129: 03 45 f8 add -0x8(%rbp),%eax
12c: 83 d0 00 adc $0x0,%eax
Are you using Clang? There is a long outstanding code generation bug
where an "rm" constraint is converted to "m" internally.
https://bugs.llvm.org/show_bug.cgi?id=47530
Even a stopgap of pretending "rm" means "r" would result in far better
code, 99% of the time.
Apparently add32_with_carry() forces the use of use of a temporary in memory
While
asm(" addl 0*4(%[src]),%[res]\n"
" adcl $0,%[res]\n"
: [res] "=r" (result)
: [src] "r" (buff), "[res]" (result)
: "memory");
Just as a minor note about the asm constraints here and elsewhere
: [res] "=r" (result)
: "res" (result)
ought to be just [res] "+r" (result). The result variable really is
read and written by the asm fragments.
~Andrew
From: Eric Dumazet <edumazet@google.com> Date: 2021-11-11 19:03:04
On Thu, Nov 11, 2021 at 10:18 AM Andrew Cooper
[off-list ref] wrote:
On 11/11/2021 16:52, Eric Dumazet wrote:
quoted
On Thu, Nov 11, 2021 at 8:02 AM Eric Dumazet [off-list ref] wrote:
quoted
Thanks Peter !
This is more or less the first version I wrote. (I was doing tests for
(len & 32), (len & 16) .. to not have to update len in these blocks.
Then, I tried to add an inline version, a la ip_fast_csum() but for IPv6.
Then I came up with the version I sent, for some reason my .config had
temporarily disabled CONFIG_RETPOLINE,
thanks for reminding me this !
I also missed this warning anyway :
arch/x86/lib/csum-partial_64.o: warning: objtool: csum_partial()+0x2f:
unannotated intra-function call
I will spend a bit more time on this before sending a V2, thanks again !
BTW, I could not understand why :
result = add32_with_carry(result, *(u32 *)buff);
generates this code :
123: 41 8b 09 mov (%r9),%ecx
126: 89 4d f8 mov %ecx,-0x8(%rbp)
129: 03 45 f8 add -0x8(%rbp),%eax
12c: 83 d0 00 adc $0x0,%eax
Are you using Clang? There is a long outstanding code generation bug
where an "rm" constraint is converted to "m" internally.
Yes, this is what I realized later. This is a clang bug.
Apparently add32_with_carry() forces the use of use of a temporary in memory
While
asm(" addl 0*4(%[src]),%[res]\n"
" adcl $0,%[res]\n"
: [res] "=r" (result)
: [src] "r" (buff), "[res]" (result)
: "memory");
Just as a minor note about the asm constraints here and elsewhere
: [res] "=r" (result)
: "res" (result)
ought to be just [res] "+r" (result). The result variable really is
read and written by the asm fragments.
From: David Laight <hidden> Date: 2021-11-14 13:07:36
From: Eric Dumazet
Sent: 11 November 2021 06:53
With more NIC supporting CHECKSUM_COMPLETE, and IPv6 being widely used.
csum_partial() is heavily used with small amount of bytes,
and is consuming many cycles.
IPv6 header size for instance is 40 bytes.
Another thing to consider is that NET_IP_ALIGN is 0 on x86,
meaning that network headers in RX path are not word-aligned,
unless the driver forces this.
This means that csum_partial() fetches one u16
to 'align the buffer', then perform seven u64 additions
with carry in a loop, then a remaining u32, then a remaining u16.
With this new version, we perform 10 u32 adds with carry, to
avoid the expensive 64->32 transformation. Using 5 u64 adds
plus one add32_with_carry() is more expensive.
Also note that this avoids loops for less than ~60 bytes.
I spent far too long looking at this code a while back :-)
I did post a patch - probably 10th May 2020.
Prior to Sandy bridge ADC always took two clocks.
Even on Sandy bridge there is a two clock delay for the sum,
only the carry flag is available earlier.
Broadwell (and I think all AMD cpu) do ADC in 1 clock.
This can be avoided by adding to alternate registers.
There are also issues on some cpu with the partial updates
to the flags register (DEC sets Z but not C) causing unwanted
register dependencies.
I think misaligned memory reads take an extra clock.
But more recent cpu can do two memory reads per clock.
So unless the code is trying to beat 8 bytes/clock it
shouldn't have much effect.
The fastest loop I found (for large buffers) used:
+ asm( " bt $4, %[len]\n"
+ " jnc 10f\n"
+ " add (%[buff], %[len]), %[sum_0]\n"
+ " adc 8(%[buff], %[len]), %[sum_1]\n"
+ " lea 16(%[len]), %[len]\n"
+ "10: jecxz 20f\n"
+ " adc (%[buff], %[len]), %[sum_0]\n"
+ " adc 8(%[buff], %[len]), %[sum_1]\n"
+ " lea 32(%[len]), %[len_tmp]\n"
+ " adc 16(%[buff], %[len]), %[sum_0]\n"
+ " adc 24(%[buff], %[len]), %[sum_1]\n"
+ " mov %[len_tmp], %[len]\n"
+ " jmp 10b\n"
+ "20: adc %[sum_0], %[sum]\n"
+ " adc %[sum_1], %[sum]\n"
+ " adc $0, %[sum]\n"
+ : [sum] "+&r" (sum), [sum_0] "+&r" (sum_0), [sum_1] "+&r" (sum_1),
+ [len] "+&c" (len), [len_tmp] "=&r" (len_tmp)
+ : [buff] "r" (buff)
+ : "memory" );
The principle is that 'buff' points to the end on the buffer.
The 'length' (in %cx) is negative and then increased until it hits zero.
This runs at 8 bytes/clock on anything recent (and approaches it on Ivy bridge).
Splitting the 'add 32' did make a slight improvement.
If you aren't worried (too much) about cpu before Bradwell then IIRC
this loop gets close to 8 bytes/clock:
+ "10: jecxz 20f\n"
+ " adc (%[buff], %[len]), %[sum]\n"
+ " adc 8(%[buff], %[len]), %[sum]\n"
+ " lea 16(%[len]), %[tmp]\n"
+ " jmp 10b\n"
+ " 20:"
I also toyed with this loop:
count = (lim + 7 - buf) & -64;
buf += count;
count = -count;
asm( " xor %[sum_odd], %[sum_odd]\n" // Also clears carry and overflow
"10: jrcxz 20f\n"
" adcx (%[buf], %[count]), %[sum]\n"
" adox 8(%[buf], %[count]), %[sum_odd]\n"
" adcx 16(%[buf], %[count]), %[sum]\n"
" adox 24(%[buf], %[count]), %[sum_odd]\n"
" adcx 32(%[buf], %[count]), %[sum]\n"
" adox 40(%[buf], %[count]), %[sum_odd]\n"
" adcx 48(%[buf], %[count]), %[sum]\n"
" adox 56(%[buf], %[count]), %[sum_odd]\n"
" lea 64(%[count]), %[count]\n"
" jmp 10b\n"
"20: adox %[count], %[sum_odd]\n" // [count] is zero
" adcx %[sum_odd], %[sum]\n"
" adcx %[count], %[sum]"
: [sum] "=&r" (sum), [count] "=&c" (count), [sum_odd] "=&r" (sum_odd)
: [buf] "r" (buf), "0" (sum), "1" (count)
: "memory");
}
My notes say it achieved 12 bytes/clock on an i7-7700.
However it is only really useful for long aligned buffers.
It is completely annoying that you can't use LOOP (dec %cx and jump nz)
because it is very slow on Intel cpu - even ones that support adox).
JCZX is fine.
It is also possible to reduce the checksum to 16 bits using:
sum = (sum % 0xffff) ^ 0xffff;
I think this is faster if gcc uses a 'multiply by reciprocal'
but (in my tests) it sometimes used a divide.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)
From: David Laight <hidden> Date: 2021-11-14 14:12:38
From: David Laight
Sent: 14 November 2021 13:07
..
If you aren't worried (too much) about cpu before Bradwell then IIRC
this loop gets close to 8 bytes/clock:
+ "10: jecxz 20f\n"
+ " adc (%[buff], %[len]), %[sum]\n"
+ " adc 8(%[buff], %[len]), %[sum]\n"
+ " lea 16(%[len]), %[tmp]\n"
+ " jmp 10b\n"
+ " 20:"
It is even possible a loop based on:
10: adc (%[buff], %[len], 8), %sum
inc %[len]
jnz 10b
will run at 8 bytes per clock on very recent Intel cpu.
The 'adc' needs P06 and P23, the 'inc' P0156 and the
'jnz' P6 (predicted taken) (on Broadwell and probably later).
(The 'inc' and 'jnz' might alse be fusable to a single P6 u-op.)
Using 'lea' instead of 'inc' constrains it to P15.
That might actually generate better scheduling since it
is guaranteed to 'miss' the 'adc'.
So if the right ports are selected it is possible to
execute all the instructions in parallel.
It certainly isn't necessary to unroll the loop any more
than two reads for Bradwell onwards.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)
From: David Laight <hidden> Date: 2021-11-15 10:23:43
From: David Laight
Sent: 14 November 2021 14:12
..
quoted
If you aren't worried (too much) about cpu before Broadwell then IIRC
this loop gets close to 8 bytes/clock:
+ "10: jecxz 20f\n"
+ " adc (%[buff], %[len]), %[sum]\n"
+ " adc 8(%[buff], %[len]), %[sum]\n"
+ " lea 16(%[len]), %[tmp]\n"
+ " jmp 10b\n"
+ " 20:"
It is even possible a loop based on:
10: adc (%[buff], %[len], 8), %sum
inc %[len]
jnz 10b
will run at 8 bytes per clock on very recent Intel cpu.
It doesn't on i7-7700.
(which I probably tested last year).
But the first loop does run twice as fast - and will only
be beaten by the adcx/adox loop.
So there is no need to unroll to more than 2 reads/loop.
For cpu between Ivy bridge and Broadwell you want to use
separate 'sum' registers to avoid the 2 clock latency
of the adc result.
That should beat the 4 bytes/clock of the current loop.
But does need an extra unroll to get near 8 bytes/clock.
For older cpu (nehalem/core2) the 'jecxz' loop is about the
only way to 'loop carry' the carry flag without the
6 clock penalty for the partial flags register update.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)