Commit 694fc88ce271f ("powerpc/string: Implement optimized
memset variants") added memset16(), memset32() and memset64()
for the 64 bits PPC.
On 32 bits, memset64() is not relevant, and as shown below,
the generic version of memset32() gives a good code, so only
memset16() is candidate for an optimised version.
000009c0 <memset32>:
9c0: 2c 05 00 00 cmpwi r5,0
9c4: 39 23 ff fc addi r9,r3,-4
9c8: 4d 82 00 20 beqlr
9cc: 7c a9 03 a6 mtctr r5
9d0: 94 89 00 04 stwu r4,4(r9)
9d4: 42 00 ff fc bdnz 9d0 <memset32+0x10>
9d8: 4e 80 00 20 blr
The last part of memset() handling the not 4-bytes multiples
operates on bytes, making it unsuitable for handling word without
modification. As it would increase memset() complexity, it is
better to implement memset16() from scratch. In addition it
has the advantage of allowing a more optimised memset16() than what
we would have by using the memset() function.
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/include/asm/string.h | 4 +++-
arch/powerpc/lib/copy_32.S | 14 ++++++++++++++
2 files changed, 17 insertions(+), 1 deletion(-)
Commit 9445aa1a3062a ("ppc: move exports to definitions")
added EXPORT_SYMBOL() for memset() and flush_hash_pages() in
the middle of the functions.
This patch moves them at the end of the two functions.
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/lib/copy_32.S | 2 +-
arch/powerpc/mm/hash_low_32.S | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
memset() is patched after initialisation to activate the
optimised part which uses cache instructions.
Today we have a 'b 2f' to skip the optimised patch, which then gets
replaced by a NOP, implying a useless cycle consumption.
As we have a 'bne 2f' just before, we could use that instruction
for the live patching, hence removing the need to have a
dedicated 'b 2f' to be replaced by a NOP.
This patch changes the 'bne 2f' by a 'b 2f'. During init, that
'b 2f' is then replaced by 'bne 2f'
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/kernel/setup_32.c | 7 ++++++-
arch/powerpc/lib/copy_32.S | 7 +++++--
2 files changed, 11 insertions(+), 3 deletions(-)
@@ -98,6 +98,9 @@ extern unsigned int memset_nocache_branch; /* Insn to be replaced by NOP */notracevoid__initmachine_init(u64dt_ptr){+unsignedint*addr=&memset_nocache_branch;+unsignedlonginsn;+/* Configure static keys first, now that we're relocated. */setup_feature_keys();
@@ -105,7 +108,9 @@ notrace void __init machine_init(u64 dt_ptr)udbg_early_init();patch_instruction((unsignedint*)&memcpy,PPC_INST_NOP);-patch_instruction(&memset_nocache_branch,PPC_INST_NOP);++insn=create_cond_branch(addr,branch_target(addr),0x820000);+patch_instruction(addr,insn);/* replace b by bne cr0 *//* Do some early initialization based on the flat device tree */early_init_devtree(__va(dt_ptr));
There is no need to extend the set value to an int when the length
is lower than 4 as in that case we only do byte stores.
We can therefore immediately branch to the part handling it.
By separating it from the normal case, we are able to eliminate
a few actions on the destination pointer.
Signed-off-by: Christophe Leroy <redacted>
---
arch/powerpc/lib/copy_32.S | 21 ++++++++++++++-------
1 file changed, 14 insertions(+), 7 deletions(-)
From: Michael Ellerman <mpe@ellerman.id.au> Date: 2017-08-24 10:51:06
Christophe Leroy [off-list ref] writes:
memset() is patched after initialisation to activate the
optimised part which uses cache instructions.
Today we have a 'b 2f' to skip the optimised patch, which then gets
replaced by a NOP, implying a useless cycle consumption.
As we have a 'bne 2f' just before, we could use that instruction
for the live patching, hence removing the need to have a
dedicated 'b 2f' to be replaced by a NOP.
This patch changes the 'bne 2f' by a 'b 2f'. During init, that
'b 2f' is then replaced by 'bne 2f'
I'm not sure what the sequence is during boot for the 32-bit code, but
can you use an ALT_FTR section for this? Possibly that doesn't get done
at the right time though.
cheers
memset() is patched after initialisation to activate the
optimised part which uses cache instructions.
Today we have a 'b 2f' to skip the optimised patch, which then gets
replaced by a NOP, implying a useless cycle consumption.
As we have a 'bne 2f' just before, we could use that instruction
for the live patching, hence removing the need to have a
dedicated 'b 2f' to be replaced by a NOP.
This patch changes the 'bne 2f' by a 'b 2f'. During init, that
'b 2f' is then replaced by 'bne 2f'
I'm not sure what the sequence is during boot for the 32-bit code, but
can you use an ALT_FTR section for this? Possibly that doesn't get done
at the right time though.
Unfortunately, as we discussed in 2015
(https://lkml.org/lkml/2015/9/10/608), the ALT_FTR does things too
early, while the cache is not enabled yet.
Christophe
From: Michael Ellerman <mpe@ellerman.id.au> Date: 2017-08-25 00:15:10
Christophe LEROY [off-list ref] writes:
Le 24/08/2017 =C3=A0 12:51, Michael Ellerman a =C3=A9crit=C2=A0:
quoted
Christophe Leroy [off-list ref] writes:
=20
quoted
memset() is patched after initialisation to activate the
optimised part which uses cache instructions.
Today we have a 'b 2f' to skip the optimised patch, which then gets
replaced by a NOP, implying a useless cycle consumption.
As we have a 'bne 2f' just before, we could use that instruction
for the live patching, hence removing the need to have a
dedicated 'b 2f' to be replaced by a NOP.
This patch changes the 'bne 2f' by a 'b 2f'. During init, that
'b 2f' is then replaced by 'bne 2f'
=20
I'm not sure what the sequence is during boot for the 32-bit code, but
can you use an ALT_FTR section for this? Possibly that doesn't get done
at the right time though.
Haha, you expect me to remember things I said then! ;)
the ALT_FTR does things too early, while the cache is not enabled yet.
OK. Ben did do some reworks to the early init since then, but I don't
think he changed that.
I notice we do setup_feature_keys() in machine_init(), which is the jump
label equivalent of apply_feature_fixups(). So I wonder if we could
actually move apply_feature_fixups() to there. But it would need some
serious review.
cheers
From: Michael Ellerman <hidden> Date: 2017-09-01 13:30:00
On Wed, 2017-08-23 at 14:54:32 UTC, Christophe Leroy wrote:
Commit 694fc88ce271f ("powerpc/string: Implement optimized
memset variants") added memset16(), memset32() and memset64()
for the 64 bits PPC.
On 32 bits, memset64() is not relevant, and as shown below,
the generic version of memset32() gives a good code, so only
memset16() is candidate for an optimised version.
000009c0 <memset32>:
9c0: 2c 05 00 00 cmpwi r5,0
9c4: 39 23 ff fc addi r9,r3,-4
9c8: 4d 82 00 20 beqlr
9cc: 7c a9 03 a6 mtctr r5
9d0: 94 89 00 04 stwu r4,4(r9)
9d4: 42 00 ff fc bdnz 9d0 <memset32+0x10>
9d8: 4e 80 00 20 blr
The last part of memset() handling the not 4-bytes multiples
operates on bytes, making it unsuitable for handling word without
modification. As it would increase memset() complexity, it is
better to implement memset16() from scratch. In addition it
has the advantage of allowing a more optimised memset16() than what
we would have by using the memset() function.
Signed-off-by: Christophe Leroy <redacted>