On Sat, Mar 13, 2021 at 06:26:15AM +0100, Sedat Dilek wrote:
x86/jump_label: Mark arguments as const to satisfy asm constraints
Where do I find this patch?
x86: Remove dynamic NOP selection
objtool,x86: Use asm/nops.h
My benchmark was to build a Linux-kernel with LLVM/Clang v12.0.0-rc3
on Debian/testing AMD64.
Patchset applied for a first build:
Performance counter stats for 'make V=1 -j4 LLVM=1 LLVM_IAS=1
PAHOLE=/opt/pahole/bin/pahole LOCALVERSION=-7-amd64-clang12-cfi
KBUILD_VERBOSE=1 KBUILD_BUILD_HOST=iniza
There's a reason I have -s for silent in the build - printing output
during the build creates a *lot* of variance. And you have excessive
printing with V=1 and KBUILD_VERBOSE=1.
Also, you need to repeat those workloads a couple of times - one is not
enough. That's why I have --repeat 5 in there.
Also, you need --pre=/root/bin/pre-build-kernel.sh where that script is:
---
#!/bin/bash
echo $0
make -s clean
echo 3 > /proc/sys/vm/drop_caches
---
so that you can avoid pagecache influence.
Lemme rerun here with clang.
--
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette
x86: Remove dynamic NOP selection
objtool,x86: Use asm/nops.h
My benchmark was to build a Linux-kernel with LLVM/Clang v12.0.0-rc3
on Debian/testing AMD64.
Patchset applied for a first build:
Performance counter stats for 'make V=1 -j4 LLVM=1 LLVM_IAS=1
PAHOLE=/opt/pahole/bin/pahole LOCALVERSION=-7-amd64-clang12-cfi
KBUILD_VERBOSE=1 KBUILD_BUILD_HOST=iniza
There's a reason I have -s for silent in the build - printing output
during the build creates a *lot* of variance. And you have excessive
printing with V=1 and KBUILD_VERBOSE=1.
Also, you need to repeat those workloads a couple of times - one is not
enough. That's why I have --repeat 5 in there.
Also, you need --pre=/root/bin/pre-build-kernel.sh where that script is:
---
#!/bin/bash
echo $0
make -s clean
echo 3 > /proc/sys/vm/drop_caches
---
so that you can avoid pagecache influence.
That's why I told earlier you to use tip/master - that patch is already
in it and all you would've needed to do is to apply the two nop patches.
--
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette
That's why I told earlier you to use tip/master - that patch is already
in it and all you would've needed to do is to apply the two nop patches.
Thanks for all your testings and suggestions.
For me it was easier to apply these 3 patches on top of my custom
patchset to see what impact Peter's patchset.
AFAICS you did a 5 times x86-64 defconfig with dropped pagecache and `make -j9`?
I run my "normal" workflow(s) (and build-script) for easier comparison
on my side.
Big thank-you for testing with LLVM/Clang v11.x - twice as slow as with GCC :-(.
A selfmade ThinLTO+PGO optimized LLVM tooolchain v11.x/v12-rcX/v13-git
is here as fast as Debian's GCC-v10.2.1 to build a Linux-kernel -
approx. 03:30 [hh:mm] - full adapted Debian v5.10.y kernel-config.
Does your distribution offer LLVM/Clang v12.0.0-rc3 (released this
week) binaries?
- Sedat -
On Sat, Mar 13, 2021 at 1:49 PM Borislav Petkov [off-list ref] wrote:
On Sat, Mar 13, 2021 at 01:38:22PM +0100, Sedat Dilek wrote:
quoted
AFAICS you did a 5 times x86-64 defconfig with dropped pagecache and `make -j9`?
The tailored .config for that particular test box.
quoted
Does your distribution offer LLVM/Clang v12.0.0-rc3 (released this
week) binaries?
The partition on that box I used is debian testing, so no:
$ apt search llvm-1* 2>/dev/null | grep llvm-1
libllvm-11-ocaml-dev/testing,testing 1:11.0.1-2 amd64
llvm-10/now 1:10.0.1-8+b1 amd64 [installed,local]
llvm-10-dev/now 1:10.0.1-8+b1 amd64 [installed,local]
llvm-10-runtime/now 1:10.0.1-8+b1 amd64 [installed,local]
llvm-10-tools/now 1:10.0.1-8+b1 amd64 [installed,local]
llvm-11/testing,testing,now 1:11.0.1-2 amd64 [installed,automatic]
llvm-11-dev/testing,testing,now 1:11.0.1-2 amd64 [installed,automatic]
llvm-11-doc/testing,testing 1:11.0.1-2 all
llvm-11-examples/testing,testing 1:11.0.1-2 all
llvm-11-runtime/testing,testing,now 1:11.0.1-2 amd64 [installed,automatic]
llvm-11-tools/testing,testing,now 1:11.0.1-2 amd64 [installed,automatic]
You can add Debian/experimental APT sources.list ...
[ /etc/apt/sources.list.d/debian-experimental.list ]
deb http://ftp.debian.org/debian experimental main contrib non-free
deb https://deb.debian.org/debian experimental main non-free contrib
[ /etc/apt/preferences.d/99_debian-experimental.pref ]
Package: *
Pin: release o=Debian,a=experimental
Pin-Priority: 99
This gives LLVM/Clang v12 packages an APT prio of 99 - meaning no
auto-upgrade installations will be done.
You have full control by doing it manually.
Renew informations from APT repositories:
root# apt-get update
What clang-12 version is/are available?
root# apt-cache policy clang-12
Simulate an install (note: --no-install-recommends option):
root# apt-get install llvm-12 clang-12 lld-12 llvm-12-tools
--no-install-recommends -t experimental -s
option -s: simulate
Really do an installation of LLVM/Clang v12 stuff:
root# apt-get install llvm-12 clang-12 lld-12 llvm-12-tools
--no-install-recommends -t experimental -y
option -y: yes
If you like to test.
Of course you can use packages from <apt-llvm.org> repositories.
I can give you APT sources.list plus pref files if you desire.
Have more fun.
- Sedat -
On Sat, Mar 13, 2021 at 2:29 PM Borislav Petkov [off-list ref] wrote:
On Sat, Mar 13, 2021 at 01:58:56PM +0100, Sedat Dilek wrote:
quoted
You can add Debian/experimental APT sources.list ...
I could but I don't expect clang12 to behave any differently here.
Agreed in things of build-time.
There were some improvements and optimizations to LLVM/Clang but twice
as slow is really hard compared with GCC.
I was thinking more in the direction of "compatibility" of tip tree
with recent LLVM/Clang other than what is officially supported via
Kbuild-system.
Let me look if I will do a selfmade ThinLTO+PGO optimized LLVM
toolchain v12.0.0-rc3 this weekend.
- Sedat -
On Mon, Mar 15, 2021 at 06:04:41PM +0100, Sedat Dilek wrote:
Here some fresh numbers:
Lemme paste my previous reply which still holds true here:
"There's a reason I have -s for silent in the build - printing output
during the build creates a *lot* of variance. And you have excessive
printing with V=1 and KBUILD_VERBOSE=1.
Also, you need to repeat those workloads a couple of times - one is not
enough. That's why I have --repeat 5 in there.
Also, you need --pre=/root/bin/pre-build-kernel.sh where that script is:
---
#!/bin/bash
echo $0
make -s clean
echo 3 > /proc/sys/vm/drop_caches
---
so that you can avoid pagecache influence."
--
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette
On Mon, Mar 15, 2021 at 6:15 PM Borislav Petkov [off-list ref] wrote:
On Mon, Mar 15, 2021 at 06:04:41PM +0100, Sedat Dilek wrote:
quoted
Here some fresh numbers:
Lemme paste my previous reply which still holds true here:
"There's a reason I have -s for silent in the build - printing output
during the build creates a *lot* of variance. And you have excessive
printing with V=1 and KBUILD_VERBOSE=1.
I have this for diagnostic reasons.
Yes, I can drop V=1 and KBUILD_VERBOSE=1.
This is a good idea for a fast build.
Also, you need to repeat those workloads a couple of times - one is not
enough. That's why I have --repeat 5 in there.
Also, you need --pre=/root/bin/pre-build-kernel.sh where that script is:
---
#!/bin/bash
echo $0
make -s clean
echo 3 > /proc/sys/vm/drop_caches
---
so that you can avoid pagecache influence."
On Mon, Mar 15, 2021 at 06:19:34PM +0100, Sedat Dilek wrote:
With my next build I try to apply this.
Your perf tool command should look something like this:
perf stat --repeat 5 --sync --pre=/root/bin/pre-build-kernel.sh -- make -s -j9 LLVM=1 LLVM_IAS=1 bzImage
Also, needless to say, your box needs to not run anything else during
the measurement.
Thx.
--
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette
From: Peter Zijlstra <peterz@infradead.org> Date: 2021-03-15 18:11:00
On Mon, Mar 15, 2021 at 06:04:41PM +0100, Sedat Dilek wrote:
make V=1 -j4 LLVM=1 LLVM_IAS=1
So for giggles I checked, neither GCC nor LLVM seem to emit prefix NOPs
when building with -march=sandybridge, they always use MOPL.
Furthermore, the kernel explicitly sets: -falign-jumps=1
-falign-loops=1, which, when not specified, default to 16 or so.
This means that your userspace is *littered* with NOPL, even when you
build your entire distro from source with -march=sandybridge.
(arch/gentoo FTW I suppose).
(The only good new is that recent LLVM has a pass to use alternative
instruction encoding in order to grow a basic block in size in order to
minimize the amount of NOP it needs to emit at the end in order to
satisfy the jump/loop alignment.)
So if you *really* deeply care about NOP performance on your SNB, start
by teaching LLVM about prefix NOPs and rebuild your complete userspace.
At that point, you can do some trivial patches to the kernel to make it
use -march=sandybridge and prefix NOPs too.
Until that time, the vast majority of NOPs your CPU will execute will be
NOPL.
On Mon, Mar 15, 2021 at 7:10 PM Peter Zijlstra [off-list ref] wrote:
On Mon, Mar 15, 2021 at 06:04:41PM +0100, Sedat Dilek wrote:
quoted
make V=1 -j4 LLVM=1 LLVM_IAS=1
So for giggles I checked, neither GCC nor LLVM seem to emit prefix NOPs
when building with -march=sandybridge, they always use MOPL.
Furthermore, the kernel explicitly sets: -falign-jumps=1
-falign-loops=1, which, when not specified, default to 16 or so.
This means that your userspace is *littered* with NOPL, even when you
build your entire distro from source with -march=sandybridge.
(arch/gentoo FTW I suppose).
(The only good new is that recent LLVM has a pass to use alternative
instruction encoding in order to grow a basic block in size in order to
minimize the amount of NOP it needs to emit at the end in order to
satisfy the jump/loop alignment.)
So if you *really* deeply care about NOP performance on your SNB, start
by teaching LLVM about prefix NOPs and rebuild your complete userspace.
At that point, you can do some trivial patches to the kernel to make it
use -march=sandybridge and prefix NOPs too.
Until that time, the vast majority of NOPs your CPU will execute will be
NOPL.
Out of curiosity I tried in my build-environment and my testing-rules
to have comparable numbers...
..without passing "V=1" and "KBUILD_VERBOSE=1" as make-options:
NOTE: Identical linux-config plus LLVM/Clang v12.0.0-rc3.
debian-5.10.19 as host-kernel:
11655.755564957 seconds time elapsed
dileks-5.12-rc3 plus x86-nops as host-kernel:
11941.439350080 seconds time elapsed
I compared the build-times only:
Approx. 04:45 [mm:ss] in the worst case.
( Brewing time of a strong Turkish tea-bag ~5mins. )
I will keep both make-options to see what's going on in my builds.
- Sedat -
On Sat, Mar 27, 2021 at 5:08 AM Sedat Dilek [off-list ref] wrote:
debian-5.10.19 as host-kernel:
11655.755564957 seconds time elapsed
dileks-5.12-rc3 plus x86-nops as host-kernel:
11941.439350080 seconds time elapsed
That's 2.5% - a huge difference. Particularly since kernel build times
shouldn't even be that kernel-intensive.
I think there's something else going on than the nops. Same config?
There are likely many other differences between 5.10.19 and 5.12-rc3.
So can you check just plain 5.12-rc3 and then 5.12-rc3 plus x86-nops,
with otherwise identical configuration?
Linus
On Sat, Mar 27, 2021 at 9:02 PM Linus Torvalds
[off-list ref] wrote:
On Sat, Mar 27, 2021 at 5:08 AM Sedat Dilek [off-list ref] wrote:
quoted
debian-5.10.19 as host-kernel:
11655.755564957 seconds time elapsed
dileks-5.12-rc3 plus x86-nops as host-kernel:
11941.439350080 seconds time elapsed
That's 2.5% - a huge difference. Particularly since kernel build times
shouldn't even be that kernel-intensive.
I think there's something else going on than the nops. Same config?
There are likely many other differences between 5.10.19 and 5.12-rc3.
So can you check just plain 5.12-rc3 and then 5.12-rc3 plus x86-nops,
with otherwise identical configuration?
Hi Linus,
I re-checked my linux-config and custom patchset.
I had "kbuild: add CONFIG_VMLINUX_MAP expert option" in my queue and
build with CONFIG_VMLINUX_MAP=y.
This option generated here an approx. 30MiB big vmlinux.map file.
Cannot say how long this is taking in seconds but that can explain the
the time-diff.
[ The above option is helpful to analyze a recent Linux-kernel build
with CONFIG_LD_DEAD_CODE_DATA_ELIMINATION=y.
Always, I was able to build but not boot on bare metal with
CONFIG_LD_DEAD_CODE_DATA_ELIMINATION=y.
With a LLVM toolchain, of course. ]
( In the meantime Debian has a 5.20.26 kernel released - so if you
want I can re-test with Linux v5.12-rc5. )
Regards,
- Sedat -
[1] https://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild.git/commit/?h=kbuild&id=babd8cd96d333cb83c9b8abf4f01ab1f161d6ec4