From: Leonardo Bras <hidden> Date: 2021-04-07 19:56:37
According to LoPAR, ibm,query-pe-dma-window output named "IO Page Sizes"
will let the OS know all possible pagesizes that can be used for creating a
new DDW.
Currently Linux will only try using 3 of the 8 available options:
4K, 64K and 16M. According to LoPAR, Hypervisor may also offer 32M, 64M,
128M, 256M and 16G.
Enabling bigger pages would be interesting for direct mapping systems
with a lot of RAM, while using less TCE entries.
Signed-off-by: Leonardo Bras <redacted>
---
arch/powerpc/platforms/pseries/iommu.c | 49 ++++++++++++++++++++++----
1 file changed, 42 insertions(+), 7 deletions(-)
From: kernel test robot <hidden> Date: 2021-04-08 00:23:13
Hi Leonardo,
Thank you for the patch! Perhaps something to improve:
[auto build test WARNING on powerpc/next]
[also build test WARNING on v5.12-rc6 next-20210407]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Leonardo-Bras/powerpc-iommu-Enable-remaining-IOMMU-Pagesizes-present-in-LoPAR/20210408-035800
base: https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git next
config: powerpc-allyesconfig (attached as .config)
compiler: powerpc64-linux-gcc (GCC) 9.3.0
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/faa8b10e5b9652dbd56ed8e759a1cc09b95805be
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Leonardo-Bras/powerpc-iommu-Enable-remaining-IOMMU-Pagesizes-present-in-LoPAR/20210408-035800
git checkout faa8b10e5b9652dbd56ed8e759a1cc09b95805be
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross ARCH=powerpc
If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <redacted>
All warnings (new ones prefixed by >>):
In file included from include/vdso/const.h:5,
from include/linux/const.h:4,
from include/linux/bits.h:5,
from include/linux/bitops.h:6,
from include/linux/kernel.h:11,
from include/asm-generic/bug.h:20,
from arch/powerpc/include/asm/bug.h:109,
from include/linux/bug.h:5,
from include/linux/mmdebug.h:5,
from include/linux/gfp.h:5,
from include/linux/slab.h:15,
from arch/powerpc/platforms/pseries/iommu.c:15:
arch/powerpc/platforms/pseries/iommu.c: In function 'iommu_get_page_shift':
quoted
include/uapi/linux/const.h:20:19: warning: conversion from 'long long unsigned int' to 'unsigned int' changes value from '17179869184' to '0' [-Woverflow]
20 | #define __AC(X,Y) (X##Y)
| ^~~~~~
include/uapi/linux/const.h:21:18: note: in expansion of macro '__AC'
21 | #define _AC(X,Y) __AC(X,Y)
| ^~~~
include/linux/sizes.h:48:19: note: in expansion of macro '_AC'
48 | #define SZ_16G _AC(0x400000000, ULL)
| ^~~
arch/powerpc/platforms/pseries/iommu.c:1120:42: note: in expansion of macro 'SZ_16G'
1120 | { QUERY_DDW_PGSIZE_16G, __builtin_ctz(SZ_16G) },
| ^~~~~~
vim +20 include/uapi/linux/const.h
9d291e787b2b71 include/asm-x86_64/const.h Vivek Goyal 2007-05-02 6
9d291e787b2b71 include/asm-x86_64/const.h Vivek Goyal 2007-05-02 7 /* Some constant macros are used in both assembler and
9d291e787b2b71 include/asm-x86_64/const.h Vivek Goyal 2007-05-02 8 * C code. Therefore we cannot annotate them always with
6df95fd7ad9a84 include/linux/const.h Randy Dunlap 2007-05-08 9 * 'UL' and other type specifiers unilaterally. We
9d291e787b2b71 include/asm-x86_64/const.h Vivek Goyal 2007-05-02 10 * use the following macros to deal with this.
74ef649fe847fd include/linux/const.h Jeremy Fitzhardinge 2008-01-30 11 *
74ef649fe847fd include/linux/const.h Jeremy Fitzhardinge 2008-01-30 12 * Similarly, _AT() will cast an expression with a type in C, but
74ef649fe847fd include/linux/const.h Jeremy Fitzhardinge 2008-01-30 13 * leave it unchanged in asm.
9d291e787b2b71 include/asm-x86_64/const.h Vivek Goyal 2007-05-02 14 */
9d291e787b2b71 include/asm-x86_64/const.h Vivek Goyal 2007-05-02 15
9d291e787b2b71 include/asm-x86_64/const.h Vivek Goyal 2007-05-02 16 #ifdef __ASSEMBLY__
9d291e787b2b71 include/asm-x86_64/const.h Vivek Goyal 2007-05-02 17 #define _AC(X,Y) X
74ef649fe847fd include/linux/const.h Jeremy Fitzhardinge 2008-01-30 18 #define _AT(T,X) X
9d291e787b2b71 include/asm-x86_64/const.h Vivek Goyal 2007-05-02 19 #else
9d291e787b2b71 include/asm-x86_64/const.h Vivek Goyal 2007-05-02 @20 #define __AC(X,Y) (X##Y)
9d291e787b2b71 include/asm-x86_64/const.h Vivek Goyal 2007-05-02 21 #define _AC(X,Y) __AC(X,Y)
74ef649fe847fd include/linux/const.h Jeremy Fitzhardinge 2008-01-30 22 #define _AT(T,X) ((T)(X))
9d291e787b2b71 include/asm-x86_64/const.h Vivek Goyal 2007-05-02 23 #endif
9d291e787b2b71 include/asm-x86_64/const.h Vivek Goyal 2007-05-02 24
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org
From: Michael Ellerman <mpe@ellerman.id.au> Date: 2021-04-08 05:39:08
Leonardo Bras [off-list ref] writes:
According to LoPAR, ibm,query-pe-dma-window output named "IO Page Sizes"
will let the OS know all possible pagesizes that can be used for creating a
new DDW.
Currently Linux will only try using 3 of the 8 available options:
4K, 64K and 16M. According to LoPAR, Hypervisor may also offer 32M, 64M,
128M, 256M and 16G.
Do we know of any hardware & hypervisor combination that will actually
give us bigger pages?
quoted hunk
Enabling bigger pages would be interesting for direct mapping systems
with a lot of RAM, while using less TCE entries.
Signed-off-by: Leonardo Bras <redacted>
---
arch/powerpc/platforms/pseries/iommu.c | 49 ++++++++++++++++++++++----
1 file changed, 42 insertions(+), 7 deletions(-)
From: Leonardo Bras <hidden> Date: 2021-04-08 06:20:27
Hello Michael, thank you for this feedback!
Comments inline:
On Thu, 2021-04-08 at 15:37 +1000, Michael Ellerman wrote:
Leonardo Bras [off-list ref] writes:
quoted
According to LoPAR, ibm,query-pe-dma-window output named "IO Page Sizes"
will let the OS know all possible pagesizes that can be used for creating a
new DDW.
Currently Linux will only try using 3 of the 8 available options:
4K, 64K and 16M. According to LoPAR, Hypervisor may also offer 32M, 64M,
128M, 256M and 16G.
Do we know of any hardware & hypervisor combination that will actually
give us bigger pages?
quoted
Enabling bigger pages would be interesting for direct mapping systems
with a lot of RAM, while using less TCE entries.
Signed-off-by: Leonardo Bras <redacted>
---
arch/powerpc/platforms/pseries/iommu.c | 49 ++++++++++++++++++++++----
1 file changed, 42 insertions(+), 7 deletions(-)
According to LoPAR, ibm,query-pe-dma-window output named "IO Page Sizes"
will let the OS know all possible pagesizes that can be used for creating a
new DDW.
Currently Linux will only try using 3 of the 8 available options:
4K, 64K and 16M. According to LoPAR, Hypervisor may also offer 32M, 64M,
128M, 256M and 16G.
Do we know of any hardware & hypervisor combination that will actually
give us bigger pages?
On P8 16MB host pages and 16MB hardware iommu pages worked.
On P9, VM's 16MB IOMMU pages worked on top of 2MB host pages + 2MB
hardware IOMMU pages.
quoted
Enabling bigger pages would be interesting for direct mapping systems
with a lot of RAM, while using less TCE entries.
Signed-off-by: Leonardo Bras <redacted>
---
arch/powerpc/platforms/pseries/iommu.c | 49 ++++++++++++++++++++++----
1 file changed, 42 insertions(+), 7 deletions(-)
From: kernel test robot <hidden> Date: 2021-04-08 07:48:43
Hi Leonardo,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on powerpc/next]
[also build test ERROR on v5.12-rc6 next-20210407]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Leonardo-Bras/powerpc-iommu-Enable-remaining-IOMMU-Pagesizes-present-in-LoPAR/20210408-035800
base: https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git next
config: powerpc-randconfig-r016-20210407 (attached as .config)
compiler: powerpc64-linux-gcc (GCC) 9.3.0
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/faa8b10e5b9652dbd56ed8e759a1cc09b95805be
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Leonardo-Bras/powerpc-iommu-Enable-remaining-IOMMU-Pagesizes-present-in-LoPAR/20210408-035800
git checkout faa8b10e5b9652dbd56ed8e759a1cc09b95805be
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross ARCH=powerpc
If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <redacted>
All errors (new ones prefixed by >>):
In file included from include/vdso/const.h:5,
from include/linux/const.h:4,
from include/linux/bits.h:5,
from include/linux/bitops.h:6,
from include/linux/kernel.h:11,
from include/asm-generic/bug.h:20,
from arch/powerpc/include/asm/bug.h:109,
from include/linux/bug.h:5,
from include/linux/mmdebug.h:5,
from include/linux/gfp.h:5,
from include/linux/slab.h:15,
from arch/powerpc/platforms/pseries/iommu.c:15:
arch/powerpc/platforms/pseries/iommu.c: In function 'iommu_get_page_shift':
quoted
include/uapi/linux/const.h:20:19: error: conversion from 'long long unsigned int' to 'unsigned int' changes value from '17179869184' to '0' [-Werror=overflow]
20 | #define __AC(X,Y) (X##Y)
| ^~~~~~
include/uapi/linux/const.h:21:18: note: in expansion of macro '__AC'
21 | #define _AC(X,Y) __AC(X,Y)
| ^~~~
include/linux/sizes.h:48:19: note: in expansion of macro '_AC'
48 | #define SZ_16G _AC(0x400000000, ULL)
| ^~~
arch/powerpc/platforms/pseries/iommu.c:1120:42: note: in expansion of macro 'SZ_16G'
1120 | { QUERY_DDW_PGSIZE_16G, __builtin_ctz(SZ_16G) },
| ^~~~~~
cc1: all warnings being treated as errors
vim +20 include/uapi/linux/const.h
9d291e787b2b71 include/asm-x86_64/const.h Vivek Goyal 2007-05-02 6
9d291e787b2b71 include/asm-x86_64/const.h Vivek Goyal 2007-05-02 7 /* Some constant macros are used in both assembler and
9d291e787b2b71 include/asm-x86_64/const.h Vivek Goyal 2007-05-02 8 * C code. Therefore we cannot annotate them always with
6df95fd7ad9a84 include/linux/const.h Randy Dunlap 2007-05-08 9 * 'UL' and other type specifiers unilaterally. We
9d291e787b2b71 include/asm-x86_64/const.h Vivek Goyal 2007-05-02 10 * use the following macros to deal with this.
74ef649fe847fd include/linux/const.h Jeremy Fitzhardinge 2008-01-30 11 *
74ef649fe847fd include/linux/const.h Jeremy Fitzhardinge 2008-01-30 12 * Similarly, _AT() will cast an expression with a type in C, but
74ef649fe847fd include/linux/const.h Jeremy Fitzhardinge 2008-01-30 13 * leave it unchanged in asm.
9d291e787b2b71 include/asm-x86_64/const.h Vivek Goyal 2007-05-02 14 */
9d291e787b2b71 include/asm-x86_64/const.h Vivek Goyal 2007-05-02 15
9d291e787b2b71 include/asm-x86_64/const.h Vivek Goyal 2007-05-02 16 #ifdef __ASSEMBLY__
9d291e787b2b71 include/asm-x86_64/const.h Vivek Goyal 2007-05-02 17 #define _AC(X,Y) X
74ef649fe847fd include/linux/const.h Jeremy Fitzhardinge 2008-01-30 18 #define _AT(T,X) X
9d291e787b2b71 include/asm-x86_64/const.h Vivek Goyal 2007-05-02 19 #else
9d291e787b2b71 include/asm-x86_64/const.h Vivek Goyal 2007-05-02 @20 #define __AC(X,Y) (X##Y)
9d291e787b2b71 include/asm-x86_64/const.h Vivek Goyal 2007-05-02 21 #define _AC(X,Y) __AC(X,Y)
74ef649fe847fd include/linux/const.h Jeremy Fitzhardinge 2008-01-30 22 #define _AT(T,X) ((T)(X))
9d291e787b2b71 include/asm-x86_64/const.h Vivek Goyal 2007-05-02 23 #endif
9d291e787b2b71 include/asm-x86_64/const.h Vivek Goyal 2007-05-02 24
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org
From: Michael Ellerman <mpe@ellerman.id.au> Date: 2021-04-08 09:04:54
Alexey Kardashevskiy [off-list ref] writes:
On 08/04/2021 15:37, Michael Ellerman wrote:
quoted
Leonardo Bras [off-list ref] writes:
quoted
According to LoPAR, ibm,query-pe-dma-window output named "IO Page Sizes"
will let the OS know all possible pagesizes that can be used for creating a
new DDW.
Currently Linux will only try using 3 of the 8 available options:
4K, 64K and 16M. According to LoPAR, Hypervisor may also offer 32M, 64M,
128M, 256M and 16G.
Do we know of any hardware & hypervisor combination that will actually
give us bigger pages?
On P8 16MB host pages and 16MB hardware iommu pages worked.
On P9, VM's 16MB IOMMU pages worked on top of 2MB host pages + 2MB
hardware IOMMU pages.
The current code already tries 16MB though.
I'm wondering if we're going to ask for larger sizes that have never
been tested and possibly expose bugs. But it sounds like this is mainly
targeted at future platforms.
According to LoPAR, ibm,query-pe-dma-window output named "IO Page Sizes"
will let the OS know all possible pagesizes that can be used for creating a
new DDW.
Currently Linux will only try using 3 of the 8 available options:
4K, 64K and 16M. According to LoPAR, Hypervisor may also offer 32M, 64M,
128M, 256M and 16G.
Do we know of any hardware & hypervisor combination that will actually
give us bigger pages?
On P8 16MB host pages and 16MB hardware iommu pages worked.
On P9, VM's 16MB IOMMU pages worked on top of 2MB host pages + 2MB
hardware IOMMU pages.
The current code already tries 16MB though.
I'm wondering if we're going to ask for larger sizes that have never
been tested and possibly expose bugs. But it sounds like this is mainly
targeted at future platforms.
I tried for fun to pass through a PCI device to a guest with this patch as:
pbuild/qemu-killslof-aiku1904le-ppc64/qemu-system-ppc64 \
-nodefaults \
-chardev stdio,id=STDIO0,signal=off,mux=on \
-device spapr-vty,id=svty0,reg=0x71000110,chardev=STDIO0 \
-mon id=MON0,chardev=STDIO0,mode=readline \
-nographic \
-vga none \
-enable-kvm \
-m 16G \
-kernel ./vmldbg \
-initrd /home/aik/t/le.cpio \
-device vfio-pci,id=vfio0001_01_00_0,host=0001:01:00.0 \
-mem-prealloc \
-mem-path qemu_hp_1G_node0 \
-global spapr-pci-host-bridge.pgsz=0xffffff000 \
-machine cap-cfpc=broken,cap-ccf-assist=off \
-smp 1,threads=1 \
-L /home/aik/t/qemu-ppc64-bios/ \
-trace events=qemu_trace_events \
-d guest_errors,mmu \
-chardev socket,id=SOCKET0,server=on,wait=off,path=qemu.mon.1_1_0_0 \
-mon chardev=SOCKET0,mode=control
The guest created a huge window:
xhci_hcd 0000:00:00.0: ibm,create-pe-dma-window(2027) 0 8000000 20000000
22 22 returned 0 (liobn = 0x80000001 starting addr = 8000000 0)
The first "22" is page_shift in hex (16GB), the second "22" is
window_shift (so we have 1 TCE).
On the host side the window#1 was created with 1GB pages:
pci 0001:01 : [PE# fd] Setting up window#1
800000000000000..80007ffffffffff pg=40000000
The XHCI seems working. Without the patch 16MB was the maximum.
I'm not sure the #defines really gain us much vs just putting the
literal values in the array below?
Then someone says "uuuuu magic values" :) I do not mind either way. Thanks,
Yeah that's true. But #defining them doesn't make them less magic, if
you only use them in one place :)
Defining them with "QUERY_DDW" in the names kinda tells where they are
from. Can also grep QEMU using these to see how the other side handles
it. Dunno.
btw the bot complained about __builtin_ctz(SZ_16G) which should be
__builtin_ctzl(SZ_16G) so we have to ask Leonardo to repost anyway :)
--
Alexey
From: Leonardo Bras <hidden> Date: 2021-04-09 04:49:37
Em sex., 9 de abr. de 2021 01:36, Alexey Kardashevskiy [off-list ref]
escreveu:
On 08/04/2021 19:04, Michael Ellerman wrote:
quoted
Alexey Kardashevskiy [off-list ref] writes:
quoted
On 08/04/2021 15:37, Michael Ellerman wrote:
quoted
Leonardo Bras [off-list ref] writes:
quoted
According to LoPAR, ibm,query-pe-dma-window output named "IO Page
Sizes"
quoted
quoted
quoted
quoted
will let the OS know all possible pagesizes that can be used for
creating a
quoted
quoted
quoted
quoted
new DDW.
Currently Linux will only try using 3 of the 8 available options:
4K, 64K and 16M. According to LoPAR, Hypervisor may also offer 32M,
64M,
quoted
quoted
quoted
quoted
128M, 256M and 16G.
Do we know of any hardware & hypervisor combination that will actually
give us bigger pages?
On P8 16MB host pages and 16MB hardware iommu pages worked.
On P9, VM's 16MB IOMMU pages worked on top of 2MB host pages + 2MB
hardware IOMMU pages.
The current code already tries 16MB though.
I'm wondering if we're going to ask for larger sizes that have never
been tested and possibly expose bugs. But it sounds like this is mainly
targeted at future platforms.
I tried for fun to pass through a PCI device to a guest with this patch as:
pbuild/qemu-killslof-aiku1904le-ppc64/qemu-system-ppc64 \
-nodefaults \
-chardev stdio,id=STDIO0,signal=off,mux=on \
-device spapr-vty,id=svty0,reg=0x71000110,chardev=STDIO0 \
-mon id=MON0,chardev=STDIO0,mode=readline \
-nographic \
-vga none \
-enable-kvm \
-m 16G \
-kernel ./vmldbg \
-initrd /home/aik/t/le.cpio \
-device vfio-pci,id=vfio0001_01_00_0,host=0001:01:00.0 \
-mem-prealloc \
-mem-path qemu_hp_1G_node0 \
-global spapr-pci-host-bridge.pgsz=0xffffff000 \
-machine cap-cfpc=broken,cap-ccf-assist=off \
-smp 1,threads=1 \
-L /home/aik/t/qemu-ppc64-bios/ \
-trace events=qemu_trace_events \
-d guest_errors,mmu \
-chardev socket,id=SOCKET0,server=on,wait=off,path=qemu.mon.1_1_0_0 \
-mon chardev=SOCKET0,mode=control
The guest created a huge window:
xhci_hcd 0000:00:00.0: ibm,create-pe-dma-window(2027) 0 8000000 20000000
22 22 returned 0 (liobn = 0x80000001 starting addr = 8000000 0)
The first "22" is page_shift in hex (16GB), the second "22" is
window_shift (so we have 1 TCE).
On the host side the window#1 was created with 1GB pages:
pci 0001:01 : [PE# fd] Setting up window#1
800000000000000..80007ffffffffff pg=40000000
The XHCI seems working. Without the patch 16MB was the maximum.
I'm not sure the #defines really gain us much vs just putting the
literal values in the array below?
Then someone says "uuuuu magic values" :) I do not mind either way.
Thanks,
quoted
Yeah that's true. But #defining them doesn't make them less magic, if
you only use them in one place :)
Defining them with "QUERY_DDW" in the names kinda tells where they are
from. Can also grep QEMU using these to see how the other side handles
it. Dunno.
btw the bot complained about __builtin_ctz(SZ_16G) which should be
__builtin_ctzl(SZ_16G) so we have to ask Leonardo to repost anyway :)
Thanks for testing!
http://patchwork.ozlabs.org/project/linuxppc-dev/patch/20210408201915.174217-1-leobras.c@gmail.com/
I sent a v3 a few hours ago, fixing this by using __builtin_ctzll() instead
of __builtin_ctz() in all sizes, and it worked like a charm.
I also reverted to the previous approach of not having QUERY_DDW defines
for masks, as Michael suggested.
I can revert back to v2 approach if you guys decide it's better.
Best regards,
Leonardo Bras
I'm not sure the #defines really gain us much vs just putting the
literal values in the array below?
Then someone says "uuuuu magic values" :) I do not mind either way.
Thanks,
Yeah that's true. But #defining them doesn't make them less magic, if
you only use them in one place :)
Defining them with "QUERY_DDW" in the names kinda tells where they are
from. Can also grep QEMU using these to see how the other side handles
it. Dunno.
And *not* defining anything reduces the mental load a lot. You can add
a comment at the single spot you use them, explaining what this is, in a
much better way!
Comments are *good*.
Segher
I'm not sure the #defines really gain us much vs just putting the
literal values in the array below?
Then someone says "uuuuu magic values" :) I do not mind either way.
Thanks,
Yeah that's true. But #defining them doesn't make them less magic, if
you only use them in one place :)
Defining them with "QUERY_DDW" in the names kinda tells where they are
from. Can also grep QEMU using these to see how the other side handles
it. Dunno.
And *not* defining anything reduces the mental load a lot. You can add
a comment at the single spot you use them, explaining what this is, in a
much better way!
Comments are *good*.
Segher