From: Cédric Le Goater <clg@kaod.org> Date: 2019-07-18 14:13:29
On 18/07/2019 14:49, Michael Ellerman wrote:
Anyone else seen this?
This is running ~176 VMs on a Power9 (1 per thread), host crashes:
This is beyond the underlying limits of XIVE.
As we allocate 2K vCPUs per VM, that is 16K EQs for interrupt events. The overall
EQ count is 1M. I let you calculate what is our max number of VMs ...
[ 66.403750][ T6423] xive: OPAL failed to allocate VCPUs order 11, err -10
Hence, the OPAL XIVE driver fails which is good but ...
From: Cédric Le Goater <clg@kaod.org> Date: 2019-07-18 23:48:53
On 18/07/2019 15:14, Cédric Le Goater wrote:
On 18/07/2019 14:49, Michael Ellerman wrote:
quoted
Anyone else seen this?
This is running ~176 VMs on a Power9 (1 per thread), host crashes:
This is beyond the underlying limits of XIVE.
As we allocate 2K vCPUs per VM, that is 16K EQs for interrupt events. The overall
EQ count is 1M. I let you calculate what is our max number of VMs ...
quoted
[ 66.403750][ T6423] xive: OPAL failed to allocate VCPUs order 11, err -10
Hence, the OPAL XIVE driver fails which is good but ...
... the rollback code in case of such error must be bogus. It was never tested
clearly :/
Here is a fix. Could you give it a try on your system ?
Thanks,
C.
From b6f728ca19a9540c8bf4f5a56991c4e3dab4cf56 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= <clg@kaod.org>
Date: Thu, 18 Jul 2019 22:15:31 +0200
Subject: [PATCH] KVM: PPC: Book3S HV: XIVE: fix rollback when
kvmppc_xive_create fails
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
The XIVE device structure is now allocated in kvmppc_xive_get_device()
and kfree'd in kvmppc_core_destroy_vm(). In case of an OPAL error when
allocating the XIVE VPs, the kfree() call in kvmppc_xive_*create()
will result in a double free and corrupt the host memory.
Fixes: 5422e95103cf ("KVM: PPC: Book3S HV: XIVE: Replace the 'destroy' method by a 'release' method")
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
arch/powerpc/kvm/book3s_xive.c | 4 +---
arch/powerpc/kvm/book3s_xive_native.c | 4 ++--
2 files changed, 3 insertions(+), 5 deletions(-)
From: Michael Ellerman <mpe@ellerman.id.au> Date: 2019-07-19 07:10:57
Cédric Le Goater [off-list ref] writes:
On 18/07/2019 15:14, Cédric Le Goater wrote:
quoted
On 18/07/2019 14:49, Michael Ellerman wrote:
quoted
Anyone else seen this?
This is running ~176 VMs on a Power9 (1 per thread), host crashes:
This is beyond the underlying limits of XIVE.
As we allocate 2K vCPUs per VM, that is 16K EQs for interrupt events. The overall
EQ count is 1M. I let you calculate what is our max number of VMs ...
quoted
[ 66.403750][ T6423] xive: OPAL failed to allocate VCPUs order 11, err -10
Hence, the OPAL XIVE driver fails which is good but ...
... the rollback code in case of such error must be bogus. It was never tested
clearly :/
Here is a fix. Could you give it a try on your system ?
Yeah that fixes it, thanks.
Will apply it to my fixes branch.
cheers
quoted hunk
From b6f728ca19a9540c8bf4f5a56991c4e3dab4cf56 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= <clg@kaod.org>
Date: Thu, 18 Jul 2019 22:15:31 +0200
Subject: [PATCH] KVM: PPC: Book3S HV: XIVE: fix rollback when
kvmppc_xive_create fails
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
The XIVE device structure is now allocated in kvmppc_xive_get_device()
and kfree'd in kvmppc_core_destroy_vm(). In case of an OPAL error when
allocating the XIVE VPs, the kfree() call in kvmppc_xive_*create()
will result in a double free and corrupt the host memory.
Fixes: 5422e95103cf ("KVM: PPC: Book3S HV: XIVE: Replace the 'destroy' method by a 'release' method")
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
arch/powerpc/kvm/book3s_xive.c | 4 +---
arch/powerpc/kvm/book3s_xive_native.c | 4 ++--
2 files changed, 3 insertions(+), 5 deletions(-)
From: Michael Ellerman <mpe@ellerman.id.au> Date: 2019-07-19 11:22:49
Cédric Le Goater [off-list ref] writes:
On 18/07/2019 14:49, Michael Ellerman wrote:
quoted
Anyone else seen this?
This is running ~176 VMs on a Power9 (1 per thread), host crashes:
This is beyond the underlying limits of XIVE.
As we allocate 2K vCPUs per VM, that is 16K EQs for interrupt events. The overall
EQ count is 1M. I let you calculate what is our max number of VMs ...
We need to fix it somehow, people will expect to be able to run a VM per
thread.
cheers
From: Cédric Le Goater <clg@kaod.org> Date: 2019-07-19 12:13:54
On 19/07/2019 13:20, Michael Ellerman wrote:
Cédric Le Goater [off-list ref] writes:
quoted
On 18/07/2019 14:49, Michael Ellerman wrote:
quoted
Anyone else seen this?
This is running ~176 VMs on a Power9 (1 per thread), host crashes:
This is beyond the underlying limits of XIVE.
As we allocate 2K vCPUs per VM, that is 16K EQs for interrupt events. The overall
EQ count is 1M. I let you calculate what is our max number of VMs ...
We need to fix it somehow, people will expect to be able to run a VM per
thread.
we are limited by two spaces : VP space (1 << 19) system overall and
EQ space (1 << 20 per chip, this one we could increase). But one of
the big issue is the way we allocate the XIVE VPs in the XIVE devices.
As we have no idea of how much vCPUs we should provision for, we take
the max: 2048 ...
If we had the maxcpu of the VM (from QEMU) or at least some hints on
a rough figure, lets say a power of 2 [ 32 - 4096 ] CPUs, we would
fragment less our VP space and increase a lot our #VMs per system.
It could be a kernel global, sysfs or what ever, a new KVM PPC control
on the VM to tune maxcpu, or a KVM device creation parameter. we
could also register multiple KVM devices each having its maximum :
tiny (5), small (6), normal (8), big (11, default legacy), huge (12),
and create from QEMU the one we think fits the best.
I have to think this over.
Nevertheless, I am trying to increase by 2 or 4 the XIVE spaces for
POWER10.
C.
From: Michael Ellerman <hidden> Date: 2019-07-22 02:57:01
On Thu, 2019-07-18 at 21:51:54 UTC, =?UTF-8?Q?C=c3=a9dric_Le_Goater?= wrote:
On 18/07/2019 15:14, C��dric Le Goater wrote:
...
Here is a fix. Could you give it a try on your system ?
Thanks,
C.
quoted
From b6f728ca19a9540c8bf4f5a56991c4e3dab4cf56 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= <clg@kaod.org>
Date: Thu, 18 Jul 2019 22:15:31 +0200
Subject: [PATCH] KVM: PPC: Book3S HV: XIVE: fix rollback when
kvmppc_xive_create fails
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
The XIVE device structure is now allocated in kvmppc_xive_get_device()
and kfree'd in kvmppc_core_destroy_vm(). In case of an OPAL error when
allocating the XIVE VPs, the kfree() call in kvmppc_xive_*create()
will result in a double free and corrupt the host memory.
Fixes: 5422e95103cf ("KVM: PPC: Book3S HV: XIVE: Replace the 'destroy' method by a 'release' method")
Signed-off-by: C��dric Le Goater <clg@kaod.org>