On Wed, Mar 30, 2016 at 03:13:20PM +0100, Vladimir Murzin wrote:

On 30/03/16 14:56, Christoffer Dall wrote:

quoted

Hi Vladimir,

On Wed, Mar 30, 2016 at 12:52:27PM +0100, Vladimir Murzin wrote:

quoted

On 30/03/16 12:42, Vladimir Murzin wrote:

quoted

On 29/03/16 14:12, Vladimir Murzin wrote:

quoted

Hi Andre,

On 25/03/16 02:04, Andre Przywara wrote:

quoted

Please have a look at the series, review it and give the code some
serious testing (and possibly debugging). All feedback is appreciated.

I see that with the new vgic implementation kvmtool starts throwing:

kvm [1273]: Unable to register VGICv3 redist MMIO regions

It comes from kvm_io_bus_register_dev()

if (bus->dev_count - bus->ioeventfd_count > NR_IOBUS_DEVS - 1)
	return -ENOSPC;

with bus->dev_count being 1000

[<ffffffc00009f93c>] kvm_io_bus_register_dev+0x130/0x148
[<ffffffc0000ad274>] kvm_vgic_register_mmio_region+0xac/0xdc
[<ffffffc0000ad4d4>] vgic_register_redist_regions+0xb8/0x158
[<ffffffc0000abb90>] vgic_v3_map_resources+0x5c/0xf0
[<ffffffc0000aae04>] kvm_vgic_map_resources+0x40/0x84
[<ffffffc0000a23d0>] kvm_arch_vcpu_ioctl_run+0x3f0/0x400
[<ffffffc00009d484>] kvm_vcpu_ioctl+0x2d4/0x6ec
[<ffffffc0001c4eec>] do_vfs_ioctl+0xb4/0x760
[<ffffffc0001c561c>] SyS_ioctl+0x84/0x98
[<ffffffc000085d30>] el0_svc_naked+0x24/0x28

and one more thing I've forgotten to mention... something odd happens on
destroy path

INFO: task kvm-vcpu-0:1123 blocked for more than 120 seconds.
      Tainted: G        W       4.5.0-rc6+ #432
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kvm-vcpu-0      D ffffffc000086bcc     0  1123   1114 0x00000008
Call trace:
[<ffffffc000086bcc>] __switch_to+0x90/0xa4
[<ffffffc0006c4390>] __schedule+0x188/0x59c
[<ffffffc0006c47e0>] schedule+0x3c/0xa0
[<ffffffc0006c4bf4>] schedule_preempt_disabled+0x20/0x38
[<ffffffc0006c618c>] __mutex_lock_slowpath+0xc4/0x148
[<ffffffc0006c6254>] mutex_lock+0x44/0x5c
[<ffffffc0000aacb8>] kvm_vgic_destroy+0x20/0xb0
[<ffffffc0000abbdc>] vgic_v3_map_resources+0xa8/0xf0
[<ffffffc0000aae04>] kvm_vgic_map_resources+0x40/0x84
[<ffffffc0000a23d0>] kvm_arch_vcpu_ioctl_run+0x3f0/0x400
[<ffffffc00009d484>] kvm_vcpu_ioctl+0x2d4/0x6ec
[<ffffffc0001c4eec>] do_vfs_ioctl+0xb4/0x760
[<ffffffc0001c561c>] SyS_ioctl+0x84/0x98
[<ffffffc000085d30>] el0_svc_naked+0x24/0x28

Is this also with many many VCPUs or in general?

It's seen with many many VCPUs. Basically, I see

kvm [1273]: Unable to register VGICv3 redist MMIO regions

and then (if I'm patient enough) I see that backtrace. So the flow looks
like:

el0_svc_naked
...
vgic_v3_map_resources
   vgic_register_redist_regions
       kvm_vgic_register_mmio_region
           kvm_io_bus_register_dev	// return -ENOSPC
   kvm_vgic_destroy
       mutex_lock

ok, thanks for the clarification!

-Christoffer