The size of 'struct vhost_scsi' is order-10 (~2.3MB). It may take long time
delay by kzalloc() to compact memory pages by retrying multiple times when
there is a lack of high-order pages. As a result, there is latency to
create a VM (with vhost-scsi) or to hotadd vhost-scsi-based storage.
The prior commit 595cb754983d ("vhost/scsi: use vmalloc for order-10
allocation") prefers to fallback only when really needed, while this patch
allocates with kvzalloc() with __GFP_NORETRY implicitly set to avoid
retrying memory pages compact for multiple times.
The __GFP_NORETRY is implicitly set if the size to allocate is more than
PAGE_SZIE and when __GFP_RETRY_MAYFAIL is not explicitly set.
Cc: Aruna Ramakrishna <redacted>
Cc: Joe Jin <redacted>
Signed-off-by: Dongli Zhang <redacted>
---
Changed since v1:
- To combine kzalloc() and vzalloc() as kvzalloc()
(suggested by Jason Wang)
drivers/vhost/scsi.c | 9 +++------
1 file changed, 3 insertions(+), 6 deletions(-)
According to my "git send-email" history, I have CCed jasowang@redhat.com. Not
sure why Jason is not on the list.
CCed Jason. Thank you very much!
Dongli Zhang
On 1/23/21 12:08 AM, Dongli Zhang wrote:
quoted hunk
The size of 'struct vhost_scsi' is order-10 (~2.3MB). It may take long time
delay by kzalloc() to compact memory pages by retrying multiple times when
there is a lack of high-order pages. As a result, there is latency to
create a VM (with vhost-scsi) or to hotadd vhost-scsi-based storage.
The prior commit 595cb754983d ("vhost/scsi: use vmalloc for order-10
allocation") prefers to fallback only when really needed, while this patch
allocates with kvzalloc() with __GFP_NORETRY implicitly set to avoid
retrying memory pages compact for multiple times.
The __GFP_NORETRY is implicitly set if the size to allocate is more than
PAGE_SZIE and when __GFP_RETRY_MAYFAIL is not explicitly set.
Cc: Aruna Ramakrishna <redacted>
Cc: Joe Jin <redacted>
Signed-off-by: Dongli Zhang <redacted>
---
Changed since v1:
- To combine kzalloc() and vzalloc() as kvzalloc()
(suggested by Jason Wang)
drivers/vhost/scsi.c | 9 +++------
1 file changed, 3 insertions(+), 6 deletions(-)
From: Jason Wang <hidden> Date: 2021-01-25 03:14:52
On 2021/1/23 下午4:08, Dongli Zhang wrote:
quoted hunk
The size of 'struct vhost_scsi' is order-10 (~2.3MB). It may take long time
delay by kzalloc() to compact memory pages by retrying multiple times when
there is a lack of high-order pages. As a result, there is latency to
create a VM (with vhost-scsi) or to hotadd vhost-scsi-based storage.
The prior commit 595cb754983d ("vhost/scsi: use vmalloc for order-10
allocation") prefers to fallback only when really needed, while this patch
allocates with kvzalloc() with __GFP_NORETRY implicitly set to avoid
retrying memory pages compact for multiple times.
The __GFP_NORETRY is implicitly set if the size to allocate is more than
PAGE_SZIE and when __GFP_RETRY_MAYFAIL is not explicitly set.
Cc: Aruna Ramakrishna <redacted>
Cc: Joe Jin <redacted>
Signed-off-by: Dongli Zhang <redacted>
---
Changed since v1:
- To combine kzalloc() and vzalloc() as kvzalloc()
(suggested by Jason Wang)
drivers/vhost/scsi.c | 9 +++------
1 file changed, 3 insertions(+), 6 deletions(-)
Can anyone help to review this patch and give a review-by for it please?
Thanks,
Joe
On 1/24/21 7:12 PM, Jason Wang wrote:
On 2021/1/23 下午4:08, Dongli Zhang wrote:
quoted
The size of 'struct vhost_scsi' is order-10 (~2.3MB). It may take long time
delay by kzalloc() to compact memory pages by retrying multiple times when
there is a lack of high-order pages. As a result, there is latency to
create a VM (with vhost-scsi) or to hotadd vhost-scsi-based storage.
The prior commit 595cb754983d ("vhost/scsi: use vmalloc for order-10
allocation") prefers to fallback only when really needed, while this patch
allocates with kvzalloc() with __GFP_NORETRY implicitly set to avoid
retrying memory pages compact for multiple times.
The __GFP_NORETRY is implicitly set if the size to allocate is more than
PAGE_SZIE and when __GFP_RETRY_MAYFAIL is not explicitly set.
Cc: Aruna Ramakrishna <redacted>
Cc: Joe Jin <redacted>
Signed-off-by: Dongli Zhang <redacted>
---
Changed since v1:
- To combine kzalloc() and vzalloc() as kvzalloc()
(suggested by Jason Wang)
drivers/vhost/scsi.c | 9 +++------
1 file changed, 3 insertions(+), 6 deletions(-)
The size of 'struct vhost_scsi' is order-10 (~2.3MB). It may take long time
delay by kzalloc() to compact memory pages by retrying multiple times when
there is a lack of high-order pages. As a result, there is latency to
create a VM (with vhost-scsi) or to hotadd vhost-scsi-based storage.
The prior commit 595cb754983d ("vhost/scsi: use vmalloc for order-10
allocation") prefers to fallback only when really needed, while this patch
allocates with kvzalloc() with __GFP_NORETRY implicitly set to avoid
retrying memory pages compact for multiple times.
The __GFP_NORETRY is implicitly set if the size to allocate is more than
PAGE_SZIE and when __GFP_RETRY_MAYFAIL is not explicitly set.
Cc: Aruna Ramakrishna <redacted>
Cc: Joe Jin <redacted>
Signed-off-by: Dongli Zhang <redacted>
---
Changed since v1:
- To combine kzalloc() and vzalloc() as kvzalloc()
(suggested by Jason Wang)
drivers/vhost/scsi.c | 9 +++------
1 file changed, 3 insertions(+), 6 deletions(-)
Is it possible that the issue is not due to this change?
This change is just to call different API to allocate memory, which is
equivalent to kzalloc()+vzalloc().
Before the change:
try kzalloc(sizeof(*vs), GFP_KERNEL | __GFP_NOWARN | __GFP_RETRY_MAYFAIL);
... and then below if the former is failed.
vzalloc(sizeof(*vs));
After the change:
try kmalloc_node(size, FP_KERNEL|GFP_ZERO|__GFP_NOWARN|__GFP_NORETRY, node);
... and then below if the former is failed
__vmalloc_node(size, 1, GFP_KERNEL|GFP_ZERO, node, __builtin_return_address(0));
The below is the first WARNING in uploaded dmesg. I assume it was called before
to open /dev/vhost-scsi.
Will this test try to open /dev/vhost-scsi?
[ 5.095515] =============================
[ 5.095515] WARNING: suspicious RCU usage
[ 5.095515] 5.11.0-rc4-00008-g79991caf5202 #1 Not tainted
[ 5.095534] -----------------------------
[ 5.096041] security/smack/smack_lsm.c:351 RCU-list traversed in non-reader
section!!
[ 5.096982]
[ 5.096982] other info that might help us debug this:
[ 5.096982]
[ 5.097953]
[ 5.097953] rcu_scheduler_active = 1, debug_locks = 1
[ 5.098739] no locks held by kthreadd/2.
[ 5.099237]
[ 5.099237] stack backtrace:
[ 5.099537] CPU: 0 PID: 2 Comm: kthreadd Not tainted
5.11.0-rc4-00008-g79991caf5202 #1
[ 5.100470] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
1.12.0-1 04/01/2014
[ 5.101442] Call Trace:
[ 5.101807] dump_stack+0x15f/0x1bf
[ 5.102298] smack_cred_prepare+0x400/0x420
[ 5.102840] ? security_prepare_creds+0xd4/0x120
[ 5.103441] security_prepare_creds+0x84/0x120
[ 5.103515] prepare_creds+0x3f1/0x580
[ 5.103515] copy_creds+0x65/0x480
[ 5.103515] copy_process+0x7b4/0x3600
[ 5.103515] ? check_prev_add+0xa40/0xa40
[ 5.103515] ? lockdep_enabled+0xd/0x60
[ 5.103515] ? lock_is_held_type+0x1a/0x100
[ 5.103515] ? __cleanup_sighand+0xc0/0xc0
[ 5.103515] ? lockdep_unlock+0x39/0x160
[ 5.103515] kernel_clone+0x165/0xd20
[ 5.103515] ? copy_init_mm+0x20/0x20
[ 5.103515] ? pvclock_clocksource_read+0xd9/0x1a0
[ 5.103515] ? sched_clock_local+0x99/0xc0
[ 5.103515] ? kthread_insert_work_sanity_check+0xc0/0xc0
[ 5.103515] kernel_thread+0xba/0x100
[ 5.103515] ? __ia32_sys_clone3+0x40/0x40
[ 5.103515] ? kthread_insert_work_sanity_check+0xc0/0xc0
[ 5.103515] ? do_raw_spin_unlock+0xa9/0x160
[ 5.103515] kthreadd+0x68f/0x7a0
[ 5.103515] ? kthread_create_on_cpu+0x160/0x160
[ 5.103515] ? lockdep_hardirqs_on+0x77/0x100
[ 5.103515] ? _raw_spin_unlock_irq+0x24/0x60
[ 5.103515] ? kthread_create_on_cpu+0x160/0x160
[ 5.103515] ret_from_fork+0x22/0x30
Thank you very much!
Dongli Zhang
On 2/6/21 7:03 PM, kernel test robot wrote:
From: "Michael S. Tsirkin" <mst@redhat.com> Date: 2021-02-08 10:09:44
On Sun, Feb 07, 2021 at 11:03:30AM +0800, kernel test robot wrote:
Greeting,
FYI, we noticed the following commit (built with gcc-9):
commit: 79991caf5202c7989928be534727805f8f68bb8d ("vdpa_sim_net: Add support for user supported devices")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git Dongli-Zhang/vhost-scsi-alloc-vhost_scsi-with-kvzalloc-to-avoid-delay/20210129-191605
in testcase: trinity
version: trinity-static-x86_64-x86_64-f93256fb_2019-08-28
with following parameters:
runtime: 300s
test-description: Trinity is a linux system call fuzz tester.
test-url: http://codemonkey.org.uk/projects/trinity/
on test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 8G
caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace):
Is it possible that the issue is not due to this change?
Looks this issue does not related your change, from dmesg output, when issue occurred, virtio was not loaded:
[ 502.508450] ------------[ cut here ]------------
[ 502.511859] WARNING: CPU: 0 PID: 1 at drivers/gpu/drm/vkms/vkms_crtc.c:21 vkms_vblank_simulate+0x22a/0x240
[ 502.524018] Modules linked in:
[ 502.539642] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.11.0-rc4-00008-g79991caf5202 #1
This change is just to call different API to allocate memory, which is
equivalent to kzalloc()+vzalloc().
Before the change:
try kzalloc(sizeof(*vs), GFP_KERNEL | __GFP_NOWARN | __GFP_RETRY_MAYFAIL);
... and then below if the former is failed.
vzalloc(sizeof(*vs));
After the change:
try kmalloc_node(size, FP_KERNEL|GFP_ZERO|__GFP_NOWARN|__GFP_NORETRY, node);
... and then below if the former is failed
__vmalloc_node(size, 1, GFP_KERNEL|GFP_ZERO, node, __builtin_return_address(0));
The below is the first WARNING in uploaded dmesg. I assume it was called before
to open /dev/vhost-scsi.
Will this test try to open /dev/vhost-scsi?
[ 5.095515] =============================
[ 5.095515] WARNING: suspicious RCU usage
[ 5.095515] 5.11.0-rc4-00008-g79991caf5202 #1 Not tainted
[ 5.095534] -----------------------------
[ 5.096041] security/smack/smack_lsm.c:351 RCU-list traversed in non-reader
section!!
[ 5.096982]
[ 5.096982] other info that might help us debug this:
[ 5.096982]
[ 5.097953]
[ 5.097953] rcu_scheduler_active = 1, debug_locks = 1
[ 5.098739] no locks held by kthreadd/2.
[ 5.099237]
[ 5.099237] stack backtrace:
[ 5.099537] CPU: 0 PID: 2 Comm: kthreadd Not tainted
5.11.0-rc4-00008-g79991caf5202 #1
[ 5.100470] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
1.12.0-1 04/01/2014
[ 5.101442] Call Trace:
[ 5.101807] dump_stack+0x15f/0x1bf
[ 5.102298] smack_cred_prepare+0x400/0x420
[ 5.102840] ? security_prepare_creds+0xd4/0x120
[ 5.103441] security_prepare_creds+0x84/0x120
[ 5.103515] prepare_creds+0x3f1/0x580
[ 5.103515] copy_creds+0x65/0x480
[ 5.103515] copy_process+0x7b4/0x3600
[ 5.103515] ? check_prev_add+0xa40/0xa40
[ 5.103515] ? lockdep_enabled+0xd/0x60
[ 5.103515] ? lock_is_held_type+0x1a/0x100
[ 5.103515] ? __cleanup_sighand+0xc0/0xc0
[ 5.103515] ? lockdep_unlock+0x39/0x160
[ 5.103515] kernel_clone+0x165/0xd20
[ 5.103515] ? copy_init_mm+0x20/0x20
[ 5.103515] ? pvclock_clocksource_read+0xd9/0x1a0
[ 5.103515] ? sched_clock_local+0x99/0xc0
[ 5.103515] ? kthread_insert_work_sanity_check+0xc0/0xc0
[ 5.103515] kernel_thread+0xba/0x100
[ 5.103515] ? __ia32_sys_clone3+0x40/0x40
[ 5.103515] ? kthread_insert_work_sanity_check+0xc0/0xc0
[ 5.103515] ? do_raw_spin_unlock+0xa9/0x160
[ 5.103515] kthreadd+0x68f/0x7a0
[ 5.103515] ? kthread_create_on_cpu+0x160/0x160
[ 5.103515] ? lockdep_hardirqs_on+0x77/0x100
[ 5.103515] ? _raw_spin_unlock_irq+0x24/0x60
[ 5.103515] ? kthread_create_on_cpu+0x160/0x160
[ 5.103515] ret_from_fork+0x22/0x30
Thank you very much!
Dongli Zhang
On 2/6/21 7:03 PM, kernel test robot wrote:
From: Oliver Sang <hidden> Date: 2021-03-18 06:33:56
Hi Joe and Dongli,
On Mon, Feb 08, 2021 at 08:10:21AM -0800, Joe Jin wrote:
On 2/7/21 12:15 PM, Dongli Zhang wrote:
quoted
Is it possible that the issue is not due to this change?
Looks this issue does not related your change, from dmesg output, when issue occurred, virtio was not loaded:
[ 502.508450] ------------[ cut here ]------------
[ 502.511859] WARNING: CPU: 0 PID: 1 at drivers/gpu/drm/vkms/vkms_crtc.c:21 vkms_vblank_simulate+0x22a/0x240
[ 502.524018] Modules linked in:
[ 502.539642] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.11.0-rc4-00008-g79991caf5202 #1
thanks for explanation and sorry for false positive.
we will investigate further to avoid such kind of false positive in the future.
quoted
This change is just to call different API to allocate memory, which is
equivalent to kzalloc()+vzalloc().
Before the change:
try kzalloc(sizeof(*vs), GFP_KERNEL | __GFP_NOWARN | __GFP_RETRY_MAYFAIL);
... and then below if the former is failed.
vzalloc(sizeof(*vs));
After the change:
try kmalloc_node(size, FP_KERNEL|GFP_ZERO|__GFP_NOWARN|__GFP_NORETRY, node);
... and then below if the former is failed
__vmalloc_node(size, 1, GFP_KERNEL|GFP_ZERO, node, __builtin_return_address(0));
The below is the first WARNING in uploaded dmesg. I assume it was called before
to open /dev/vhost-scsi.
Will this test try to open /dev/vhost-scsi?
[ 5.095515] =============================
[ 5.095515] WARNING: suspicious RCU usage
[ 5.095515] 5.11.0-rc4-00008-g79991caf5202 #1 Not tainted
[ 5.095534] -----------------------------
[ 5.096041] security/smack/smack_lsm.c:351 RCU-list traversed in non-reader
section!!
[ 5.096982]
[ 5.096982] other info that might help us debug this:
[ 5.096982]
[ 5.097953]
[ 5.097953] rcu_scheduler_active = 1, debug_locks = 1
[ 5.098739] no locks held by kthreadd/2.
[ 5.099237]
[ 5.099237] stack backtrace:
[ 5.099537] CPU: 0 PID: 2 Comm: kthreadd Not tainted
5.11.0-rc4-00008-g79991caf5202 #1
[ 5.100470] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
1.12.0-1 04/01/2014
[ 5.101442] Call Trace:
[ 5.101807] dump_stack+0x15f/0x1bf
[ 5.102298] smack_cred_prepare+0x400/0x420
[ 5.102840] ? security_prepare_creds+0xd4/0x120
[ 5.103441] security_prepare_creds+0x84/0x120
[ 5.103515] prepare_creds+0x3f1/0x580
[ 5.103515] copy_creds+0x65/0x480
[ 5.103515] copy_process+0x7b4/0x3600
[ 5.103515] ? check_prev_add+0xa40/0xa40
[ 5.103515] ? lockdep_enabled+0xd/0x60
[ 5.103515] ? lock_is_held_type+0x1a/0x100
[ 5.103515] ? __cleanup_sighand+0xc0/0xc0
[ 5.103515] ? lockdep_unlock+0x39/0x160
[ 5.103515] kernel_clone+0x165/0xd20
[ 5.103515] ? copy_init_mm+0x20/0x20
[ 5.103515] ? pvclock_clocksource_read+0xd9/0x1a0
[ 5.103515] ? sched_clock_local+0x99/0xc0
[ 5.103515] ? kthread_insert_work_sanity_check+0xc0/0xc0
[ 5.103515] kernel_thread+0xba/0x100
[ 5.103515] ? __ia32_sys_clone3+0x40/0x40
[ 5.103515] ? kthread_insert_work_sanity_check+0xc0/0xc0
[ 5.103515] ? do_raw_spin_unlock+0xa9/0x160
[ 5.103515] kthreadd+0x68f/0x7a0
[ 5.103515] ? kthread_create_on_cpu+0x160/0x160
[ 5.103515] ? lockdep_hardirqs_on+0x77/0x100
[ 5.103515] ? _raw_spin_unlock_irq+0x24/0x60
[ 5.103515] ? kthread_create_on_cpu+0x160/0x160
[ 5.103515] ret_from_fork+0x22/0x30
Thank you very much!
Dongli Zhang
On 2/6/21 7:03 PM, kernel test robot wrote: