I think this should be ">=" since cpus are numbered starting at zero.
In your VM with 64 CPUs, having CPU #64 in the list would be error.
quoted hunk
+ pr_emerg("ERROR_HYPERV: cpu_last=%*pbl", cpumask_pr_args(cpus));
+ }
+
/*
* Only check the mask _after_ interrupt has been disabled to avoid the
* mask changing under our feet.
2) disable the Hyper-V specific flush routines:
void hyperv_setup_mmu_ops(void)
{
+ return;
if (!(ms_hyperv.hints & HV_X64_REMOTE_TLB_FLUSH_RECOMMENDED))
return;
Otherwise, this code looks good to me and matches what I had in mind.
Note that the function native_flush_tlb_others() is used when the Hyper-V specific
flush function is disabled per patch #2 above, or when hv_cpu_to_vp_index() returns
VP_INVALID. In a quick glance through the code, it appears that native_flush_tlb_others()
will work even if there's a non-existent CPU in the cpumask that is passed as an argument.
So perhaps an immediate workaround is Patch #2 above.
Perhaps hyperv_flush_tlb_others() should be made equally tolerant of a non-existent
CPU being in the list. But if you are willing, I'm still interested in the results of an
experiment with just Patch #1. I'm curious about what the CPU list looks like when
it has a non-existent CPU. Is it complete garbage, or is there just one non-existent
CPU?
The other curiosity is that I haven't seen this Linux panic reported by other users,
and I think it would have come to our attention if it were happening with any frequency.
You see the problem fairly regularly. So I'm wondering what the difference is.
Michael
I think this should be ">=" since cpus are numbered starting at zero.
In your VM with 64 CPUs, having CPU #64 in the list would be error.
quoted
+ pr_emerg("ERROR_HYPERV: cpu_last=%*pbl", cpumask_pr_args(cpus));
+ }
+
/*
* Only check the mask _after_ interrupt has been disabled to avoid the
* mask changing under our feet.
2) disable the Hyper-V specific flush routines:
void hyperv_setup_mmu_ops(void)
{
+ return;
if (!(ms_hyperv.hints & HV_X64_REMOTE_TLB_FLUSH_RECOMMENDED))
return;
Otherwise, this code looks good to me and matches what I had in mind.
Note that the function native_flush_tlb_others() is used when the Hyper-V specific
flush function is disabled per patch #2 above, or when hv_cpu_to_vp_index() returns
VP_INVALID. In a quick glance through the code, it appears that native_flush_tlb_others()
will work even if there's a non-existent CPU in the cpumask that is passed as an argument.
So perhaps an immediate workaround is Patch #2 above.
The current code of hv_cpu_to_vp_index (where I generated the warning ) is returning VP_INVALID in this case (see previous mail) and look like it is not completely workaround the issue.
the cpu is hanging even not panic Will continue watching .
Perhaps hyperv_flush_tlb_others() should be made equally tolerant of a non-existent
CPU being in the list. But if you are willing, I'm still interested in the results of an
experiment with just Patch #1. I'm curious about what the CPU list looks like when
it has a non-existent CPU. Is it complete garbage, or is there just one non-existent
CPU?
We will do my be not next week since vacation but the week after
The other curiosity is that I haven't seen this Linux panic reported by other users,
and I think it would have come to our attention if it were happening with any frequency.
You see the problem fairly regularly. So I'm wondering what the difference is.
Michael
I think this should be ">=" since cpus are numbered starting at zero.
In your VM with 64 CPUs, having CPU #64 in the list would be error.
quoted
+ pr_emerg("ERROR_HYPERV: cpu_last=%*pbl", cpumask_pr_args(cpus));
+ }
+
/*
* Only check the mask _after_ interrupt has been disabled to avoid the
* mask changing under our feet.
2) disable the Hyper-V specific flush routines:
void hyperv_setup_mmu_ops(void)
{
+ return;
if (!(ms_hyperv.hints & HV_X64_REMOTE_TLB_FLUSH_RECOMMENDED))
return;
Otherwise, this code looks good to me and matches what I had in mind.
Note that the function native_flush_tlb_others() is used when the Hyper-V specific
flush function is disabled per patch #2 above, or when hv_cpu_to_vp_index() returns
VP_INVALID. In a quick glance through the code, it appears that native_flush_tlb_others()
will work even if there's a non-existent CPU in the cpumask that is passed as an argument.
So perhaps an immediate workaround is Patch #2 above.
The current code of hv_cpu_to_vp_index (where I generated the warning ) is returning VP_INVALID in this case (see previous mail) and look like it is not completely workaround the issue.
the cpu is hanging even not panic Will continue watching .
Perhaps hyperv_flush_tlb_others() should be made equally tolerant of a non-existent
CPU being in the list. But if you are willing, I'm still interested in the results of an
experiment with just Patch #1. I'm curious about what the CPU list looks like when
it has a non-existent CPU. Is it complete garbage, or is there just one non-existent
CPU?
We will do my be not next week since vacation but the week after
The other curiosity is that I haven't seen this Linux panic reported by other users,
and I think it would have come to our attention if it were happening with any frequency.
You see the problem fairly regularly. So I'm wondering what the difference is.
Michael
From: Wei Liu <wei.liu@kernel.org> Date: 2021-08-17 11:30:00
Please use the "reply all" button in your mail client and avoid
top-posting. It is very difficult for me to decipher this thread...
On Tue, Aug 17, 2021 at 09:16:45AM +0000, David Mozes wrote:
Hi Michael and all .
I am back from the Holiday and did your saggestiones /requstes
1. While running with patch number-2 (disable the Hyper-V specific flush routines)
As you suspected, we got panic similar to what we got with the Hyper-V specific flash routines.
Below is the trace we got:
[32097.577728] kernel BUG at kernel/sched/rt.c:1004!
[32097.577738] invalid opcode: 0000 [#1] SMP
[32097.578711] CPU: 45 PID: 51244 Comm: STAR4BLKS0_WORK Kdump: loaded Tainted: G OE 4.19.195-KM9 #1
It seems that you have out of tree module(s) loaded. Please make sure
they don't do anything unusual.
It looks like the scheduler is in an irrecoverable state. The stack
trace does not show anything related to TLB flush, so it is unclear to
me this has anything to do with the original report.
Have you tried running the same setup on baremetal?
[32097.578711] futex_wait_queue_me+0xb9/0x120
[32097.578711] futex_wait+0x139/0x250
[32097.578711] ? try_to_wake_up+0x54/0x460
[32097.578711] ? enqueue_task_rt+0x9f/0xc0
[32097.578711] ? get_futex_key+0x2ee/0x450
[32097.578711] do_futex+0x2eb/0x9f0
[32097.578711] __x64_sys_futex+0x143/0x180
[32097.578711] do_syscall_64+0x59/0x1b0
[32097.578711] ? prepare_exit_to_usermode+0x70/0x90
[32097.578711] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[32097.578711] RIP: 0033:0x7fa2ae151334
[32097.578711] Code: 66 0f 1f 44 00 00 41 52 52 4d 31 d2 ba 02 00 00 00 81 f6 80 00 00 00 64 23 34 25 48 00 00 00 39 d0 75 07 b8 ca 00 00 00 0f 05 <89> d0 87 07 85 c0 75 f1 5a 41 5a c3 83 3d f1 df 20 00 00 74 59 48
[32097.578711] RSP: 002b:00007f7b0ce9f3b0 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
[32097.578711] RAX: ffffffffffffffda RBX: 00007f7c1da5bc18 RCX: 00007fa2ae151334
[32097.578711] RDX: 0000000000000002 RSI: 0000000000000080 RDI: 00007f7c1da5bc58
[32097.578711] RBP: 00007f7b0ce9f5b0 R08: 00007f7c1da5bc58 R09: 000000000000c82c
[32097.578711] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f7b1a149cf0
[32097.578711] R13: 00007f7c1da5bc58 R14: 0000000000000001 R15: 00000000000005a1
2. as you requested and to help to the community we running patch no 1 as well :
And that is what we got:
Aug 17 05:36:22 10.230.247.7 [40544.392690] Hyper-V: ERROR_HYPERV: cpu_last=
It looks like we got an empty cpumask !
Assuming this is from the patch below, the code already handles empty
cpumask a few lines later.
You should perhaps move your change after that to right before cpus is
actually used.
Wei.
Would you please let us know what father info you need and what Is the next step for debugging this interesting issue
Thx
David
I think this should be ">=" since cpus are numbered starting at zero.
In your VM with 64 CPUs, having CPU #64 in the list would be error.
quoted
+ pr_emerg("ERROR_HYPERV: cpu_last=%*pbl", cpumask_pr_args(cpus));
+ }
+
/*
* Only check the mask _after_ interrupt has been disabled to avoid the
* mask changing under our feet.
From: David Mozes <hidden> Date: 2021-08-19 11:05:33
Hi Wei ,
Per your request I move the print cpumask to other two places after the treatment on the empty mask see below
And I got the folwing:
Aug 19 02:01:51 c-node05 kernel: [25936.562674] Hyper-V: ERROR_HYPERV2: cpu_last=
Aug 19 02:01:51 c-node05 kernel: [25936.562686] WARNING: CPU: 11 PID: 56432 at arch/x86/include/asm/mshyperv.h:301 hyperv_flush_tlb_others+0x23f/0x7b0
So we got empty cpumask on a different place on the code .
Let me know if you need further information from us.
How you sagest to handle this situation?
Thx
David
The new print cpu mask patch
From: Wei Liu <wei.liu@kernel.org>
Sent: Tuesday, August 17, 2021 2:30 PM
To: David Mozes <redacted>
Cc: David Moses <redacted>; Michael Kelley <redacted>; תומר אבוטבול <redacted>; linux-hyperv@vger.kernel.org; linux-kernel@vger.kernel.org; Wei Liu <wei.liu@kernel.org>
Subject: Re: [PATCH] x86/hyper-v: guard against cpu mask changes in hyperv_flush_tlb_others()
Please use the "reply all" button in your mail client and avoid
top-posting. It is very difficult for me to decipher this thread...
On Tue, Aug 17, 2021 at 09:16:45AM +0000, David Mozes wrote:
Hi Michael and all .
I am back from the Holiday and did your saggestiones /requstes
1. While running with patch number-2 (disable the Hyper-V specific flush routines)
As you suspected, we got panic similar to what we got with the Hyper-V specific flash routines.
Below is the trace we got:
[32097.577728] kernel BUG at kernel/sched/rt.c:1004!
[32097.577738] invalid opcode: 0000 [#1] SMP
[32097.578711] CPU: 45 PID: 51244 Comm: STAR4BLKS0_WORK Kdump: loaded Tainted: G OE 4.19.195-KM9 #1
It seems that you have out of tree module(s) loaded. Please make sure
they don't do anything unusual.
It looks like the scheduler is in an irrecoverable state. The stack
trace does not show anything related to TLB flush, so it is unclear to
me this has anything to do with the original report.
Have you tried running the same setup on baremetal?
[32097.578711] futex_wait_queue_me+0xb9/0x120
[32097.578711] futex_wait+0x139/0x250
[32097.578711] ? try_to_wake_up+0x54/0x460
[32097.578711] ? enqueue_task_rt+0x9f/0xc0
[32097.578711] ? get_futex_key+0x2ee/0x450
[32097.578711] do_futex+0x2eb/0x9f0
[32097.578711] __x64_sys_futex+0x143/0x180
[32097.578711] do_syscall_64+0x59/0x1b0
[32097.578711] ? prepare_exit_to_usermode+0x70/0x90
[32097.578711] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[32097.578711] RIP: 0033:0x7fa2ae151334
[32097.578711] Code: 66 0f 1f 44 00 00 41 52 52 4d 31 d2 ba 02 00 00 00 81 f6 80 00 00 00 64 23 34 25 48 00 00 00 39 d0 75 07 b8 ca 00 00 00 0f 05 <89> d0 87 07 85 c0 75 f1 5a 41 5a c3 83 3d f1 df 20 00 00 74 59 48
[32097.578711] RSP: 002b:00007f7b0ce9f3b0 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
[32097.578711] RAX: ffffffffffffffda RBX: 00007f7c1da5bc18 RCX: 00007fa2ae151334
[32097.578711] RDX: 0000000000000002 RSI: 0000000000000080 RDI: 00007f7c1da5bc58
[32097.578711] RBP: 00007f7b0ce9f5b0 R08: 00007f7c1da5bc58 R09: 000000000000c82c
[32097.578711] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f7b1a149cf0
[32097.578711] R13: 00007f7c1da5bc58 R14: 0000000000000001 R15: 00000000000005a1
2. as you requested and to help to the community we running patch no 1 as well :
And that is what we got:
Aug 17 05:36:22 10.230.247.7 [40544.392690] Hyper-V: ERROR_HYPERV: cpu_last=
It looks like we got an empty cpumask !
Assuming this is from the patch below, the code already handles empty
cpumask a few lines later.
You should perhaps move your change after that to right before cpus is
actually used.
Wei.
Would you please let us know what father info you need and what Is the next step for debugging this interesting issue
Thx
David
I think this should be ">=" since cpus are numbered starting at zero.
In your VM with 64 CPUs, having CPU #64 in the list would be error.
quoted
+ pr_emerg("ERROR_HYPERV: cpu_last=%*pbl", cpumask_pr_args(cpus));
+ }
+
/*
* Only check the mask _after_ interrupt has been disabled to avoid the
* mask changing under our feet.