Mark Rutland [off-list ref] writes:

On Thu, May 05, 2022 at 03:52:24PM +0200, Vitaly Kuznetsov wrote:

quoted

"Guilherme G. Piccoli" [off-list ref] writes:

quoted

On 05/05/2022 09:53, Mark Rutland wrote:

quoted

[...]
Looking at those, the cleanup work is all arch-specific. What exactly would we
need to do on arm64, and why does it need to happen at that point specifically?
On arm64 we don't expect as much paravirtualization as on x86, so it's not
clear to me whether we need anything at all.

quoted

Anyway, the idea here was to gather a feedback on how "receptive" arm64
community would be to allow such customization, appreciated your feedback =)

... and are you trying to do this for Hyper-V or just using that as an example?

I think we're not going to be very receptive without a more concrete example of
what you want.

What exactly do *you* need, and *why*? Is that for Hyper-V or another hypervisor?

Thanks
Mark.

Hi Mark, my plan would be doing that for Hyper-V - kind of the same
code, almost. For example, in hv_crash_handler() there is a stimer
clean-up and the vmbus unload - my understanding is that this same code
would need to run in arm64. Michael Kelley is CCed, he was discussing
with me in the panic notifiers thread and may elaborate more on the needs.

But also (not related with my specific plan), I've seen KVM quiesce code
on x86 as well [see kvm_crash_shutdown() on arch/x86] , I'm not sure if
this is necessary for arm64 or if this already executing in some
abstracted form, I didn't dig deep - probably Vitaly is aware of that,
hence I've CCed him here.

Speaking about the difference between reboot notifiers call chain and
machine_ops.crash_shutdown for KVM/x86, the main difference is that
reboot notifier is called on some CPU while the VM is fully functional,
this way we may e.g. still use IPIs (see kvm_pv_reboot_notify() doing
on_each_cpu()). When we're in a crash situation,
machine_ops.crash_shutdown is called on the CPU which crashed. We can't
count on IPIs still being functional so we do the very basic minimum so
*this* CPU can boot kdump kernel. There's no guarantee other CPUs can
still boot but normally we do kdump with 'nprocs=1'.

Sure; IIUC the IPI problem doesn't apply to arm64, though, since that doesn't
use a PV mechanism (and practically speaking will either be GICv2 or GICv3).

This isn't really about PV: when the kernel is crashing, you have no
idea what's going on on other CPUs, they may be crashing too, locked in
a tight loop, ... so sending an IPI there to do some work and expecting
it to report back is dangerous.

quoted

For Hyper-V, the situation is similar: hv_crash_handler() intitiates
VMbus unload on the crashing CPU only, there's no mechanism to do
'global' unload so other CPUs will likely not be able to connect Vmbus
devices in kdump kernel but this should not be necessary.

Given kdump is best-effort (and we can't rely on secondary CPUs even making it
into the kdump kernel), I also don't think that should be necessary.

Yes, exactly.

quoted

There's a crash_kexec_post_notifiers mechanism which can be used instead
but it's disabled by default so using machine_ops.crash_shutdown is
better.

Another option is to defer this to the kdump kernel. On arm64 at least, we know
if we're in a kdump kernel early on, and can reset some state based upon that.

Looking at x86's hyperv_cleanup(), everything relevant to arm64 can be deferred
to just before the kdump kernel detects and initializes anything relating to
hyperv. So AFAICT we could have hyperv_init() check is_kdump_kernel() prior to
the first hypercall, and do the cleanup/reset there.

In theory yes, it is possible to try sending CHANNELMSG_UNLOAD on kdump
kernel boot and not upon crash, I don't remember if this approach was
tried in the past. 

Maybe we need more data for the vmbus bits? ... if so it seems that could blow
up anyway when the first kernel was tearing down.

Not sure I understood what you mean... From what I remember, there were
issues with CHANNELMSG_UNLOAD handling on the Hyper-V host side in the
past (it was taking *minutes* for the host to reply) but this is
orthogonal to the fact that we need to do this cleanup so kdump kernel
is able to connect to Vmbus devices again.

-- 
Vitaly