Hi,

On 2023/08/24 19:29, Laurence Oberman wrote:

On Mon, 2023-06-12 at 11:40 -0700, Bart Van Assche wrote:

quoted

On 6/9/23 00:29, Jaco Kroon wrote:

quoted

I'm attaching dmesg -T and ps axf.  dmesg in particular may provide
clues as it provides a number of stack traces indicating stalling
at
IO time.

Once this has triggered, even commands such as "lvs" goes into
uninterruptable wait, I unfortunately didn't test "dmsetup ls" now
and triggered a reboot already (system needs to be up).

To me the call traces suggest that an I/O request got stuck.
Unfortunately call traces are not sufficient to identify the root
cause
in case I/O gets stuck. Has debugfs been mounted? If so, how about
dumping the contents of /sys/kernel/debug/block/ into a tar file
after
the lockup has been reproduced and sharing that information?

tar -czf- -C /sys/kernel/debug/block . >block.tgz

Thanks,

Bart.

One I am aware of is this
commit 106397376c0369fcc01c58dd189ff925a2724a57
Author: David Jeffery [off-list ref]

Can we try get a vmcore (assuming its not a secure site)

Certainly.  Obviously on any host handling any kind of sensitive data 
there is a likelihood that sensitive data may be present in the vmcore, 
as such I more than happy to create a vmcore, I'm assuming this will 
create a kernel version of a core dump ... with 256GB of RAM (most of 
which goes towards disk caches) I'm further assuming this file can be 
potentially large.  Where will this get stored should the capture be 
made?  (I need to ensure that the filesystem has sufficient storage 
available)

Add these to /etc/sysctl.conf

kernel.panic_on_io_nmi = 1
kernel.panic_on_unrecovered_nmi = 1
kernel.unknown_nmi_panic = 1

Run sysctl -p
Ensure kdump is running and can capture a vmcore

Done.  Had to enable a few extra kernel options to get all the other 
requirements, so scheduled a reboot to activate the new kernel. This 
will happen on Saturday morning very early.

When it locks up again
send an NMI via the SuperMicro Web Managemnt interface

Possible to send from sysrq at the keyboard?  Otherwise I'll just need 
to set up the RMI, will just be easier to do this from the keyboard if 
possible, it's not always if it's left too late.

Share the vmcore, or we can have you capture some specifics from it to
triage.

I'd prefer you let me know what you need ... security concerns and all 
... frankly, I highly doubt there is any data that is really so 
sensitive that it can be classified as "top secret" but we do have NDAs 
in place prohibiting me from sharing anything that may potentially 
contain customer related data ...

Kind regards,
Jaco