From: Paul Menzel <hidden> Date: 2026-05-11 21:58:04
[Cc: +linuxppc-dev]
Dear Michal,
Thank you for your reply.
Am 06.05.26 um 19:30 schrieb Michal Pecio:
On Wed, 6 May 2026 18:06:20 +0200, Paul Menzel wrote:
quoted
On the IBM Power S822LC (8335-GCA POWER8), rebooting into Linux 7.1-rc2+
with kexec results in the warning below:
[ 0.000000] Linux version 7.1.0-rc2+ (x@b) (gcc (Ubuntu 11.2.0-7ubuntu2) 11.2.0, GNU ld (GNU Binutils for Ubuntu) 2.37) #3 SMP PREEMPT Wed May 6 08:50:5
[…]
[ 0.000000] Hardware name: 8335-GCA POWER8 (raw) 0x4d0200 opal:skiboot-5.4.8-5787ad3 PowerNV
[…]
[ 1.593760] NET: Registered PF_UNIX/PF_LOCAL protocol family
[ 1.593859] pci 0021:0d:00.0: enabling device (0140 -> 0142)
[ 1.627080] pci 0021:0d:00.0: xHCI HW did not halt within 32000 usec status = 0x0
[ 1.627094] pci 0021:0d:00.0: quirk_usb_early_handoff+0x0/0x300 took 32465 usecs
[ 1.627123] PCI: CLS 0 bytes, default 128
Does it work any better if kexecing other kernel versions?
No, the problem goes as far back as 5.17-rc7. (I didn’t try anything before.)
What if you increase XHCI_MAX_HALT_USEC by 10* or 100* ?
I have to test this.
Does the controller work normally after this warning?
It does not look like it. In the log attached to my report, later on there is:
[ 1.739374] xhci_hcd 0021:0d:00.0: xHCI Host Controller
[ 1.739431] xhci_hcd 0021:0d:00.0: new USB bus registered, assigned bus number 1
[ 1.794727] Freeing initrd memory: 52928K
[ 1.801984] xhci_hcd 0021:0d:00.0: Host halt failed, -110
[ 1.801988] xhci_hcd 0021:0d:00.0: can't setup: -110
[ 1.802137] xhci_hcd 0021:0d:00.0: USB bus 1 deregistered
[ 1.802154] xhci_hcd 0021:0d:00.0: init 0021:0d:00.0 fail, -110
[ 1.802250] xhci_hcd 0021:0d:00.0: probe with driver xhci_hcd failed with error -110
`lsusb` also does not list the device. But I need to check on hardware.
Kind regards,
Paul
PS: Claude Sonnet 4.6 cooked up the attached patch, which does *not* help though, but does get it to the return code 0x10, which Claude replied to with:
● The status change 0x0 → 0x10 is meaningful: 0x10 is PCD (Port Change Detect, bit 4),
HCHalted=0. The old-kernel reset (from our commit) did take effect …
From: Michal Pecio <hidden> Date: 2026-05-11 23:20:13
On Mon, 11 May 2026 23:57:33 +0200, Paul Menzel wrote:
Am 06.05.26 um 19:30 schrieb Michal Pecio:
quoted
On Wed, 6 May 2026 18:06:20 +0200, Paul Menzel wrote:
quoted
On the IBM Power S822LC (8335-GCA POWER8), rebooting into Linux 7.1-rc2+
with kexec results in the warning below:
[ 0.000000] Linux version 7.1.0-rc2+ (x@b) (gcc (Ubuntu 11.2.0-7ubuntu2) 11.2.0, GNU ld (GNU Binutils for Ubuntu) 2.37) #3 SMP PREEMPT Wed May 6 08:50:5
[…]
[ 0.000000] Hardware name: 8335-GCA POWER8 (raw) 0x4d0200 opal:skiboot-5.4.8-5787ad3 PowerNV
[…]
[ 1.593760] NET: Registered PF_UNIX/PF_LOCAL protocol family
[ 1.593859] pci 0021:0d:00.0: enabling device (0140 -> 0142)
[ 1.627080] pci 0021:0d:00.0: xHCI HW did not halt within 32000 usec status = 0x0
[ 1.627094] pci 0021:0d:00.0: quirk_usb_early_handoff+0x0/0x300 took 32465 usecs
[ 1.627123] PCI: CLS 0 bytes, default 128
quoted
Does it work any better if kexecing other kernel versions?
No, the problem goes as far back as 5.17-rc7. (I didn’t try anything
before.)
quoted
What if you increase XHCI_MAX_HALT_USEC by 10* or 100* ?
I have to test this.
I missed your dmesg attachment previously.
This may not help if another halt attempt 200ms later fails too.
Per spec (5.4.1.1), the HC is supposed to complete halt in 16ms.
quoted
Does the controller work normally after this warning?
It does not look like it. In the log attached to my report, later on
there is:
[ 1.739374] xhci_hcd 0021:0d:00.0: xHCI Host Controller
[ 1.739431] xhci_hcd 0021:0d:00.0: new USB bus registered,
assigned bus number 1
[ 1.794727] Freeing initrd memory: 52928K
[ 1.801984] xhci_hcd 0021:0d:00.0: Host halt failed, -110
[ 1.801988] xhci_hcd 0021:0d:00.0: can't setup: -110
[ 1.802137] xhci_hcd 0021:0d:00.0: USB bus 1 deregistered
[ 1.802154] xhci_hcd 0021:0d:00.0: init 0021:0d:00.0 fail, -110
[ 1.802250] xhci_hcd 0021:0d:00.0: probe with driver xhci_hcd
failed with error -110
Right, this chip seems stuck and the driver fails to reinitialize it.
PS: Claude Sonnet 4.6 cooked up the attached patch, which does *not*
help though, but does get it to the return code 0x10, which Claude
replied to with:
quoted
● The status change 0x0 → 0x10 is meaningful: 0x10 is PCD (Port Change Detect, bit 4),
HCHalted=0. The old-kernel reset (from our commit) did take effect …
Do you mean that running xhci_reset() before kexec() causes the new
kernel to see 0x10 instead of 0x0 in the status register? Is this
reproducible, not random or a one time fluke?
A little odd, one could expect reset to have the opposite effect.
Is there truly some machine firmware running during kexec() and using
the HC, as your LLM says?
I honestly don't know what to do with this. I think I would start with
looking whether xhci_shutdown() in the old kernel manages to halt it
successfully or if it also fails, and what's the USBSTS there.
It seems that you can get such information by enabling dynamic debug
echo 'module xhci_hcd +p' >/proc/dynamic_debug/control
and capcturing old kernel's log up to kexec() through a serial cable.
Regards,
Michal
From: Paul Menzel <hidden> Date: 2026-05-12 06:17:48
Dear Michal,
Thank you for your reply.
Am 12.05.26 um 01:20 schrieb Michal Pecio:
On Mon, 11 May 2026 23:57:33 +0200, Paul Menzel wrote:
quoted
Am 06.05.26 um 19:30 schrieb Michal Pecio:
quoted
On Wed, 6 May 2026 18:06:20 +0200, Paul Menzel wrote:
quoted
On the IBM Power S822LC (8335-GCA POWER8), rebooting into Linux 7.1-rc2+
with kexec results in the warning below:
[ 0.000000] Linux version 7.1.0-rc2+ (x@b) (gcc (Ubuntu 11.2.0-7ubuntu2) 11.2.0, GNU ld (GNU Binutils for Ubuntu) 2.37) #3 SMP PREEMPT Wed May 6 08:50:5
[…]
[ 0.000000] Hardware name: 8335-GCA POWER8 (raw) 0x4d0200 opal:skiboot-5.4.8-5787ad3 PowerNV
[…]
[ 1.593760] NET: Registered PF_UNIX/PF_LOCAL protocol family
[ 1.593859] pci 0021:0d:00.0: enabling device (0140 -> 0142)
[ 1.627080] pci 0021:0d:00.0: xHCI HW did not halt within 32000 usec status = 0x0
[ 1.627094] pci 0021:0d:00.0: quirk_usb_early_handoff+0x0/0x300 took 32465 usecs
[ 1.627123] PCI: CLS 0 bytes, default 128
quoted
Does it work any better if kexecing other kernel versions?
No, the problem goes as far back as 5.17-rc7. (I didn’t try anything
before.)
quoted
What if you increase XHCI_MAX_HALT_USEC by 10* or 100* ?
I have to test this.
I missed your dmesg attachment previously.
This may not help if another halt attempt 200ms later fails too.
Per spec (5.4.1.1), the HC is supposed to complete halt in 16ms.
quoted
quoted
Does the controller work normally after this warning?
It does not look like it. In the log attached to my report, later on
there is:
[ 1.739374] xhci_hcd 0021:0d:00.0: xHCI Host Controller
[ 1.739431] xhci_hcd 0021:0d:00.0: new USB bus registered,
assigned bus number 1
[ 1.794727] Freeing initrd memory: 52928K
[ 1.801984] xhci_hcd 0021:0d:00.0: Host halt failed, -110
[ 1.801988] xhci_hcd 0021:0d:00.0: can't setup: -110
[ 1.802137] xhci_hcd 0021:0d:00.0: USB bus 1 deregistered
[ 1.802154] xhci_hcd 0021:0d:00.0: init 0021:0d:00.0 fail, -110
[ 1.802250] xhci_hcd 0021:0d:00.0: probe with driver xhci_hcd failed with error -110
Right, this chip seems stuck and the driver fails to reinitialize it.
quoted
PS: Claude Sonnet 4.6 cooked up the attached patch, which does *not*
help though, but does get it to the return code 0x10, which Claude
replied to with:
quoted
● The status change 0x0 → 0x10 is meaningful: 0x10 is PCD (Port Change Detect, bit 4),
HCHalted=0. The old-kernel reset (from our commit) did take effect …
Do you mean that running xhci_reset() before kexec() causes the new
kernel to see 0x10 instead of 0x0 in the status register? Is this
reproducible, not random or a one time fluke?
It’s reproducible.
A little odd, one could expect reset to have the opposite effect.
Is there truly some machine firmware running during kexec() and using
the HC, as your LLM says?
Sorry, I should have only posted the diff. Claude Sonnet’s assumption, that OPAL is involved when kexec’ing is incorrect to my knowledge. Also, I do not know, why there is no such problem with the kexec-based bootloader Petitboot [1].
I honestly don't know what to do with this. I think I would start with
looking whether xhci_shutdown() in the old kernel manages to halt it
successfully or if it also fails, and what's the USBSTS there.
It seems that you can get such information by enabling dynamic debug
echo 'module xhci_hcd +p' >/proc/dynamic_debug/control
and capturing old kernel's log up to kexec() through a serial cable.
Unfortunately, nothing is logged over the serial console (BMC SOL) after running `sudo kexec -e` or `sudo systemctl reboot`. I just see:
[69530.180531343,5] OPAL: Switch to big-endian OS
[69538.407292205,5] OPAL: Switch to little-endian OS
Which is the OPAL firmware, so it might be involved? No idea, if it touches the xHCI controller. But strangely no xHCI messages are there – also after booting with Petitboot and initialized xHCI controller? No idea, if it points to, that during kexec or shutdown nothing is power off?
With `sudo systemctl reboot` only the line below are logged:
[ 121.811384] libvirt-guests.sh[3366]: Running guests on default URI:
[ 121.811988] libvirt-guests.sh[3376]: no running guests.
[ … (systemd service stop notifications)]
[ 136.254846] systemd-shutdown[1]: Waiting for process: watch_ldconfig
[ 218.549684] reboot: Restarting system
[69760.484679183,5] OPAL: Reboot request...
3.55778|Ignoring boot flags, incorrect version 0x0
3.59881|ISTEP 6. 3
On reboot with dynamic debug enabled for all files under `drivers/usb/`
[ 0.000000] Kernel command line: root=UUID=2c3dd738-785a-469b-843e-9f0ba8b47b0d ro rootflags=subvol=@ debug dyndbg="file drivers/usb/* +p"
the messages below are logged:
[…]
[ 1.747254] pci 0021:0d:00.0: enabling device (0140 -> 0142)
[ 1.780627] pci 0021:0d:00.0: xHCI HW did not halt within 32000 usec status = 0x10
[ 2.013904] pci 0021:0d:00.0: quirk_usb_early_handoff+0x0/0x304 took 260410 usecs
[…]
[ 2.126883] ehci_hcd: block sizes: qh 144 qtd 96 itd 192 sitd 96
[ 2.126979] ohci_hcd: block sizes: ed 112 td 96
[ 2.127558] xhci_hcd 0021:0d:00.0: xHCI Host Controller
[ 2.127629] xhci_hcd 0021:0d:00.0: new USB bus registered, assigned bus number 1
[ 2.127791] xhci_hcd 0021:0d:00.0: // Halt the HC
[…]
Please find attached a log of a “Petitboot reboot” with dynamic debug enabled and the xHCI controller being initialized.
Kind regards,
Paul
[1]: https://open-power.github.io/petitboot/overview.html
From: Michal Pecio <hidden> Date: 2026-05-12 08:22:40
On Tue, 12 May 2026 08:17:08 +0200, Paul Menzel wrote:>
quoted
I honestly don't know what to do with this. I think I would start with
looking whether xhci_shutdown() in the old kernel manages to halt it
successfully or if it also fails, and what's the USBSTS there.
It seems that you can get such information by enabling dynamic debug
echo 'module xhci_hcd +p' >/proc/dynamic_debug/control
and capturing old kernel's log up to kexec() through a serial cable.
Unfortunately, nothing is logged over the serial console (BMC SOL) after
running `sudo kexec -e` or `sudo systemctl reboot`. I just see:
[69530.180531343,5] OPAL: Switch to big-endian OS
[69538.407292205,5] OPAL: Switch to little-endian OS
Which is the OPAL firmware, so it might be involved? No idea, if it
touches the xHCI controller.
So some FW involvement is potentially possible.
BTW, another method of doing kexec is to setup a crash kernel and
then trigger panic with /proc/sysrq-trigger.
This probably won't run xhci_shutdown(). Not sure about OPAL FW.
Is the outcome any different?
But strangely no xHCI messages are there – also after booting with
Petitboot and initialized xHCI controller? No idea, if it points to,
that during kexec or shutdown nothing is power off?
With `sudo systemctl reboot` only the line below are logged:
[ 121.811384] libvirt-guests.sh[3366]: Running guests on default URI:
[ 121.811988] libvirt-guests.sh[3376]: no running guests.
[ … (systemd service stop notifications)]
[ 136.254846] systemd-shutdown[1]: Waiting for process: watch_ldconfig
[ 218.549684] reboot: Restarting system
[69760.484679183,5] OPAL: Reboot request...
3.55778|Ignoring boot flags, incorrect version 0x0
3.59881|ISTEP 6. 3
Only "reboot: Restarting system" looks like it's kernel. Maybe you need
to tweak loglevel before rebooting or kexecing? Try to get more kernel
messages showing over serial during operation, then kexec.
Regards,
Michal
From: Paul Menzel <hidden> Date: 2026-05-12 15:32:46
[Cc: +Benjamin]
Dear Michal,
Thank you for your reply.
Am 12.05.26 um 10:22 schrieb Michal Pecio:
On Tue, 12 May 2026 08:17:08 +0200, Paul Menzel wrote:>
quoted
quoted
I honestly don't know what to do with this. I think I would start with
looking whether xhci_shutdown() in the old kernel manages to halt it
successfully or if it also fails, and what's the USBSTS there.
It seems that you can get such information by enabling dynamic debug
echo 'module xhci_hcd +p' >/proc/dynamic_debug/control
and capturing old kernel's log up to kexec() through a serial cable.
Unfortunately, nothing is logged over the serial console (BMC SOL) after
running `sudo kexec -e` or `sudo systemctl reboot`. I just see:
[69530.180531343,5] OPAL: Switch to big-endian OS
[69538.407292205,5] OPAL: Switch to little-endian OS
Which is the OPAL firmware, so it might be involved? No idea, if it
touches the xHCI controller.
So some FW involvement is potentially possible.
BTW, another method of doing kexec is to setup a crash kernel and
then trigger panic with /proc/sysrq-trigger.
This probably won't run xhci_shutdown(). Not sure about OPAL FW.
Is the outcome any different?
Is the motivation to try to not get the OPAL message to rule out any involvement.
I have to check, how to set the crash kernel up.
quoted
But strangely no xHCI messages are there – also after booting with
Petitboot and initialized xHCI controller? No idea, if it points to,
that during kexec or shutdown nothing is power off?
With `sudo systemctl reboot` only the line below are logged:
[ 121.811384] libvirt-guests.sh[3366]: Running guests on default URI:
[ 121.811988] libvirt-guests.sh[3376]: no running guests.
[ … (systemd service stop notifications)]
[ 136.254846] systemd-shutdown[1]: Waiting for process: watch_ldconfig
[ 218.549684] reboot: Restarting system
[69760.484679183,5] OPAL: Reboot request...
3.55778|Ignoring boot flags, incorrect version 0x0
3.59881|ISTEP 6. 3
Only "reboot: Restarting system" looks like it's kernel. Maybe you need
to tweak loglevel before rebooting or kexecing? Try to get more kernel
messages showing over serial during operation, then kexec.
I actually did set the log level by adding `debug` to the Linux kernel command line, and with
$ echo 9 | sudo tee /proc/sysrq-trigger
9
and it was confirmed:
sysrq: Changing Loglevel
sysrq: Loglevel set to 9
Unfortunately, no more messages.
As a further data point, adding `ppc_pci_reset_phbs` to the command line also gets xhci_hcd to initialize the TI xHCI host controller:
$ lspci -nn -s 0021:0d:00.0
0021:0d:00.0 USB controller [0c03]: Texas Instruments TUSB73x0 SuperSpeed USB 3.0 xHCI Host Controller [104c:8241] (rev 02)
[ 14.050249] Issue PHB reset ...
[…]
[ 19.339822] ehci_hcd: block sizes: qh 144 qtd 96 itd 192 sitd 96
[ 19.339919] ohci_hcd: block sizes: ed 112 td 96
[ 19.340538] xhci_hcd 0021:0d:00.0: xHCI Host Controller
No log `xHCI HW did not halt within 32000 usec status = 0x0` (or 0x10 with the other patch). In `arch/powerpc/platforms/powernv/pci-ioda.c`, reading the comment in `pnv_pci_init_ioda_phb()` suggests, that PHB should be reset also in the kexec case:
/*
* If we're running in kdump kernel, the previous kernel never
* shutdown PCI devices correctly. We already got IODA table
* cleaned out. So we have to issue PHB reset to stop all PCI
* transactions from previous kernel. The ppc_pci_reset_phbs
* kernel parameter will force this reset too. Additionally,
* if the IODA reset above failed then use a bigger hammer.
* This can happen if we get a PHB fatal error in very early
* boot.
*/
if (is_kdump_kernel() || pci_reset_phbs || rc) {
pr_info(" Issue PHB reset ...\n");
pnv_eeh_phb_reset(hose, EEH_RESET_FUNDAMENTAL);
pnv_eeh_phb_reset(hose, EEH_RESET_DEACTIVATE);
}
At least, I’d assume that kdump and kexec are similar, that both do not shut down PCI devices? (Commit 361f2a2a1536 (powrpc/powernv: Reset PHB in kdump kernel) from 2024 adds (some) the code above.)
Kind regards,
Paul
From: Michal Pecio <hidden> Date: 2026-10-06 21:49:41
On Tue, 12 May 2026 17:32:09 +0200, Paul Menzel wrote:
As a further data point, adding `ppc_pci_reset_phbs` to the command line
also gets xhci_hcd to initialize the TI xHCI host controller:
$ lspci -nn -s 0021:0d:00.0
0021:0d:00.0 USB controller [0c03]: Texas Instruments TUSB73x0
SuperSpeed USB 3.0 xHCI Host Controller [104c:8241] (rev 02)
[ 14.050249] Issue PHB reset ...
[…]
[ 19.339822] ehci_hcd: block sizes: qh 144 qtd 96 itd 192 sitd 96
[ 19.339919] ohci_hcd: block sizes: ed 112 td 96
[ 19.340538] xhci_hcd 0021:0d:00.0: xHCI Host Controller
Hi Paul,
Are you still interested in this issue, or happy with your workaround?
Maybe both?
I have some updates:
We learned that the kernel does dodgy things with the FW handoff
protocol, particularly with TI chips. This may possibly matter if
FW is involved. I could supply patches to test this hypothesis.
A suspiciously similar case has been reported here:
https://lore.kernel.org/linux-usb/b43b9d21-4134-4e4c-89d8-f867207b125e@linux.intel.com/T/
There is a new workaround: PCI device reset before kexec. Weird.
No log `xHCI HW did not halt within 32000 usec status = 0x0` (or 0x10
with the other patch). In `arch/powerpc/platforms/powernv/pci-ioda.c`,
reading the comment in `pnv_pci_init_ioda_phb()` suggests, that PHB
should be reset also in the kexec case:
/*
* If we're running in kdump kernel, the previous kernel never
* shutdown PCI devices correctly. We already got IODA table
* cleaned out. So we have to issue PHB reset to stop all PCI
* transactions from previous kernel. The ppc_pci_reset_phbs
* kernel parameter will force this reset too. Additionally,
* if the IODA reset above failed then use a bigger hammer.
* This can happen if we get a PHB fatal error in very early
* boot.
*/
if (is_kdump_kernel() || pci_reset_phbs || rc) {
pr_info(" Issue PHB reset ...\n");
pnv_eeh_phb_reset(hose, EEH_RESET_FUNDAMENTAL);
pnv_eeh_phb_reset(hose, EEH_RESET_DEACTIVATE);
}
At least, I’d assume that kdump and kexec are similar, that both do not
shut down PCI devices? (Commit 361f2a2a1536 (powrpc/powernv: Reset PHB
in kdump kernel) from 2024 adds (some) the code above.)
Apparently, we try to shut down PCI devices on kexec, unless loading a
crash dump kernel on panic. So as I wrote in the new thread, we already
call xhci_halt() before kexec, the xHC should be halted, and for new
kernel xhci_halt() should be almost a no-op with guaranteed success.
It seems that we are failing to actually halt it, or something (FW?)
touches the chip during kexec, or something else is quite weird.
Regards,
Michal