Subject: [BUG] Bluetooth: btusb: unbounded firmware retry loop wedges MT6639

Hi,

When the MediaTek MT6639 firmware file is missing, btusb retries the
firmware setup indefinitely, USB-resetting the device on every attempt.
The retries never succeed, and after a few hundred of them the controller
firmware wedges into a state that survives reboots and requires cutting
standby power to clear.

The missing file is a packaging matter and not the subject of this report.
The problem is that a permanent, non-recoverable error is retried without
a limit, and that the retries actively damage the device state.

Hardware and system
-------------------

   Board:      ASUS ProArt X870E-Creator WiFi rev 2
   Bluetooth:  MediaTek MT6639, USB 0489:e13a
   Kernel:     7.2.3 (Fedora 44)
   Firmware:   mediatek/mt7927/BT_RAM_CODE_MT6639_2_1_hdr.bin, absent

The Wi-Fi side of the same chip (mt7925e) is unaffected and works
throughout, since its firmware is present in linux-firmware.

Observed behaviour
------------------

On a clean cold start the controller enumerates normally.  btusb binds,
creates hci0, requests the firmware, and gets -ENOENT:

[    3.068951] usb 1-6: New USB device found, idVendor=0489, idProduct=e13a
[    8.221018] usbcore: registered new interface driver btusb
[    8.233139] Bluetooth: hci0: Failed to load firmware file (-2)
[    8.233145] Bluetooth: hci0: Failed to set up firmware (-2)
[    8.564037] usb 1-6: reset high-speed USB device number 4 using xhci_hcd
[    8.817354] Bluetooth: hci0: Failed to load firmware file (-2)
[    9.147127] usb 1-6: reset high-speed USB device number 4 using xhci_hcd
[    9.402354] Bluetooth: hci0: Failed to load firmware file (-2)
[    9.727020] usb 1-6: reset high-speed USB device number 4 using xhci_hcd

This repeats roughly every 0.58 s and does not stop.  Two boots were
captured in full; USB resets and firmware failures correlate exactly:

   boot A:   398 resets,   398 firmware failures
   boot B:  1335 resets,  1335 firmware failures

In boot B the loop ran for 780 s, until the machine was shut down.

After enough resets the controller stops responding to USB entirely.  On
every subsequent boot the port detects the device electrically but never
completes enumeration:

   usb 1-6: device descriptor read/64, error -110
   usb usb1-port6: attempt power cycle
   usb 1-6: Device not responding to setup address.
   usb 1-6: device not accepting address 7, error -71
   usb usb1-port6: unable to enumerate USB device

This state persists across reboots because the controller keeps standby
power.  Only removing power clears it.  In the captured journal the
device stayed dead for five consecutive boots after one such loop, and
the cycle repeated a second time after power was cut and restored.

Secondary effect: systemd-udev-settle.service sits on the boot critical
path, and the failed enumeration retries hold it.  Measured on the same
machine, udev-settle takes 8.6-9.7 s when the device is absent and 67.8 s
when it is wedged, adding roughly 59 s to boot.

Reproducer
----------

Note that running this to completion wedges the controller and requires
a power cut to recover.

   1. Take a machine with an MT6639 Bluetooth controller (0489:e13a, or
      13d3:3588 on other boards) and a kernel with btmtk MT6639 support.
   2. Ensure the firmware is absent from every search path.  On a stock
      distribution it already is:

        find /usr/lib/firmware /lib/firmware -name 'BT_RAM_CODE_MT6639*'

   3. Cold boot.  If the controller is already wedged, remove power first.
   4. Watch the counters climb together and never stop:

        journalctl -k -b 0 | grep -c 'reset high-speed USB device'
        journalctl -k -b 0 | grep -c 'Failed to load firmware file'

   5. Power off, cold boot again.  The device no longer enumerates.

Suggested direction
-------------------

request_firmware() returning -ENOENT is not a transient condition: the
file will still be missing on the next attempt.  Retrying it without a
bound cannot succeed, and here it drives the hardware into a state the
user cannot recover in software.

Bounding the retries, backing off, or not retrying at all when setup
failed with -ENOENT and the previous attempt failed the same way, would
reduce this to a single log line reporting the missing firmware, instead
of an unusable controller.

I did not send a patch because I am not sure at which layer the fix
belongs -- the retry appears to be driven from the btusb probe/setup
path rather than from btmtk itself -- and I would rather follow the
maintainers' preference on that.

Happy to test patches on this hardware.

Thanks,
Douglas Santos