Fwd: Trying to get nvme and altera-pci drivers to play nice.

Hello,
I'm a hardware guy, debugging an embedded system using a Cyclone V
SOC. It utilizes pcie 1.0 with four lanes connected to an NVMe drive
for main storage.

Bringup has come to a halt as I haven't been able to get the
altera-pci driver to play nice with the nvme driver.

The altera-pcie module loads and lspci -v shows the nvme drive and the
number of lanes connected and speed, but when I load the nvme driver
it hangs on the first try:

(pardon my printk's, I've been trying to track this down to find out
what I'm doing wrong in the hardware)

nvme: Successfully exiting nvme_pci_enable.
pci.c: nvme_reset_work: calling nvme_pci_configure_admin_queue.
nvme: core.c: nvme_wait_ready entry
nvme: core.c: nvme_wait_ready return
nvme: core.c: nvme_wait_ready entry
nvme: core.c: nvme_wait_ready return
pci.c: nvme_reset_work: calling nvme_alloc_admin_tags.
nvme: core.c: nvme_change_ctrl_state entry
pci.c: nvme_reset_work: calling nvme_init_identify.
nvme: core.c: entering nvme_init_identify
nvme: core.c: nvme_init_identify: calling nvme_identify_ctrl
nvme: core.c: Entering nvme_identify_ctrl
nvme: core.c: nvme_identify_ctrl: getting nvme_admin_identify...
nvme: core.c: nvme_identify_ctrl: kmalloc(nvme_id_ctrl)
nvme: core.c: nvme_identify_ctrl: calling nvme_submit_sync_cmd
nvme: core.c: Entering __nvme_submit_sync_cmd
nvme: core.c: __nvme_submit_sync_cmd: calling nvme_alloc_request
nvme: core.c: nvme_alloc_request entry
nvme: core.c: nvme_alloc_request calling blk_mq_alloc_request
nvme: core.c: nvme_alloc_request done. cmd = 0xef22fd08
nvme: core.c: __nvme_submit_sync_cmd: calling blk_rq_map_kern
nvme: core.c: __nvme_submit_sync_cmd: calling blk_execute_rq
blk-exec.c: Entering blk_execute_rq
blk-exec.c: blk_execute_rq: calling blk_execute_rq_nowait
blk-exec.c: blk_execute_rq_nowait: calling blk_mq_sched_insert_request
nvme: core.c: nvme_setup_cmd entry
nvme: core.c: nvme_setup_cmd: adding command: 34 to nvme_req queue
nvme: core.c: nvme_setup_cmd return
blk-exec.c: blk_execute_rq: starting wait_for_completion_io_timeout()


After a watchdog, reboot and reload of both modules, it produces this
message sometimes other times it repeats the watchdog reset:

[   64.471966] nvme nvme0: I/O 16 QID 0 timeout, disable controller
[   64.478147] nvme nvme0: Identify Controller failed (-4)
[   64.483380] nvme nvme0: Removing after probe failure status: -5

According to my printks the nvme driver is attempting a drive
identify. But the blk read never returns. I can't figure out exactly
what code is being called by the nvme driver as none of the altera-pci
routine printks I have installed are showing up before the hang. The
code traverses into blk-exec.c and never comes out.

I know it's a big ask, but can someone advise me on how to proceed to
debug this?

Brian