quoted

Btw, in that case the patch is not actually correct, as even workqueue
with a higher concurrency level MAY deadlock under enough memory
pressure.  We'll need separate workqueues to handle this case I think.

quoted

Yes?  And the
reconnect worker was never completing?  Why is that?  Here are a few

tidbits

quoted

quoted

about iWARP connections:  address resolution == neighbor discovery.  So if

the

quoted

quoted

neighbor is unreachable, it will take a few seconds for the OS to give up

and

quoted

quoted

fail the resolution.  If the neigh entry is valid and the peer becomes
unreachable during connection setup, it might take 60 seconds or so for a
connect operation to give up and fail.  So this is probably slowing the
reconnect thread down.   But shouldn't the reconnect thread notice that a

delete

quoted

quoted

is trying to happen and bail out?

I think we should aim for a state machine that can detect this, but
we'll have to see if that will end up in synchronization overkill.

Looking at the state machine I don't see why the reconnect thread would get
stuck continually rescheduling once the controller was deleted.  Changing from
RECONNECTING to DELETING will be done by nvme_change_ctrl_state().  So once
that
happens, in __nvme_rdma_del_ctrl() , the thread running reconnect logic should
stop rescheduling due to this in the failure logic of
nvme_rdma_reconnect_ctrl_work():

...
requeue:
        /* Make sure we are not resetting/deleting */
        if (ctrl->ctrl.state == NVME_CTRL_RECONNECTING) {
                dev_info(ctrl->ctrl.device,
                        "Failed reconnect attempt, requeueing...\n");
                queue_delayed_work(nvme_rdma_wq, &ctrl->reconnect_work,
                                        ctrl->reconnect_delay * HZ);
        }
...

So something isn't happening like I think it is, I guess.

I see what happens.  Assume the 10 controllers are reconnecting and failing,
thus they reschedule each time.  I then run a script to delete all 10 devices
sequentially.  Like this:

for i in $(seq 1 10); do nvme disconnect -d nvme${i}n1; done

The first device, nvme1n1 gets a disconnect/delete command and changes the
controller state from RECONNECTING to DELETING, and then schedules
nvme_rdma_del_ctrl_work(), but that is stuck behind the 9 others continually
reconnecting, failing, and rescheduling.  I'm not sure why the delete never gets
run though?  I would think if it is scheduled, then it would get executed before
the reconnect threads rescheduling?  Maybe we need some round-robin mode for our
workq?  And because the first delete is stuck, none of the subsequent delete
commands get executed.  Note: If I run each disconnect command in the
background, then they all get cleaned up ok.   Like this:

for i in $(seq 1 10); do nvme disconnect -d nvme${i}n1 & done