Hi Paolo,

On Thu, Dec 07, 2017 at 07:04:54PM +0100, Paolo Valente wrote:

quoted

Il giorno 04 dic 2017, alle ore 11:57, Ming Lei [off-list ref] ha scritto:

On Fri, Dec 01, 2017 at 06:04:29PM +0100, Alban Browaeys wrote:

quoted

I initially reported as https://bugzilla.kernel.org/show_bug.cgi?id=198
023 .

I have now bisected this issue to commit a6a252e6491443c1c1 "blk-mq-
sched: decide how to handle flush rq via RQF_FLUSH_SEQ".

This is with an USB stick Sandisk Cruzer (USB Version:  2.10) I
regressed with.
systemctl restart systemd-udevd restores sanity.

PS: With an USB3 Lexar (USB Version:  3.00) I get more severe an issue
(not bisected) where I find no way out of reboot. My report to bugzilla
has logs when I was swapping between the these keys. The logs attached
there mixes what looks like two different behaviors.

Hi Paolo,

From both Alban's trace and my trace, looks this issue is in BFQ,
since request can't be retrieved via e->type->ops.mq.dispatch_request()
in blk_mq_do_dispatch_sched() after it is inserted into BFQ's queue.

       https://bugzilla.kernel.org/show_bug.cgi?id=198023#c4
       https://marc.info/?l=linux-block&m=151214241518562&w=2

BTW, I have tried to reproduce the issue with scsi_debug, but not succeed,
and it can't be reproduced with other schedulers(mq-deadline, none) too.

So could you take a look?

Hi Ming, all,
sorry for the delay, but we preferred to reply directly after finding
the cause of the problem.  And the cause is that gdisk makes an I/O

Not a problem, :-)

In the previous mail, I just want to share you our findings.

request that is dispatched to the drive, but apparently never
completed (as Serena, in CC discovered).  Or, at least, the execution
of completed_request in bfq is never triggered.

I can understand the case a bit, and the following info may be helpful
for you:

1) USB's queue depth is one

2) the only pending request is completed, and scsi_finish_command() is called

3) inside scsi_finish_command(), scsi_device_unbusy() is called at the
beginning, once it is done, blk_mq_get_dispatch_budget() in blk_mq_do_dispatch_sched()
returns true, then we can start to try to dispatch request

4) e->type->ops.mq.dispatch_request() is called, but the request in 2)
isn't completed yet, completed_request in bfq isn't be run yet because
it is called later from scsi_end_request()(<-scsi_io_completion()<-scsi_finish_command())

Then no request can be dispatched any more, and hang happens, but
finally completed_request should be run later.

In more detail: disk is a process for which bfq performs device idling
(for good reasons), and, for one such process, bfq does not switch to
serving another process until the last pending request of the process
is completed, after which device idling is started, to wait for the
next request of the process.  So, if such a last request is never
completed, bfq remains forever waiting for such an event, and then
refuses forever to deliver requests of other queues.

As for why bfq_completed_request is not executed for the above,

It should be run.

dispatched request, the reason is either that the bfq_finish_request
hook is not invoked at all, or that it is invoked, but the request
does not have the RQF_STARTED flag set.  Discovering which event

The flag of RQF_STARTED is set only if there is one request found by
__bfq_dispatch_request(), which can never happen in this case, since
we observed no request is found by __bfq_dispatch_request() even though
it has been inserted to BFQ queue already.

occurs is our next step.

We'll let you know.

Thanks,
Ming