On 2/27/20 12:45 AM, Sagi Grimberg wrote:

quoted

As per NVMe-oF spec sq flow control is actually mandatory, and we should
be implementing it to avoid the controller to return a fatal status
error, and try to play nicely with controllers using sq flow control
to implement QoS.

Hannes,

Can you please clarify why the individual transports aren't sufficient
for this QoS feature you are talking about?

If we look at the transports landscape, each transport has a credit
mechanism that can throttle bulk data transfers. In FC exchanges the
target is in control pulling data from the host with xfer_ready,
In RDMA the target decides when to issue rdma_read, and in TCP the
target decides when to issue R2T.

These are all credits that give the control to the target to
back-pressure the host. Now if the target doesn't want the host to send
more commands, it can throttle sending completions thus controlling the
pace.

Yes, that's true. However, when using this mechanism it requires the 
target to already allocate resources to hold the first part of the 
transfer, ie the command structure itself will have to be allocated at 
the target.
So with this method we'll have issues when the target goes out of memory 
due to high traffic, as not all drivers are coded carefully to avoid 
memory allocation in the I/O path.
The other problem we have on FC is that the SAN carries an internal 
timeout (RATOV, resource allocation timeout), during which a transaction 
needs to be completed. So any delay in sending RTS etc cannot exceed 
this value.
However, the real problem is that we're unable to detect a conformant 
implementation. Per default we do not disable flow control, and do not 
look for SQ Head updates. So if we run against a conformant target which 
decides to block I/O by not returning SQ Head updates the controller 
will eventually terminate the transport connection with no indication as 
to why the reset happened.

I must say that returning BLK_STS_RESOURCE for host managed SQ_HEAD is a
bit awkward in my mind, but that just one's opinion, what do others have
to say?

What would be the alternative?
Reducing the queue size for the hardware queue seems a bit excessive, 
but I'm open to have a different return code here.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke            Teamlead Storage & Networking
hare@suse.de                               +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer

_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme