[PATCH net 1/1] qede: Fix barrier usage after tx doorbell write.

Since commit c5ad119fb6c09b0297446be05bd66602fa564758
("net: sched: pfifo_fast use skb_array") driver is exposed
to an issue where it is hitting NULL skbs while handling TX
completions. Driver uses mmiowb() to flush the writes to the
doorbell bar which is a write-combined bar, however on x86
mmiowb() does not flush the write combined buffer.

This patch fixes this problem by replacing mmiowb() with wmb()
after the write combined doorbell write so that writes are
flushed and synchronized from more than one processor.

Signed-off-by: Ariel Elior <redacted>
Signed-off-by: Manish Chopra <redacted>
---
 drivers/net/ethernet/qlogic/qede/qede_fp.c |   10 ++++------
 1 files changed, 4 insertions(+), 6 deletions(-)

diff --git a/drivers/net/ethernet/qlogic/qede/qede_fp.c b/drivers/net/ethernet/qlogic/qede/qede_fp.c
index dafc079..2e921ca 100644
--- a/drivers/net/ethernet/qlogic/qede/qede_fp.c
+++ b/drivers/net/ethernet/qlogic/qede/qede_fp.c

@@ -320,13 +320,11 @@ static inline void qede_update_tx_producer(struct qede_tx_queue *txq)
 	barrier();
 	writel(txq->tx_db.raw, txq->doorbell_addr);
 
-	/* mmiowb is needed to synchronize doorbell writes from more than one
-	 * processor. It guarantees that the write arrives to the device before
-	 * the queue lock is released and another start_xmit is called (possibly
-	 * on another CPU). Without this barrier, the next doorbell can bypass
-	 * this doorbell. This is applicable to IA64/Altix systems.
+	/* Fence required to flush the write combined buffer, since another
+	 * CPU may write to the same doorbell address and data may be lost
+	 * due to relaxed order nature of write combined bar.
 	 */
-	mmiowb();
+	wmb();
 }
 
 static int qede_xdp_xmit(struct qede_dev *edev, struct qede_fastpath *fp,

-- 
1.7.1

Hi Dave,
This patch appears as "superseded" in patchwork. I am not really sure why
that is - I noticed some other barrier work is going on, but none of it will
solve this issue. This patch solves an important bug in the driver - please
consider applying it.
Thanks,
Ariel

-----Original Message-----
From: Elior, Ariel
Sent: Wednesday, March 21, 2018 7:10 PM
To: davem@davemloft.net
Cc: netdev@vger.kernel.org; Kalderon, Michal <redacted>;
Chopra, Manish [off-list ref]
Subject: RE: [PATCH net 1/1] qede: Fix barrier usage after tx doorbell write.

quoted

Subject: [PATCH net 1/1] qede: Fix barrier usage after tx doorbell write.

Since commit c5ad119fb6c09b0297446be05bd66602fa564758
("net: sched: pfifo_fast use skb_array") driver is exposed to an issue
where it is hitting NULL skbs while handling TX completions. Driver
uses mmiowb() to flush the writes to the doorbell bar which is a
write-combined bar, however on x86
mmiowb() does not flush the write combined buffer.

This patch fixes this problem by replacing mmiowb() with wmb() after
the write combined doorbell write so that writes are flushed and
synchronized from more than one processor.

Signed-off-by: Ariel Elior <redacted>
Signed-off-by: Manish Chopra <redacted>
---
 drivers/net/ethernet/qlogic/qede/qede_fp.c |   10 ++++------
 1 files changed, 4 insertions(+), 6 deletions(-)

diff --git a/drivers/net/ethernet/qlogic/qede/qede_fp.c

b/drivers/net/ethernet/qlogic/qede/qede_fp.c
index dafc079..2e921ca 100644

--- a/drivers/net/ethernet/qlogic/qede/qede_fp.c
+++ b/drivers/net/ethernet/qlogic/qede/qede_fp.c

@@ -320,13 +320,11 @@ static inline void

qede_update_tx_producer(struct qede_tx_queue *txq)
 	barrier();
 	writel(txq->tx_db.raw, txq->doorbell_addr);

-	/* mmiowb is needed to synchronize doorbell writes from more than

one

quoted

-	 * processor. It guarantees that the write arrives to the device before
-	 * the queue lock is released and another start_xmit is called (possibly
-	 * on another CPU). Without this barrier, the next doorbell can bypass
-	 * this doorbell. This is applicable to IA64/Altix systems.
+	/* Fence required to flush the write combined buffer, since another
+	 * CPU may write to the same doorbell address and data may be lost
+	 * due to relaxed order nature of write combined bar.
 	 */
-	mmiowb();
+	wmb();
 }

 static int qede_xdp_xmit(struct qede_dev *edev, struct qede_fastpath
*fp,
--
1.7.1

Hi Dave,
This patch appears as "superseded" in patchwork. I am not really sure why that is
- I noticed some other barrier work is going on, but none of it will solve this
issue. This patch solves an important bug in the driver - please consider applying
it.
Thanks,
Ariel

Hi Dave,

The other patchwork which is going on to use writel_relaxed doesn't solve this issue.
I think the writel_relaxed patchwork might have caused this fix as superseded since those changes are in same area/function.
This is an important fix which is after the write combined doorbell write.
Please let me know if I should re-submit this fix or not ?

Thanks,
Manish