[PATCH net v2] ibmvnic: Ensure that CRQ entry read are correctly ordered

Ensure that received Command-Response Queue (CRQ) entries are
properly read in order by the driver. dma_rmb barrier has
been added before accessing the CRQ descriptor to ensure
the entire descriptor is read before processing.

Fixes: 032c5e82847a ("Driver for IBM System i/p VNIC protocol")
Signed-off-by: Lijun Pan <redacted>
---
v2: drop dma_wmb according to Jakub's opinion

 drivers/net/ethernet/ibm/ibmvnic.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/drivers/net/ethernet/ibm/ibmvnic.c b/drivers/net/ethernet/ibm/ibmvnic.c
index 9778c83150f1..d84369bd5fc9 100644
--- a/drivers/net/ethernet/ibm/ibmvnic.c
+++ b/drivers/net/ethernet/ibm/ibmvnic.c

@@ -5084,6 +5084,14 @@ static void ibmvnic_tasklet(struct tasklet_struct *t)
 	while (!done) {
 		/* Pull all the valid messages off the CRQ */
 		while ((crq = ibmvnic_next_crq(adapter)) != NULL) {
+			/* Ensure that the entire CRQ descriptor queue->msgs
+			 * has been loaded before reading its contents.
+			 * This barrier makes sure ibmvnic_next_crq()'s
+			 * crq->generic.first & IBMVNIC_CRQ_CMD_RSP is loaded
+			 * before ibmvnic_handle_crq()'s
+			 * switch(gen_crq->first) and switch(gen_crq->cmd).
+			 */
+			dma_rmb();
 			ibmvnic_handle_crq(crq, adapter);
 			crq->generic.first = 0;
 		}

-- 
2.23.0

On Mon, 25 Jan 2021 17:20:23 -0600 Lijun Pan wrote:

I still find this sentence confusing, maybe you mean to say stored
instead of loaded?

+			 * This barrier makes sure ibmvnic_next_crq()'s
+			 * crq->generic.first & IBMVNIC_CRQ_CMD_RSP is loaded
+			 * before ibmvnic_handle_crq()'s
+			 * switch(gen_crq->first) and switch(gen_crq->cmd).

Yup, that makes perfect sense. It's about ordering of the loads. 

+			 */
+			dma_rmb();
 			ibmvnic_handle_crq(crq, adapter);
 			crq->generic.first = 0;
 		}

On Wed, Jan 27, 2021 at 7:06 PM Jakub Kicinski [off-list ref] wrote:

On Mon, 25 Jan 2021 17:20:23 -0600 Lijun Pan wrote:

quoted

Ensure that received Command-Response Queue (CRQ) entries are
properly read in order by the driver. dma_rmb barrier has
been added before accessing the CRQ descriptor to ensure
the entire descriptor is read before processing.

Fixes: 032c5e82847a ("Driver for IBM System i/p VNIC protocol")
Signed-off-by: Lijun Pan <redacted>
---
v2: drop dma_wmb according to Jakub's opinion

 drivers/net/ethernet/ibm/ibmvnic.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/drivers/net/ethernet/ibm/ibmvnic.c b/drivers/net/ethernet/ibm/ibmvnic.c
index 9778c83150f1..d84369bd5fc9 100644
--- a/drivers/net/ethernet/ibm/ibmvnic.c
+++ b/drivers/net/ethernet/ibm/ibmvnic.c

@@ -5084,6 +5084,14 @@ static void ibmvnic_tasklet(struct tasklet_struct *t)
      while (!done) {
              /* Pull all the valid messages off the CRQ */
              while ((crq = ibmvnic_next_crq(adapter)) != NULL) {
+                     /* Ensure that the entire CRQ descriptor queue->msgs
+                      * has been loaded before reading its contents.

I still find this sentence confusing, maybe you mean to say stored
instead of loaded?

The above 2 lines are the general description. The below 4 lines are
detailed explanations. If it is still confusing, we can delete the above
2 lines of comments.

quoted

+                      * This barrier makes sure ibmvnic_next_crq()'s
+                      * crq->generic.first & IBMVNIC_CRQ_CMD_RSP is loaded
+                      * before ibmvnic_handle_crq()'s
+                      * switch(gen_crq->first) and switch(gen_crq->cmd).

Yup, that makes perfect sense. It's about ordering of the loads.

quoted

+                      */
+                     dma_rmb();
                      ibmvnic_handle_crq(crq, adapter);
                      crq->generic.first = 0;
              }

On Wed, 27 Jan 2021 19:22:25 -0600 Lijun Pan wrote:

On Wed, Jan 27, 2021 at 7:06 PM Jakub Kicinski [off-list ref] wrote:

quoted

On Mon, 25 Jan 2021 17:20:23 -0600 Lijun Pan wrote:  

quoted

Ensure that received Command-Response Queue (CRQ) entries are
properly read in order by the driver. dma_rmb barrier has
been added before accessing the CRQ descriptor to ensure
the entire descriptor is read before processing.

Fixes: 032c5e82847a ("Driver for IBM System i/p VNIC protocol")
Signed-off-by: Lijun Pan <redacted>
---
v2: drop dma_wmb according to Jakub's opinion

 drivers/net/ethernet/ibm/ibmvnic.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/drivers/net/ethernet/ibm/ibmvnic.c b/drivers/net/ethernet/ibm/ibmvnic.c
index 9778c83150f1..d84369bd5fc9 100644
--- a/drivers/net/ethernet/ibm/ibmvnic.c
+++ b/drivers/net/ethernet/ibm/ibmvnic.c

@@ -5084,6 +5084,14 @@ static void ibmvnic_tasklet(struct tasklet_struct *t)
      while (!done) {
              /* Pull all the valid messages off the CRQ */
              while ((crq = ibmvnic_next_crq(adapter)) != NULL) {
+                     /* Ensure that the entire CRQ descriptor queue->msgs
+                      * has been loaded before reading its contents.  

I still find this sentence confusing, maybe you mean to say stored
instead of loaded?  

The above 2 lines are the general description. The below 4 lines are
detailed explanations. If it is still confusing, we can delete the above
2 lines of comments.

Yes, I'd find the comment clearer without them, thanks.

On Wed, Jan 27, 2021 at 7:31 PM Jakub Kicinski [off-list ref] wrote:

On Wed, 27 Jan 2021 19:22:25 -0600 Lijun Pan wrote:

quoted

On Wed, Jan 27, 2021 at 7:06 PM Jakub Kicinski [off-list ref] wrote:

quoted

On Mon, 25 Jan 2021 17:20:23 -0600 Lijun Pan wrote:

quoted

Ensure that received Command-Response Queue (CRQ) entries are
properly read in order by the driver. dma_rmb barrier has
been added before accessing the CRQ descriptor to ensure
the entire descriptor is read before processing.

Fixes: 032c5e82847a ("Driver for IBM System i/p VNIC protocol")
Signed-off-by: Lijun Pan <redacted>
---
v2: drop dma_wmb according to Jakub's opinion

 drivers/net/ethernet/ibm/ibmvnic.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/drivers/net/ethernet/ibm/ibmvnic.c b/drivers/net/ethernet/ibm/ibmvnic.c
index 9778c83150f1..d84369bd5fc9 100644
--- a/drivers/net/ethernet/ibm/ibmvnic.c
+++ b/drivers/net/ethernet/ibm/ibmvnic.c

@@ -5084,6 +5084,14 @@ static void ibmvnic_tasklet(struct tasklet_struct *t)
      while (!done) {
              /* Pull all the valid messages off the CRQ */
              while ((crq = ibmvnic_next_crq(adapter)) != NULL) {
+                     /* Ensure that the entire CRQ descriptor queue->msgs
+                      * has been loaded before reading its contents.

I still find this sentence confusing, maybe you mean to say stored
instead of loaded?

The above 2 lines are the general description. The below 4 lines are
detailed explanations. If it is still confusing, we can delete the above
2 lines of comments.

Yes, I'd find the comment clearer without them, thanks.

v3 has been sent out:
https://patchwork.kernel.org/project/netdevbpf/patch/20210128013442.88319-1-ljp@linux.ibm.com/

Thanks,
Lijun