RE: [EXT] Re: The problem about arm64: io: Relax implicit barriers in default I/O accessors

-----Original Message-----
From: Will Deacon <will@kernel.org>
Sent: Thursday, June 17, 2021 12:42 PM
To: Catalin Marinas <catalin.marinas@arm.com>
Cc: Zhi Li <redacted>; Frank Li <frank.li@nxp.com>; Shenwei Wang
[off-list ref]; Han Xu [off-list ref]; Nitin Garg
[off-list ref]; Jason Liu [off-list ref]; linux-arm-
kernel@lists.infradead.org
Subject: [EXT] Re: The problem about arm64: io: Relax implicit barriers in
default I/O accessors

Caution: EXT Email

On Thu, Jun 17, 2021 at 06:25:28PM +0100, Will Deacon wrote:

quoted

On Thu, Jun 17, 2021 at 10:27:44AM +0100, Catalin Marinas wrote:

quoted

On Wed, Jun 16, 2021 at 02:24:39PM -0500, Zhi Li wrote:

quoted

On Wed, Jun 16, 2021 at 2:18 PM Frank Li [off-list ref] wrote:

quoted

Will Deacon wrote:

quoted

It would also be helpful to know a bit more about the hardware:

  - What is the "internal bus fabric"?

quoted

Look like ARM call as "Interconnect",  Multi AXI master and multi

AXI slave

quoted

quoted

quoted

quoted

connected together.

I  drawed simplified bus structure.

        ┌──────┐ ┌────┐
        │ A53  │ │A72 │
        └───┬──┘ └─┬──┘
            │      │
        ┌───▼──────▼──┐
        │    CCI400   │
        └─────┬───────┘
              │   1 (a)write to ddr (normal uncached memory)
              │   DMB OSHST
              │   2 (b)write to usb register(device, nGnRE)
        ┌─────▼───────────────────────┐       ┌

───────────┐

quoted

quoted

quoted

        │                             ◄───────┤   GPU     │
        │     Bus fabric              │       │           │
        └────────────────────────────┬┘       └

───────────┘

quoted

quoted

quoted

3 (b) reach usb   ▲ 4 usb read   ▲   │ 6.(a)reach
         │        │   ddr        │   │
      ┌──▼────────┴─┐            │   │
      │             │            │   │
      │  USB        │      5.usb │   │
      │             │      read  │   │
      └─────────────┘            │   │
                               ┌─┴───▼─┐
                               │       │
                               │ DDR   │
                               │       │
                               └───────┘

Since you sent an HTML message, it was rejected by the list server. The
above is a plain-text rendition by w3m (and changed barrier() to DMB
OSHST).

Is the DMB propagated to the bus fabric? IIUC, our logic is that if the
write (b) to USB is observable by, let's say, the GPU, the same GPU
should also observe the write (a) to DDR. Since the write (a) to DDR is
globally observable, the USB device read at (4) should also observe it
(well, we may be wrong).

It's pretty rare for barriers to propagate onto the fabric -- usually the
CPU just orders everything based on acknowledgements. If the CCI gives

the

quoted

write response for the non-cacheable write I could see that causing an

issue

quoted

if the bus fabric can then reorder accesses, but then I would argue

that's a

quoted

broken system because simple ring buffers in non-cacheable memory would

fail

Bus fabric don't reorder the same axi master. 
https://elinux.org/images/7/73/Deacon-weak-to-weedy.pdf
Page 42 show race condition. I think above race condition happen at our system.
I am not sure if it is exist at Armv8 system.

quoted

for peripherals hooking into the bus fabric (i.e. dma_*mb() would be
broken). I think it would also mean that DSB doesn't necessarily fix the
issue, it probably just makes it less likely because it takes longer to
get the device write out after the acknowledgement -- ndelay() would

achieve

quoted

the same effect :)

That's what I worried. 

quoted

Frank -- what happens if you try either DMB SY, or DMB OSH (without the

ST)

quoted

in writel()?

It works well for 2 hours! Normally, problem happen below 10min. So I think DMB SY
can fix it. 

diff --git a/arch/arm64/include/asm/barrier.h b/arch/arm64/include/asm/barrier.h
index c3009b0e52393..277c9d1c1a8fa 100644
--- a/arch/arm64/include/asm/barrier.h
+++ b/arch/arm64/include/asm/barrier.h

@@ -47,7 +47,7 @@

 #define dma_mb()       dmb(osh)
 #define dma_rmb()      dmb(oshld)
-#define dma_wmb()      dmb(oshst)
+#define dma_wmb()      dmb(sy)

Also, digging into the A72 TRM there are a bunch of configuration signals
in this area; see SYSBARDISABLE and BROADCASTOUTER, for example.

Does the failure happen on both a53 and a72, or only on one CPU type?

Both A53, A72 have this problem. 

Will

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel