RE: [EXT] Re: The problem about arm64: io: Relax implicit barriers in default I/O accessors

(off-list ancestor, not in this archive)

-----Original Message-----
From: Will Deacon <will@kernel.org>
Sent: Thursday, June 17, 2021 4:40 PM
To: Frank Li <frank.li@nxp.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>; Zhi Li <redacted>;
Shenwei Wang [off-list ref]; Han Xu [off-list ref]; Nitin Garg
[off-list ref]; Jason Liu [off-list ref]; linux-arm-
kernel@lists.infradead.org
Subject: Re: [EXT] Re: The problem about arm64: io: Relax implicit barriers
in default I/O accessors

Caution: EXT Email

On Thu, Jun 17, 2021 at 08:11:50PM +0000, Frank Li wrote:

-----Original Message-----
From: Will Deacon <will@kernel.org>
Sent: Thursday, June 17, 2021 12:42 PM
To: Catalin Marinas <catalin.marinas@arm.com>
Cc: Zhi Li <redacted>; Frank Li <frank.li@nxp.com>; Shenwei
Wang
[off-list ref]; Han Xu [off-list ref]; Nitin Garg
[off-list ref]; Jason Liu [off-list ref]; linux-arm-
kernel@lists.infradead.org
Subject: [EXT] Re: The problem about arm64: io: Relax implicit barriers
in
default I/O accessors

Caution: EXT Email

On Thu, Jun 17, 2021 at 06:25:28PM +0100, Will Deacon wrote:
On Thu, Jun 17, 2021 at 10:27:44AM +0100, Catalin Marinas wrote:
On Wed, Jun 16, 2021 at 02:24:39PM -0500, Zhi Li wrote:
On Wed, Jun 16, 2021 at 2:18 PM Frank Li [off-list ref] wrote:
Will Deacon wrote:
It would also be helpful to know a bit more about the
hardware:
  - What is the "internal bus fabric"?

Look like ARM call as "Interconnect",  Multi AXI master and
multi
AXI slave
connected together.
I  drawed simplified bus structure.

        ┌──────┐ ┌────┐
        │ A53  │ │A72 │
        └───┬──┘ └─┬──┘
            │      │
        ┌───▼──────▼──┐
        │    CCI400   │
        └─────┬───────┘
              │   1 (a)write to ddr (normal uncached memory)
              │   DMB OSHST
              │   2 (b)write to usb register(device, nGnRE)
        ┌─────▼───────────────────────┐
┌
───────────┐
        │                             ◄───────┤   GPU
│
        │     Bus fabric              │       │           │
        └────────────────────────────┬┘
└
───────────┘
3 (b) reach usb   ▲ 4 usb read   ▲   │ 6.(a)reach
         │        │   ddr        │   │
      ┌──▼────────┴─┐            │   │
      │             │            │   │
      │  USB        │      5.usb │   │
      │             │      read  │   │
      └─────────────┘            │   │
                               ┌─┴───▼─┐
                               │       │
                               │ DDR   │
                               │       │
                               └───────┘
Since you sent an HTML message, it was rejected by the list server.
The
above is a plain-text rendition by w3m (and changed barrier() to
DMB
OSHST).

Is the DMB propagated to the bus fabric? IIUC, our logic is that if
the
write (b) to USB is observable by, let's say, the GPU, the same GPU
should also observe the write (a) to DDR. Since the write (a) to
DDR is
globally observable, the USB device read at (4) should also observe
it
(well, we may be wrong).
It's pretty rare for barriers to propagate onto the fabric -- usually
the
CPU just orders everything based on acknowledgements. If the CCI
gives
the
write response for the non-cacheable write I could see that causing
an
issue
if the bus fabric can then reorder accesses, but then I would argue
that's a
broken system because simple ring buffers in non-cacheable memory
would
fail
Bus fabric don't reorder the same axi master.
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Felinux.or
g%2Fimages%2F7%2F73%2FDeacon-weak-to-
weedy.pdf&amp;data=04%7C01%7Cfrank.li%40nxp.com%7C5e6b6690d52d4e31d3a408d93
1d88105%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C1%7C637595628211882416%7CU
nknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLC
JXVCI6Mn0%3D%7C1000&amp;sdata=%2BEu10nmFVE1w3fBP11rXD8Wk1vVcvYLirjZQEhSIKCM
%3D&amp;reserved=0
Page 42 show race condition. I think above race condition happen at our
system.
I am not sure if it is exist at Armv8 system.
Just a word of warning here, but the Armv8 memory model was
*retrospectively* strengthened since I gave that talk, so the stuff in that
pdf is out of date (and wrong).

for peripherals hooking into the bus fabric (i.e. dma_*mb() would be
broken). I think it would also mean that DSB doesn't necessarily fix
the
issue, it probably just makes it less likely because it takes longer
to
get the device write out after the acknowledgement -- ndelay() would
achieve
the same effect :)
That's what I worried.

Frank -- what happens if you try either DMB SY, or DMB OSH (without
the
ST)
in writel()?
It works well for 2 hours! Normally, problem happen below 10min. So I
think DMB SY
can fix it.
Oh, interesting. Maybe this is a case where OSH vs SY actually makes a
difference. I'm not quite sure what it means for the coherency of normal,
non-cacheable accesses (which are outer-shareable) so that probably needs a
bit more thought.

Can you confirm that the issue *does* still occur if you use dmb(osh)
instead of dmb(oshst), please?
After get ARM support https://services.arm.com/support/s/case/5003t00001RuJHw, 
This issue have some progress. 

Our system configure SYSBARDISABLE = 0x0, So ARM core barrier propagate to CCI-400

Our DMA and USB is located below downstream of CCI-400. So USB or DMA is located
in system shared domain. Only use dmb(st), CCI-400 wait for previous transaction
Complete. When dma(osh), the response is sent when snoop responses are received for
all earlier transactions. CCI-400 don't wait for previous write finish. 

Best regards
Frank Li

Will
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

`h`	back out one level
`j`	next message in thread
`k`	previous message in thread
`l`	drill in
`Esc`	close help / fold thread tree
`?`	toggle this help