Re: The problem about arm64: io: Relax implicit barriers in default I/O accessors

On Wed, Jun 16, 2021 at 07:40:23PM +0100, Catalin Marinas wrote:

On Mon, Jun 14, 2021 at 10:41:38PM +0000, Frank Li wrote:

quoted

commit 22ec71615d824f4f11d38d0e55a88d8956b7e45f
Author: Will Deacon [off-list ref]
Date:   Fri Jun 7 15:48:58 2019 +0100

    arm64: io: Relax implicit barriers in default I/O accessors

    The arm64 implementation of the default I/O accessors requires barrier
    instructions to satisfy the memory ordering requirements documented in
    memory-barriers.txt [1], which are largely derived from the behaviour of
    I/O accesses on x86.

[...]

quoted

	If I added wmb() before xhci_ring_ep_doorbell, the problem gone.
	Writel include io_wmb, which map into dma_wmb(). 
	
	1. write ddr
	2. writel
		2a. io_wmb(),   dmb(oshst)
		2b, write usb register
	3. usb dma read ddr.

	
	Internal bus fabric only guarantee the order for the same AXID.
	1 write ddr may be slow.  USB register get data before 1 because
	GPU occupy ddr now.  So USB DMA start read from ddr and get old
	dma descriptor data and find not ready yet, then missed door
	bell. 

That's a complex topic, Will should have a better answer. I'll try some
thought exercise below introducing a hypothetical second CPU.

It would also be helpful to know a bit more about the hardware:

  - What is the "internal bus fabric"?
  - Can you be more specific about the AxIDs? I can't tell how that
    correlates back to code running on the CPU.
  - Is the device cache coherent?
  - What memory types are used to map the DDR and the USB register on the
    CPU? (I got lost in the indirection)

Also, do you know which part of the data appears to be stale when the device
reads it?

Will

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel