On Tue, Sep 17, 2019 at 02:03:04PM +0100, Robin Murphy wrote:

On 17/09/2019 13:33, Russell King - ARM Linux admin wrote:
[...]

quoted

Further debug shows:

coherent=0 - sdhci device is not cache coherent
swapper pgtable: 4k pages, 39-bit VAs, pgdp=0000000081cac000
[ffffff8010fd5200] pgd=000000237ffff003, pud=000000237ffff003,
pmd=000000237fffb003, pte=00e800236d62270f

The mapping for the ADMA table seems to be using MAIR index 3, which is
MT_MEMORY_NC, so should be non-cacheable.

vmallocinfo:
0xffffff8010fd5000-0xffffff8010fd7000    8192 dma_direct_alloc+0x4c/0x54
user

So this memory has been remapped.  Could there be an alias that has
cache lines still in the cache for the physical address, and could we
be hitting those cache lines while accessing through a non-cacheable
mapping?  (On 32-bit ARM, this is "unpredictable" and this problem
definitely _feels_ like it has unpredictable attributes!)

Also, given that this memory is mapped NC, then surely
__dma_flush_area() should have no effect?  However, it _does_ have the
effect of reliably solving the problem, which to me implies that there
_are_ cache lines in this NC mapping.

The non-cacheable mapping of the descriptor table will still have its
cacheable linear map alias, so it's quite likely that the invalidate aspect
of __dma_flush_area(), rather than the clean, is what's making the
difference - if using __dma_clean_area() instead doesn't help, it would more
or less confirm that.

One possibility in that case is that you might actually have the rare
backwards coherency problem - if the device *is* actually snooping the
cache, then it could hit lines which were speculatively prefetched via the
cacheable alias before the descriptors were fully written, rather than the
up-to-date data which went straight to RAM via the NC mapping. I'd try
declaring the device as "dma-coherent" to see if that's actually true (and
it should become pretty obvious if it isn't).

As just mentioned in my previous reply, there's a commit to the
dma-contiguous which changes where the CMA memory comes from.

[ffffff8010fd5200] pgd=000000237ffff003, pud=000000237ffff003,
	pmd=000000237fffb003, pte=00e800236d62270f

vs

[ffffff8010fd5200] pgd=000000237ffff003, pud=000000237ffff003,
	pmd=000000237fffb003, pte=00e80000f9c9a70f

Former is with the patch applied, latter is with it reverted.

This makes me question whether the cache handling for a page that is
remapped is being performed.  If there's cache lines present for a
page that is being remapped as non-cacheable, what prevents those
cache lines from being dirty and possibly being written-back at some
point after the non-cacheable mapping as been started to be used?

And yes, it looks like adding "dma-coherent" to the SDHCI controller
with the SD card in resolves the issue, so your hypothesis may be
true.  On the other hand, I haven't added "dma-coherent" to the eMMC
side, and that's also working fine over several reboots without the
offending commit reverted _nor_ with my __dma_flush_area() hack in
place.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 12.1Mbps down 622kbps up
According to speedtest.net: 11.9Mbps down 500kbps up

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel