[RFC 1/3 v3] mm: iommu: An API to unify IOMMU, CPU and device memory management

-----Original Message-----
From: linux-arm-kernel-bounces at lists.infradead.org [mailto:linux-arm-
kernel-bounces at lists.infradead.org] On Behalf Of Russell King - ARM Linux
Sent: Wednesday, July 21, 2010 4:00 AM
To: stepanm at codeaurora.org
Cc: linux-arch at vger.kernel.org; dwalker at codeaurora.org; mel at csn.ul.ie;
linux-arm-msm at vger.kernel.org; linux-kernel at vger.kernel.org; FUJITA
Tomonori; linux-mm at kvack.org; andi at firstfloor.org; Zach Pfeffer; Michael
Bohan; Tim HRM; linux-omap at vger.kernel.org; linux-arm-
kernel at lists.infradead.org; ebiederm at xmission.com
Subject: Re: [RFC 1/3 v3] mm: iommu: An API to unify IOMMU, CPU and device
memory management

On Tue, Jul 20, 2010 at 03:02:34PM -0700, stepanm at codeaurora.org wrote:

quoted

Russell-

If a driver wants to allow a device to access memory (and cache

coherency

quoted

is off/not present for device addesses), the driver needs to remap that
memory as non-cacheable.

If that memory is not part of the kernel's managed memory, then that's
fine.  But if it _is_ part of the kernel's managed memory, then it is
not permitted by the ARM architecture specification to allow maps of
the memory with differing [memory type, sharability, cache] attributes.

Basically, if a driver wants to create these kinds of mappings, then
they should expect the system to become unreliable and unpredictable.
That's not something any sane person should be aiming to do.

quoted

Suppose there exists a chunk of
physically-contiguous memory (say, memory reserved for device use) that
happened to be already mapped into the kernel as normal memory

(cacheable,

quoted

etc). One way to remap this memory is to use ioremap (and then never

touch

quoted

the original virtual mapping, which would now have conflicting
attributes).

This doesn't work, and is unpredictable on ARMv6 and ARMv7.  Not touching
the original mapping is _not_ _sufficient_ to guarantee that the mapping
is not used.  (We've seen problems on OMAP as a result of this.)

Any mapping which exists can be speculatively prefetched by such CPUs
at any time, which can lead it to be read into the cache.  Then, your
different attributes for your "other" mapping can cause problems if you
hit one of these cache lines - and then you can have (possibly silent)
data corruption.

quoted

I feel as if there should be a better way to remap memory for
device access, either by altering the attributes on the original

mapping,

quoted

or removing the original mapping and creating a new one with attributes
set to non-cacheable.

This is difficult to achieve without remapping kernel memory using L2
page tables, so we can unmap pages on 4K page granularity.  That's
going to increase TLB overhead and result in lower system performance
as there'll be a greater number of MMU misses.

However, one obvious case would be to use highmem-only pages for
remapping - but you then have to ensure that those pages are never
kmapped in any way, because those mappings will fall into the same
unpredictable category that we're already trying to avoid.  This
may be possible, but you'll have to ensure that most of the system
RAM is in highmem - which poses other problems (eg, if lowmem gets
low.)

Why can't we consider an option of removing the old mappings when 
we need to create new ones with different attributes as suggested
by Catalin on similar thread previously. This will avoid the duplicate
mapping with different attributes issue on newer ARMs.

Is this something can't be worked out?

Regards,
Santosh