ARM64 CONFIG_ZONE_DMA for 32-bit devices

On 02/03/17 18:35, Catalin Marinas wrote:

On Tue, Feb 28, 2017 at 12:42:51PM +0000, Robin Murphy wrote:

quoted

On 28/02/17 10:34, Kashyap Desai wrote:

quoted

I was reading below articles. Mine is not similar issue, but  I understand
few things about ARM64 SWIOTLB interface from below discussion.
Any input will be a great help to resolve/understand the issue.

https://patchwork.codeaurora.org/patch/143833/
https://patchwork.kernel.org/patch/9495893/

Current problem statement is -
"Trying to load kdump kernel from above 4GB memory does not work on ARM64
platform as <megaraid_sas> driver require certain DMA buffer from below
4GB memory range."

Looking for alternative/workaround for time being. Long term plan is to
remove limitation of <megaraid_sas> driver to remove 32 bit DMA mask for
SAS3.0 onwards controller.

1) I found below code @arch/arm64/mm/init.c . ARM64 kernel has provision
to support 32-bit DMA as well.  I am not sure about why CONFIG_ZONE_DMA
option is an configurable option for ARM64 ?

        /* 4GB maximum for 32-bit only capable devices */
        if (IS_ENABLED(CONFIG_ZONE_DMA))
                arm64_dma_phys_limit = max_zone_dma_phys();
        else
                arm64_dma_phys_limit = PHYS_MASK + 1;
        dma_contiguous_reserve(arm64_dma_phys_limit);

One of the reason I think "kdump" kernel can load from above 4GB memory
range provided crashkernel=<>, high and crashkernel=0, low option. So I
guess ARM64 kdump kernel may have disabled CONFIG_ZONE_DMA option, but
base kernel must have enabled CONFIG_ZONE_DMA option to support 32-bit
only capable devices.

I believe it's more that ZONE_DMA goes a bit crazy when the available
RAM starts above 4GB. It's not technically possible to turn it off
without hacking Kconfig.

I think even if you hack Kconfig, the kernel may not build. We keep the
Kconfig option so that enum zone_type has ZONE_DMA defined.

Regarding the DMA zone selection, max_zone_dma_phys() is more of a hack
(needed on Seattle where all RAM is above 4GB). Basically the first
32-bit at the start of RAM are considered for ZONE_DMA, even though the
actual physical address of the start of RAM would be well beyond 4GB.
This assumes that 32-bit only devices have the relevant dma_pfn_offset
passed via DT (not sure what we do on ACPI).

Except Seattle doesn't have dma_pfn_offsets :/

The DT has an identity-mapped "dma-ranges", and I've seen it
demonstrated that a PCI card whose driver sets a 32-bit mask simply has
all DMA API calls fail.

quoted

quoted

2.)

Typically - SWIOTLB uses DMA buffer from below 4GB range only. ARM64 is
the only architecture which support Low memory definition as per ARCH
specified. See below

[root@ linux]# grep -R ARCH_LOW_ADDRESS_LIMIT arch/
arch/s390/include/asm/processor.h:#define ARCH_LOW_ADDRESS_LIMIT
0x7fffffffUL
arch/arm64/include/asm/processor.h:#define ARCH_LOW_ADDRESS_LIMIT
(arm64_dma_phys_limit - 1)

For only ARM64, it is possible to get SWTBL DMA buffer above 4GB. See
below snippet from crashed kernel on ARM64.

[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000005fc0000000-0x0000005fffffffff]
[    0.000000]   Normal   empty

I guess that's because the kernel thinks 0x5fc00000 is the start of all
RAM that is available and just assumes that ZONE_DMA would be covered by
the lower 32-bit of this (high) range.

The physical address in the 32-bit DMA context is rather irrelevant.
What you need is the actual DMA address that the device is seeing and
this is calculated by phys_to_dma (taking dma_pfn_offset into account).

quoted

quoted

SWIOTLB can map 64MB buffer from above 4GB only on ARM64 machine and that
is causing problem for <megaraid_sas> driver.
Current megaraid_sas driver wants certain resources from below 4GB memory
and that is why it request consistent dma mask as below -
pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(32)).

If I do the same on x86_64, SWTBL INIT will fail because there is no Low
memory below 4GB. See below prints from x86_64 machine.

[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000001000-0x0000000000ffffff]
[    0.000000]   DMA32    [mem 0x0000000001000000-0x00000000ffffffff]
[    0.000000]   Normal   [mem 0x0000000100000000-0x000000407effffff]
  [    0.000000] Movable zone start for each node
   ..
   [    0.000000] Cannot allocate SWIOTLB buffer

Question is - "ARM64 platform can't allocate memory for crash kernel in
below 4GB range ?"

If you want to use a device which requires 32-bit-addressable DMA
resources with your crash kernel, and that device isn't behind an IOMMU,
then don't load your crash kernel above 4GB. It's as simple as that,
because in general there's no other way around the issue. And if said
device doesn't actually need 32-bit-addressable resources, then yeah,
fix the dma_set_mask() calls in the driver.

I agree. I don't think there is much we can do, other than parsing all
dma_pfn_offsets early on in DT and deciding whether the ZONE_DMA
heuristics actually helps (and, if not, print some warning).

quoted

That said, I think something is a bit wonky in max_zone_dma_phys() with
"It currently assumes that for memory starting above 4G, 32-bit devices
will use a DMA offset" - I think that assumption needs to be revisited
since, even disregarding cases like kdump, commonly available hardware
now exists for which that is not true (e.g. AMD Seattle). Catalin?

IIRC, I did this specifically for Seattle, though not sure whether it
was just a matter of failing memory allocations when ZONE_DMA was empty
rather than a device actually using it. That's the best we could do if
there actually is a device with the relevant dma_pfn_offset.

The alternative would be to put everything in ZONE_DMA if the RAM is
beyond 4GB but it doesn't help if we do have devices with a proper
dma_pfn_offset. Leaving ZONE_DMA empty probably has other implications
with failing allocations (you can fall back from ZONE_NORMAL to ZONE_DMA
but not the other way around).

I'd be more inclined to take the latter approach - the vast majority of
(if not all) systems where this is even a concern at all have IOMMUs,
which make ZONE_DMA rather moot as a concept once you can freely
allocate pages to back buffer mappings from anywhere you like. Of
course, it might be nice to avoid allocating a useless SWIOTLB buffer in
such cases, but I guess that's perhaps a separate problem in itself.

Robin.