Arch-specific implementions for dma_set_{coherent_,}mask() currently
rely on an inconsistent mix of arch-defined Kconfig symbols and macro
overrides. Now that we have a nice centralised home for DMA API gubbins,
let's consolidate these loose ends under consistent config options.
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
---
Here's hoping the buildbot comes by to point out what I've inevitably
missed, although I did check a cursory cross-compile of ppc64_defconfig
to iron out the obvious howlers.
The motivation here is that I'm looking at adding set_mask overrides
for arm64, and having discovered a bit of a mess it seemed prudent to
clean up before ingraining it any more.
Robin.
arch/arm/Kconfig | 3 ---
arch/powerpc/Kconfig | 4 +---
arch/powerpc/include/asm/dma-mapping.h | 3 ---
include/linux/dma-mapping.h | 4 +++-
kernel/dma/Kconfig | 6 ++++++
5 files changed, 10 insertions(+), 10 deletions(-)
As for the other mask-related hooks, standardise the arch override into
a Kconfig option, and also pull the generic implementation into the DMA
mapping code rather than having it hide away in the platform bus code.
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
---
arch/ia64/Kconfig | 1 +
arch/ia64/include/asm/dma-mapping.h | 2 --
arch/powerpc/Kconfig | 1 +
arch/powerpc/include/asm/device.h | 2 --
drivers/base/platform.c | 23 -----------------------
drivers/pci/controller/vmd.c | 4 ++--
include/linux/dma-mapping.h | 2 +-
kernel/dma/Kconfig | 3 +++
kernel/dma/mapping.c | 23 +++++++++++++++++++++++
9 files changed, 31 insertions(+), 30 deletions(-)
@@ -12,6 +12,7 @@ menu "Processor type and features"configIA64bool+selectARCH_HAS_DMA_GET_REQUIRED_MASKselectARCH_MIGHT_HAVE_PC_PARPORTselectARCH_MIGHT_HAVE_PC_SERIOselectPCIif(!IA64_HP_SIM)
@@ -1179,28 +1178,6 @@ int __init platform_bus_init(void)returnerror;}-#ifndef ARCH_HAS_DMA_GET_REQUIRED_MASK-u64dma_get_required_mask(structdevice*dev)-{-u32low_totalram=((max_pfn-1)<<PAGE_SHIFT);-u32high_totalram=((max_pfn-1)>>(32-PAGE_SHIFT));-u64mask;--if(!high_totalram){-/* convert to mask just covering totalram */-low_totalram=(1<<(fls(low_totalram)-1));-low_totalram+=low_totalram-1;-mask=low_totalram;-}else{-high_totalram=(1<<(fls(high_totalram)-1));-high_totalram+=high_totalram-1;-mask=(((u64)high_totalram)<<32)+0xffffffff;-}-returnmask;-}-EXPORT_SYMBOL_GPL(dma_get_required_mask);-#endif-static__initdataLIST_HEAD(early_platform_driver_list);static__initdataLIST_HEAD(early_platform_device_list);
On Wed, Jul 04, 2018 at 06:50:11PM +0100, Robin Murphy wrote:
Arch-specific implementions for dma_set_{coherent_,}mask() currently
rely on an inconsistent mix of arch-defined Kconfig symbols and macro
overrides. Now that we have a nice centralised home for DMA API gubbins,
let's consolidate these loose ends under consistent config options.
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
---
Here's hoping the buildbot comes by to point out what I've inevitably
missed, although I did check a cursory cross-compile of ppc64_defconfig
to iron out the obvious howlers.
The patch looks sensible to me, although I was hoping to get rid of these
hooks in this or the next merge window as they are a horrible bad idea.
The motivation here is that I'm looking at adding set_mask overrides
for arm64, and having discovered a bit of a mess it seemed prudent to
clean up before ingraining it any more.
What are you trying to do? I really don't want to see more users of
the hooks as they are are a horribly bad idea.
On Wed, Jul 04, 2018 at 06:50:12PM +0100, Robin Murphy wrote:
As for the other mask-related hooks, standardise the arch override into
a Kconfig option, and also pull the generic implementation into the DMA
mapping code rather than having it hide away in the platform bus code.
Heh, I have a somewhat similar patch in my queue. I didn't want it out
because dma_get_required_mask is rather ill defined at the moment and
I wanted to clean that up first. But I guess I could apply this first
and clean up later.
I just fear you might be wanting to add an arm64 user, so I'd really like
to understand why and how.
On Wed, Jul 04, 2018 at 06:50:11PM +0100, Robin Murphy wrote:
quoted
Arch-specific implementions for dma_set_{coherent_,}mask() currently
rely on an inconsistent mix of arch-defined Kconfig symbols and macro
overrides. Now that we have a nice centralised home for DMA API gubbins,
let's consolidate these loose ends under consistent config options.
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
---
Here's hoping the buildbot comes by to point out what I've inevitably
missed, although I did check a cursory cross-compile of ppc64_defconfig
to iron out the obvious howlers.
The patch looks sensible to me, although I was hoping to get rid of these
hooks in this or the next merge window as they are a horrible bad idea.
quoted
The motivation here is that I'm looking at adding set_mask overrides
for arm64, and having discovered a bit of a mess it seemed prudent to
clean up before ingraining it any more.
What are you trying to do? I really don't want to see more users of
the hooks as they are are a horribly bad idea.
I really need to fix the ongoing problem we have where, due to funky
integrations, devices suffer some downstream addressing limit (described
by DT dma-ranges or ACPI IORT/_DMA) which we carefully set up in
dma_configure(), but then just gets lost when the driver probes and
innocently calls dma_set_mask() with something wider. I think it's
effectively the generalised case of the VIA 32-bit quirk, if I
understand that one correctly.
The approach that seemed to me to be safest is largely based on the one
proposed in a thread from ages ago[1]; namely to make dma_configure()
better at distinguishing firmware-specified masks from bus defaults,
capture the firmware mask in dev->archdata during arch_setup_dma_ops(),
then use the custom set_mask routines to ensure any subsequent updates
never exceed that. It doesn't seem possible to make this work robustly
without storing *some* additional per-device data, and for that archdata
is a lesser evil than struct device itself. Plus even though it's not
actually an arch-specific issue it feels like there's such a risk of
breaking other platforms that I'm reticent to even try handling it
entirely in generic code.
Robin.
[1]
https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1306507.html
On Wed, Jul 04, 2018 at 06:50:12PM +0100, Robin Murphy wrote:
quoted
As for the other mask-related hooks, standardise the arch override into
a Kconfig option, and also pull the generic implementation into the DMA
mapping code rather than having it hide away in the platform bus code.
Heh, I have a somewhat similar patch in my queue. I didn't want it out
because dma_get_required_mask is rather ill defined at the moment and
I wanted to clean that up first. But I guess I could apply this first
and clean up later.
I just fear you might be wanting to add an arm64 user, so I'd really like
to understand why and how.
Nah, this guy is just pure cleanup since it also fell out of my 'git
grep ARCH_HAS_DMA'
Robin.
On Fri, Jul 06, 2018 at 03:20:34PM +0100, Robin Murphy wrote:
quoted
What are you trying to do? I really don't want to see more users of
the hooks as they are are a horribly bad idea.
I really need to fix the ongoing problem we have where, due to funky
integrations, devices suffer some downstream addressing limit (described by
DT dma-ranges or ACPI IORT/_DMA) which we carefully set up in
dma_configure(), but then just gets lost when the driver probes and
innocently calls dma_set_mask() with something wider. I think it's
effectively the generalised case of the VIA 32-bit quirk, if I understand
that one correctly.
I'd much rather fix this in generic code. How funky are your limitations?
In fact when I did the 32-bit quirk (which will also be used by a Xiling
PCIe root port usable on a lot of architectures) I did initially consider
adding a bus_dma_mask or similar to struct device, but opted for the
most simple implementation for now. I'd be happy to chanfe this.
Especially these days where busses and IP blocks are generally not tied
to a specific cpu instruction set I really believe that having any
more architecture code than absolutely required is a bad idea.
The approach that seemed to me to be safest is largely based on the one
proposed in a thread from ages ago[1]; namely to make dma_configure()
better at distinguishing firmware-specified masks from bus defaults,
capture the firmware mask in dev->archdata during arch_setup_dma_ops(),
then use the custom set_mask routines to ensure any subsequent updates
never exceed that. It doesn't seem possible to make this work robustly
without storing *some* additional per-device data, and for that archdata is
a lesser evil than struct device itself. Plus even though it's not actually
an arch-specific issue it feels like there's such a risk of breaking other
platforms that I'm reticent to even try handling it entirely in generic
code.
My plan for a few merge windows from now is that dma_mask and
coherent_mask are 100% in device control and dma_set_mask will never
fail. It will be up to the dma ops to make sure things are addressible.
On Fri, Jul 06, 2018 at 03:20:34PM +0100, Robin Murphy wrote:
quoted
quoted
What are you trying to do? I really don't want to see more users of
the hooks as they are are a horribly bad idea.
I really need to fix the ongoing problem we have where, due to funky
integrations, devices suffer some downstream addressing limit (described by
DT dma-ranges or ACPI IORT/_DMA) which we carefully set up in
dma_configure(), but then just gets lost when the driver probes and
innocently calls dma_set_mask() with something wider. I think it's
effectively the generalised case of the VIA 32-bit quirk, if I understand
that one correctly.
I'd much rather fix this in generic code. How funky are your limitations?
In fact when I did the 32-bit quirk (which will also be used by a Xiling
PCIe root port usable on a lot of architectures) I did initially consider
adding a bus_dma_mask or similar to struct device, but opted for the
most simple implementation for now. I'd be happy to chanfe this.
Especially these days where busses and IP blocks are generally not tied
to a specific cpu instruction set I really believe that having any
more architecture code than absolutely required is a bad idea.
Oh, for sure, the generic fix would be the longer-term goal, this was
just an expedient compromise because I want to get *something* landed
for 4.19. Since in practice this is predominantly affecting arm64, doing
the arch-specific fix to appease affected customers then working to
generalise it afterwards seemed to carry the lowest risk.
That said, I think I can see a relatively safe and clean alternative
approach based on converting dma_32bit_limit to a mask, so I'll spin
some patches around that idea ASAP to continue the discussion.
quoted
The approach that seemed to me to be safest is largely based on the one
proposed in a thread from ages ago[1]; namely to make dma_configure()
better at distinguishing firmware-specified masks from bus defaults,
capture the firmware mask in dev->archdata during arch_setup_dma_ops(),
then use the custom set_mask routines to ensure any subsequent updates
never exceed that. It doesn't seem possible to make this work robustly
without storing *some* additional per-device data, and for that archdata is
a lesser evil than struct device itself. Plus even though it's not actually
an arch-specific issue it feels like there's such a risk of breaking other
platforms that I'm reticent to even try handling it entirely in generic
code.
My plan for a few merge windows from now is that dma_mask and
coherent_mask are 100% in device control and dma_set_mask will never
fail. It will be up to the dma ops to make sure things are addressible.
It's entirely possible to plug an old PCI soundcard via a bridge adapter
into a modern board where the card's 24-bit DMA mask reaches nothing but
the SoC's boot flash, and no IOMMU is available (e.g. some of the
smaller NXP Layercape stuff); I still think there should be an error in
such rare cases when DMA is utterly impossible, but otherwise I agree it
would be much nicer for drivers to just provide their preferred mask and
let the ops massage it as necessary.
Robin.
On Wed, Jul 04, 2018 at 06:50:12PM +0100, Robin Murphy wrote:
As for the other mask-related hooks, standardise the arch override into
a Kconfig option, and also pull the generic implementation into the DMA
mapping code rather than having it hide away in the platform bus code.
I compared this a bit to what I had around against an older kernel,
and I think we should probably go with something more like the
version I had, which I can dust off again.
What I've done is to:
1) provide the get_required_mask unconditionally in struct dma_map_ops
2) default to what is the current dma_get_required_mask implementation
if nothing else is specified.
What I still had on my todo list but not done yet:
3) go through all instances and check if the current default
makes sense, at it based on direct addressability. For most
iommu instances it seems like we should just return a 64-bit mask.
4) figure out how to take the dma offsets into account for it
5) move the function to the dma-direct code, as that is where it
belongs
5) figure out if there is a better name for the method, as with
swiotlb & co it isn't really the required mask, but more something
like the optimal mask
6) document the whole thing..
7) sort out the powerpc indirection mess.
Do you agree with that general plan? If so I can dust off my old
patch.
On Mon, Jul 09, 2018 at 03:53:50PM +0100, Robin Murphy wrote:
Oh, for sure, the generic fix would be the longer-term goal, this was just
an expedient compromise because I want to get *something* landed for 4.19.
Since in practice this is predominantly affecting arm64, doing the
arch-specific fix to appease affected customers then working to generalise
it afterwards seemed to carry the lowest risk.
That said, I think I can see a relatively safe and clean alternative
approach based on converting dma_32bit_limit to a mask, so I'll spin some
patches around that idea ASAP to continue the discussion.
Great! I really want to sort out this area as soon as possible, and
introducing more arch specific code isn't really helping with that. In
fact my whole drive to consolidate code is based on the fact that
I want to fix issues like this in one code base instead of 20 slightly
different ones.
FYI, this is the Xilinx/RISC-V use case for the 32-bit limit,
which I'll need to respin a bit based on linux-pci feedback:
http://git.infradead.org/users/hch/misc.git/shortlog/refs/heads/xilinx-dma-32
It's entirely possible to plug an old PCI soundcard via a bridge adapter
into a modern board where the card's 24-bit DMA mask reaches nothing but
the SoC's boot flash, and no IOMMU is available (e.g. some of the smaller
NXP Layercape stuff); I still think there should be an error in such rare
cases when DMA is utterly impossible, but otherwise I agree it would be
much nicer for drivers to just provide their preferred mask and let the ops
massage it as necessary.
Ok, I guess we need still need to keep the return value for that.
On Wed, Jul 04, 2018 at 06:50:12PM +0100, Robin Murphy wrote:
quoted
As for the other mask-related hooks, standardise the arch override into
a Kconfig option, and also pull the generic implementation into the DMA
mapping code rather than having it hide away in the platform bus code.
I compared this a bit to what I had around against an older kernel,
and I think we should probably go with something more like the
version I had, which I can dust off again.
What I've done is to:
1) provide the get_required_mask unconditionally in struct dma_map_ops
2) default to what is the current dma_get_required_mask implementation
if nothing else is specified.
Yeah, there's already 17 pointers in dma_map_ops of which about half are
optional, so these awkward #ifdefs to save one more probably aren't
worth the inconsistency they bring. It feels like this guy mostly goes
hand-in-hand with dma_supported, so ack to giving it the same look and feel.
What I still had on my todo list but not done yet:
3) go through all instances and check if the current default
makes sense, at it based on direct addressability. For most
iommu instances it seems like we should just return a 64-bit mask.
That's reasonable, although in many cases we should know the effective
IOMMU input address size which would be even neater.
4) figure out how to take the dma offsets into account for it
AFAICS it might boil down to simply:
mask = roundup_pow_of_two(phys_to_dma(dev, PFN_PHYS(max_pfn))) - 1;
5) move the function to the dma-direct code, as that is where it
belongs
5) figure out if there is a better name for the method, as with
swiotlb & co it isn't really the required mask, but more something
like the optimal mask
6) document the whole thing..
7) sort out the powerpc indirection mess.
Do you agree with that general plan? If so I can dust off my old
patch.
Sounds good; in the meantime I'll happily drop these two.
Robin.
On Tue, Jul 10, 2018 at 01:29:20PM +0100, Robin Murphy wrote:
quoted
What I've done is to:
1) provide the get_required_mask unconditionally in struct dma_map_ops
2) default to what is the current dma_get_required_mask implementation
if nothing else is specified.
Yeah, there's already 17 pointers in dma_map_ops of which about half are
optional, so these awkward #ifdefs to save one more probably aren't worth
the inconsistency they bring. It feels like this guy mostly goes
hand-in-hand with dma_supported, so ack to giving it the same look and
feel.
This whole area needs a major refactoring - we currentl have three
different APIs to deal with addressability: dma_get_required_mask,
dma_capable/dma_set_mask and dma_capable from dma-direct.h, and there
is plenty of unexplainable mismatches between them.
Sorting this out has been on my TODO list, but I think it can only
effectively be done once the direct mapping implementations are
reasonably consolidated.
quoted
What I still had on my todo list but not done yet:
3) go through all instances and check if the current default
makes sense, at it based on direct addressability. For most
iommu instances it seems like we should just return a 64-bit mask.
That's reasonable, although in many cases we should know the effective
IOMMU input address size which would be even neater.
Sure. Maybe I just need to steps 1 and 2 and let maintainers fill
in.
quoted
4) figure out how to take the dma offsets into account for it
AFAICS it might boil down to simply:
mask = roundup_pow_of_two(phys_to_dma(dev, PFN_PHYS(max_pfn))) - 1;
That looks way to sensible. Which reminds me that I need to research
the history behind the low_totalram/high_totalram magic in
dma_get_required_mask.