From: Alexander Duyck <hidden> Date: 2016-10-24 18:05:08
The first 21 patches in the set add support for the DMA attribute
DMA_ATTR_SKIP_CPU_SYNC on multiple platforms/architectures. This is needed
so that we can flag the calls to dma_map/unmap_page so that we do not
invalidate cache lines that do not currently belong to the device. Instead
we have to take care of this in the driver via a call to
sync_single_range_for_cpu prior to freeing the Rx page.
Patch 22 adds support for dma_map_page_attrs and dma_unmap_page_attrs so
that we can unmap and map a page using the DMA_ATTR_SKIP_CPU_SYNC
attribute.
Patch 23 adds support for freeing a page that has multiple references being
held by a single caller. This way we can free page fragments that were
allocated by a given driver.
The last 3 patches use these updates in the igb driver to allow for us to
reimplement the use of build_skb which hands a writable page off to the
stack.
My hope is to get the series accepted into the net-next tree as I have a
number of other Intel drivers I could then begin updating once these
patches are accepted.
Any feedback is welcome. Specifically if there is something I overlooked
design-wise or an architecture I missed please let me know and I will add
it to this patch set. If needed I can look into breaking this into a
smaller set of patches but this set is all that should be needed to then
start looking at putting together a DMA page pool per device which I know
is something Jesper has been working on.
---
Alexander Duyck (26):
swiotlb: Drop unused function swiotlb_map_sg
swiotlb: Add support for DMA_ATTR_SKIP_CPU_SYNC
arch/arc: Add option to skip sync on DMA mapping
arch/arm: Add option to skip sync on DMA map and unmap
arch/avr32: Add option to skip sync on DMA map
arch/blackfin: Add option to skip sync on DMA map
arch/c6x: Add option to skip sync on DMA map and unmap
arch/frv: Add option to skip sync on DMA map
arch/hexagon: Add option to skip DMA sync as a part of mapping
arch/m68k: Add option to skip DMA sync as a part of mapping
arch/metag: Add option to skip DMA sync as a part of map and unmap
arch/microblaze: Add option to skip DMA sync as a part of map and unmap
arch/mips: Add option to skip DMA sync as a part of map and unmap
arch/nios2: Add option to skip DMA sync as a part of map and unmap
arch/openrisc: Add option to skip DMA sync as a part of mapping
arch/parisc: Add option to skip DMA sync as a part of map and unmap
arch/powerpc: Add option to skip DMA sync as a part of mapping
arch/sh: Add option to skip DMA sync as a part of mapping
arch/sparc: Add option to skip DMA sync as a part of map and unmap
arch/tile: Add option to skip DMA sync as a part of map and unmap
arch/xtensa: Add option to skip DMA sync as a part of mapping
dma: Add calls for dma_map_page_attrs and dma_unmap_page_attrs
mm: Add support for releasing multiple instances of a page
igb: Update driver to make use of DMA_ATTR_SKIP_CPU_SYNC
igb: Update code to better handle incrementing page count
igb: Revert "igb: Revert support for build_skb in igb"
arch/arc/mm/dma.c | 3
arch/arm/common/dmabounce.c | 16 +-
arch/avr32/mm/dma-coherent.c | 7 +
arch/blackfin/kernel/dma-mapping.c | 7 +
arch/c6x/kernel/dma.c | 16 ++
arch/frv/mb93090-mb00/pci-dma-nommu.c | 16 ++
arch/frv/mb93090-mb00/pci-dma.c | 7 +
arch/hexagon/kernel/dma.c | 6 +
arch/m68k/kernel/dma.c | 8 +
arch/metag/kernel/dma.c | 16 ++
arch/microblaze/kernel/dma.c | 10 +
arch/mips/loongson64/common/dma-swiotlb.c | 2
arch/mips/mm/dma-default.c | 8 +
arch/nios2/mm/dma-mapping.c | 14 ++
arch/openrisc/kernel/dma.c | 3
arch/parisc/kernel/pci-dma.c | 20 ++-
arch/powerpc/kernel/dma.c | 9 +
arch/sh/kernel/dma-nommu.c | 7 +
arch/sparc/kernel/iommu.c | 4 -
arch/sparc/kernel/ioport.c | 4 -
arch/tile/kernel/pci-dma.c | 12 +-
arch/xtensa/kernel/pci-dma.c | 7 +
drivers/net/ethernet/intel/igb/igb.h | 36 ++++-
drivers/net/ethernet/intel/igb/igb_main.c | 207 +++++++++++++++++++++++------
drivers/xen/swiotlb-xen.c | 40 +++---
include/linux/dma-mapping.h | 20 ++-
include/linux/gfp.h | 2
include/linux/swiotlb.h | 10 +
lib/swiotlb.c | 56 ++++----
mm/page_alloc.c | 14 ++
30 files changed, 435 insertions(+), 152 deletions(-)
--
Signature
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Alexander Duyck <hidden> Date: 2016-10-24 18:05:17
There are no users for swiotlb_map_sg so we might as well just drop it.
Cc: Konrad Rzeszutek Wilk <redacted>
Signed-off-by: Alexander Duyck <redacted>
---
include/linux/swiotlb.h | 4 ----
lib/swiotlb.c | 8 --------
2 files changed, 12 deletions(-)
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Alexander Duyck <hidden> Date: 2016-10-24 18:05:22
As a first step to making DMA_ATTR_SKIP_CPU_SYNC apply to architectures
beyond just ARM I need to make it so that the swiotlb will respect the
flag. In order to do that I also need to update the swiotlb-xen since it
heavily makes use of the functionality.
Cc: Konrad Rzeszutek Wilk <redacted>
Signed-off-by: Alexander Duyck <redacted>
---
drivers/xen/swiotlb-xen.c | 40 ++++++++++++++++++++++----------------
include/linux/swiotlb.h | 6 ++++--
lib/swiotlb.c | 48 +++++++++++++++++++++++++++------------------
3 files changed, 56 insertions(+), 38 deletions(-)
@@ -669,7 +675,8 @@ void swiotlb_tbl_sync_single(struct device *hwdev, phys_addr_t tlb_addr,/* DMA_TO_DEVICE to avoid memcpy in unmap_single */swiotlb_tbl_unmap_single(hwdev,paddr,-size,DMA_TO_DEVICE);+size,DMA_TO_DEVICE,+DMA_ATTR_SKIP_CPU_SYNC);gotoerr_warn;}}
@@ -699,7 +706,7 @@ void swiotlb_tbl_sync_single(struct device *hwdev, phys_addr_t tlb_addr,free_pages((unsignedlong)vaddr,get_order(size));else/* DMA_TO_DEVICE to avoid memcpy in swiotlb_tbl_unmap_single */-swiotlb_tbl_unmap_single(hwdev,paddr,size,DMA_TO_DEVICE);+swiotlb_tbl_unmap_single(hwdev,paddr,size,DMA_TO_DEVICE,0);}EXPORT_SYMBOL(swiotlb_free_coherent);
@@ -755,7 +762,7 @@ dma_addr_t swiotlb_map_page(struct device *dev, struct page *page,trace_swiotlb_bounced(dev,dev_addr,size,swiotlb_force);/* Oh well, have to allocate and map a bounce buffer. */-map=map_single(dev,phys,size,dir);+map=map_single(dev,phys,size,dir,attrs);if(map==SWIOTLB_MAP_ERROR){swiotlb_full(dev,size,dir,1);returnphys_to_dma(dev,io_tlb_overflow_buffer);
@@ -764,12 +771,13 @@ dma_addr_t swiotlb_map_page(struct device *dev, struct page *page,dev_addr=phys_to_dma(dev,map);/* Ensure that the address returned is DMA'ble */-if(!dma_capable(dev,dev_addr,size)){-swiotlb_tbl_unmap_single(dev,map,size,dir);-returnphys_to_dma(dev,io_tlb_overflow_buffer);-}+if(dma_capable(dev,dev_addr,size))+returndev_addr;++swiotlb_tbl_unmap_single(dev,map,size,dir,+attrs|DMA_ATTR_SKIP_CPU_SYNC);-returndev_addr;+returnphys_to_dma(dev,io_tlb_overflow_buffer);}EXPORT_SYMBOL_GPL(swiotlb_map_page);
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Alexander Duyck <hidden> Date: 2016-10-24 18:05:26
This change allows us to pass DMA_ATTR_SKIP_CPU_SYNC which allows us to
avoid invoking cache line invalidation if the driver will just handle it
later via a sync_for_cpu or sync_for_device call.
Cc: Vineet Gupta <redacted>
Cc: linux-snps-arc@lists.infradead.org
Signed-off-by: Alexander Duyck <redacted>
---
arch/arc/mm/dma.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
From: Alexander Duyck <hidden> Date: 2016-10-24 18:05:29
The use of DMA_ATTR_SKIP_CPU_SYNC was not consistent across all of the DMA
APIs in the arch/arm folder. This change is meant to correct that so that
we get consistent behavior.
Cc: Russell King <linux@armlinux.org.uk>
Signed-off-by: Alexander Duyck <redacted>
---
arch/arm/common/dmabounce.c | 16 ++++++++++------
1 file changed, 10 insertions(+), 6 deletions(-)
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Alexander Duyck <hidden> Date: 2016-10-24 18:05:34
The use of DMA_ATTR_SKIP_CPU_SYNC was not consistent across all of the DMA
APIs in the arch/arm folder. This change is meant to correct that so that
we get consistent behavior.
Cc: Haavard Skinnemoen <redacted>
Cc: Hans-Christian Egtvedt <redacted>
Signed-off-by: Alexander Duyck <redacted>
---
arch/avr32/mm/dma-coherent.c | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Alexander Duyck <hidden> Date: 2016-10-24 18:05:48
This change allows us to pass DMA_ATTR_SKIP_CPU_SYNC which allows us to
avoid invoking cache line invalidation if the driver will just handle it
later via a sync_for_cpu or sync_for_device call.
Cc: Mark Salter <redacted>
Cc: Aurelien Jacquiot <redacted>
Cc: linux-c6x-dev@linux-c6x.org
Signed-off-by: Alexander Duyck <redacted>
---
arch/c6x/kernel/dma.c | 16 +++++++++++-----
1 file changed, 11 insertions(+), 5 deletions(-)
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Alexander Duyck <hidden> Date: 2016-10-24 18:05:56
The use of DMA_ATTR_SKIP_CPU_SYNC was not consistent across all of the DMA
APIs in the arch/arm folder. This change is meant to correct that so that
we get consistent behavior.
Cc: Steven Miao <redacted>
Signed-off-by: Alexander Duyck <redacted>
---
arch/blackfin/kernel/dma-mapping.c | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Alexander Duyck <hidden> Date: 2016-10-24 18:06:02
This change allows us to pass DMA_ATTR_SKIP_CPU_SYNC which allows us to
avoid invoking cache line invalidation if the driver will just handle it
later via a sync_for_cpu or sync_for_device call.
Cc: Geert Uytterhoeven <geert@linux-m68k.org>
Cc: linux-m68k@lists.linux-m68k.org
Signed-off-by: Alexander Duyck <redacted>
---
arch/m68k/kernel/dma.c | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
From: Alexander Duyck <hidden> Date: 2016-10-24 18:06:06
This change allows us to pass DMA_ATTR_SKIP_CPU_SYNC which allows us to
avoid invoking cache line invalidation if the driver will just handle it
via a sync_for_cpu or sync_for_device call.
Cc: James Hogan <redacted>
Cc: linux-metag@vger.kernel.org
Signed-off-by: Alexander Duyck <redacted>
---
arch/metag/kernel/dma.c | 16 +++++++++++++---
1 file changed, 13 insertions(+), 3 deletions(-)
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Alexander Duyck <hidden> Date: 2016-10-24 18:06:12
This change allows us to pass DMA_ATTR_SKIP_CPU_SYNC which allows us to
avoid invoking cache line invalidation if the driver will just handle it
via a sync_for_cpu or sync_for_device call.
Cc: Michal Simek <monstr@monstr.eu>
Signed-off-by: Alexander Duyck <redacted>
---
arch/microblaze/kernel/dma.c | 10 ++++++++--
1 file changed, 8 insertions(+), 2 deletions(-)
@@ -61,6 +61,10 @@ static int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl,/* FIXME this part of code is untested */for_each_sg(sgl,sg,nents,i){sg->dma_address=sg_phys(sg);++if(attrs&DMA_ATTR_SKIP_CPU_SYNC)+continue;+__dma_sync(page_to_phys(sg_page(sg))+sg->offset,sg->length,direction);}
From: Alexander Duyck <hidden> Date: 2016-10-24 18:06:16
This change allows us to pass DMA_ATTR_SKIP_CPU_SYNC which allows us to
avoid invoking cache line invalidation if the driver will just handle it
via a sync_for_cpu or sync_for_device call.
Cc: Ralf Baechle <redacted>
Cc: Keguang Zhang <keguang.zhang@gmail.com>
Cc: linux-mips@linux-mips.org
Signed-off-by: Alexander Duyck <redacted>
---
arch/mips/loongson64/common/dma-swiotlb.c | 2 +-
arch/mips/mm/dma-default.c | 8 +++++---
2 files changed, 6 insertions(+), 4 deletions(-)
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Alexander Duyck <hidden> Date: 2016-10-24 18:06:26
This change allows us to pass DMA_ATTR_SKIP_CPU_SYNC which allows us to
avoid invoking cache line invalidation if the driver will just handle it
via a sync_for_cpu or sync_for_device call.
Cc: Ley Foon Tan <redacted>
Signed-off-by: Alexander Duyck <redacted>
---
arch/nios2/mm/dma-mapping.c | 14 +++++++++++---
1 file changed, 11 insertions(+), 3 deletions(-)
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Alexander Duyck <hidden> Date: 2016-10-24 18:06:32
This change allows us to pass DMA_ATTR_SKIP_CPU_SYNC which allows us to
avoid invoking cache line invalidation if the driver will just handle it
via a sync_for_cpu or sync_for_device call.
Cc: "James E.J. Bottomley" <redacted>
Cc: Helge Deller <deller@gmx.de>
Cc: linux-parisc@vger.kernel.org
Signed-off-by: Alexander Duyck <redacted>
---
arch/parisc/kernel/pci-dma.c | 20 +++++++++++++++-----
1 file changed, 15 insertions(+), 5 deletions(-)
@@ -510,14 +518,16 @@ static void pa11_dma_unmap_sg(struct device *dev, struct scatterlist *sglist,BUG_ON(direction==DMA_NONE);+if(attrs&DMA_ATTR_SKIP_CPU_SYNC)+return;+if(direction==DMA_TO_DEVICE)-return;+return;/* once we do combining we'll need to use phys_to_virt(sg_dma_address(sglist)) */for_each_sg(sglist,sg,nents,i)flush_kernel_vmap_range(sg_virt(sg),sg->length);-return;}staticvoidpa11_dma_sync_single_for_cpu(structdevice*dev,--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Alexander Duyck <hidden> Date: 2016-10-24 18:06:38
This change allows us to pass DMA_ATTR_SKIP_CPU_SYNC which allows us to
avoid invoking cache line invalidation if the driver will just handle it
via a sync_for_cpu or sync_for_device call.
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Paul Mackerras <redacted>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: linuxppc-dev@lists.ozlabs.org
Signed-off-by: Alexander Duyck <redacted>
---
arch/powerpc/kernel/dma.c | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Alexander Duyck <hidden> Date: 2016-10-24 18:06:44
This change allows us to pass DMA_ATTR_SKIP_CPU_SYNC which allows us to
avoid invoking cache line invalidation if the driver will just handle it
via a sync_for_cpu or sync_for_device call.
Cc: Yoshinori Sato <ysato@users.sourceforge.jp>
Cc: Rich Felker <dalias@libc.org>
Cc: linux-sh@vger.kernel.org
Signed-off-by: Alexander Duyck <redacted>
---
arch/sh/kernel/dma-nommu.c | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Alexander Duyck <hidden> Date: 2016-10-24 18:06:48
This change allows us to pass DMA_ATTR_SKIP_CPU_SYNC which allows us to
avoid invoking cache line invalidation if the driver will just handle it
via a sync_for_cpu or sync_for_device call.
Cc: "David S. Miller" <davem@davemloft.net>
Cc: sparclinux@vger.kernel.org
Signed-off-by: Alexander Duyck <redacted>
---
arch/sparc/kernel/iommu.c | 4 ++--
arch/sparc/kernel/ioport.c | 4 ++--
2 files changed, 4 insertions(+), 4 deletions(-)
From: Alexander Duyck <hidden> Date: 2016-10-24 18:06:55
This change allows us to pass DMA_ATTR_SKIP_CPU_SYNC which allows us to
avoid invoking cache line invalidation if the driver will just handle it
via a sync_for_cpu or sync_for_device call.
Cc: Chris Metcalf <redacted>
Signed-off-by: Alexander Duyck <redacted>
---
arch/tile/kernel/pci-dma.c | 12 ++++++++++--
1 file changed, 10 insertions(+), 2 deletions(-)
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Alexander Duyck <hidden> Date: 2016-10-24 18:07:02
This change allows us to pass DMA_ATTR_SKIP_CPU_SYNC which allows us to
avoid invoking cache line invalidation if the driver will just handle it
via a sync_for_cpu or sync_for_device call.
Cc: Max Filippov <jcmvbkbc@gmail.com>
Signed-off-by: Alexander Duyck <redacted>
---
arch/xtensa/kernel/pci-dma.c | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Alexander Duyck <hidden> Date: 2016-10-24 18:07:07
Add support for mapping and unmapping a page with attributes. The primary
use for this is currently to allow for us to pass the
DMA_ATTR_SKIP_CPU_SYNC attribute when mapping and unmapping a page. On
some architectures such as ARM the synchronization has significant overhead
and if we are already taking care of the sync_for_cpu and sync_for_device
from the driver there isn't much need to handle this in the map/unmap calls
as well.
Signed-off-by: Alexander Duyck <redacted>
---
include/linux/dma-mapping.h | 20 +++++++++++++-------
1 file changed, 13 insertions(+), 7 deletions(-)
@@ -385,6 +389,8 @@ static inline void dma_sync_single_range_for_device(struct device *dev,#define dma_unmap_single(d, a, s, r) dma_unmap_single_attrs(d, a, s, r, 0)#define dma_map_sg(d, s, n, r) dma_map_sg_attrs(d, s, n, r, 0)#define dma_unmap_sg(d, s, n, r) dma_unmap_sg_attrs(d, s, n, r, 0)+#define dma_map_page(d, p, o, s, r) dma_map_page_attrs(d, p, o, s, r, 0)+#define dma_unmap_page(d, a, s, r) dma_unmap_page_attrs(d, a, s, r, 0)externintdma_common_mmap(structdevice*dev,structvm_area_struct*vma,void*cpu_addr,dma_addr_tdma_addr,size_tsize);--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Alexander Duyck <hidden> Date: 2016-10-24 18:07:18
The ARM architecture provides a mechanism for deferring cache line
invalidation in the case of map/unmap. This patch makes use of this
mechanism to avoid unnecessary synchronization.
A secondary effect of this change is that the portion of the page that has
been synchronized for use by the CPU should be writable and could be passed
up the stack (at least on ARM).
The last bit that occurred to me is that on architectures where the
sync_for_cpu call invalidates cache lines we were prefetching and then
invalidating the first 128 bytes of the packet. To avoid that I have moved
the sync up to before we perform the prefetch and allocate the skbuff so
that we can actually make use of it.
Signed-off-by: Alexander Duyck <redacted>
---
drivers/net/ethernet/intel/igb/igb_main.c | 53 ++++++++++++++++++-----------
1 file changed, 33 insertions(+), 20 deletions(-)
@@ -3947,10 +3947,21 @@ static void igb_clean_rx_ring(struct igb_ring *rx_ring)if(!buffer_info->page)continue;-dma_unmap_page(rx_ring->dev,-buffer_info->dma,-PAGE_SIZE,-DMA_FROM_DEVICE);+/* Invalidate cache lines that may have been written to by+*devicesothatweavoidcorruptingmemory.+*/+dma_sync_single_range_for_cpu(rx_ring->dev,+buffer_info->dma,+buffer_info->page_offset,+IGB_RX_BUFSZ,+DMA_FROM_DEVICE);++/* free resources associated with mapping */+dma_unmap_page_attrs(rx_ring->dev,+buffer_info->dma,+PAGE_SIZE,+DMA_FROM_DEVICE,+DMA_ATTR_SKIP_CPU_SYNC);__free_page(buffer_info->page);buffer_info->page=NULL;
@@ -6808,12 +6819,6 @@ static void igb_reuse_rx_page(struct igb_ring *rx_ring,/* transfer page from old buffer to new buffer */*new_buff=*old_buff;--/* sync the buffer for use by the device */-dma_sync_single_range_for_device(rx_ring->dev,old_buff->dma,-old_buff->page_offset,-IGB_RX_BUFSZ,-DMA_FROM_DEVICE);}staticinlinebooligb_page_is_reserved(structpage*page)
@@ -6934,6 +6939,13 @@ static struct sk_buff *igb_fetch_rx_buffer(struct igb_ring *rx_ring,page=rx_buffer->page;prefetchw(page);+/* we are reusing so sync this buffer for CPU use */+dma_sync_single_range_for_cpu(rx_ring->dev,+rx_buffer->dma,+rx_buffer->page_offset,+size,+DMA_FROM_DEVICE);+if(likely(!skb)){void*page_addr=page_address(page)+rx_buffer->page_offset;
@@ -6958,21 +6970,15 @@ static struct sk_buff *igb_fetch_rx_buffer(struct igb_ring *rx_ring,prefetchw(skb->data);}-/* we are reusing so sync this buffer for CPU use */-dma_sync_single_range_for_cpu(rx_ring->dev,-rx_buffer->dma,-rx_buffer->page_offset,-size,-DMA_FROM_DEVICE);-/* pull page into skb */if(igb_add_rx_frag(rx_ring,rx_buffer,size,rx_desc,skb)){/* hand second half of page back to the ring */igb_reuse_rx_page(rx_ring,rx_buffer);}else{/* we are not reusing the buffer so unmap it */-dma_unmap_page(rx_ring->dev,rx_buffer->dma,-PAGE_SIZE,DMA_FROM_DEVICE);+dma_unmap_page_attrs(rx_ring->dev,rx_buffer->dma,+PAGE_SIZE,DMA_FROM_DEVICE,+DMA_ATTR_SKIP_CPU_SYNC);}/* clear contents of rx_buffer */
@@ -7230,7 +7236,8 @@ static bool igb_alloc_mapped_page(struct igb_ring *rx_ring,}/* map page for use */-dma=dma_map_page(rx_ring->dev,page,0,PAGE_SIZE,DMA_FROM_DEVICE);+dma=dma_map_page_attrs(rx_ring->dev,page,0,PAGE_SIZE,+DMA_FROM_DEVICE,DMA_ATTR_SKIP_CPU_SYNC);/* if mapping failed free memory back to system since*thereisn'tmuchpointinholdingmemorywecan'tuse
@@ -7271,6 +7278,12 @@ void igb_alloc_rx_buffers(struct igb_ring *rx_ring, u16 cleaned_count)if(!igb_alloc_mapped_page(rx_ring,bi))break;+/* sync the buffer for use by the device */+dma_sync_single_range_for_device(rx_ring->dev,bi->dma,+bi->page_offset,+IGB_RX_BUFSZ,+DMA_FROM_DEVICE);+/* Refresh the desc even if buffer_addrs didn't change*becauseeachwrite-backerasesthisinfo.*/--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Alexander Duyck <hidden> Date: 2016-10-24 18:07:20
This patch adds a function that allows us to batch free a page that has
multiple references outstanding. Specifically this function can be used to
drop a page being used in the page frag alloc cache. With this drivers can
make use of functionality similar to the page frag alloc cache without
having to do any workarounds for the fact that there is no function that
frees multiple references.
Cc: linux-mm@kvack.org
Signed-off-by: Alexander Duyck <redacted>
---
include/linux/gfp.h | 2 ++
mm/page_alloc.c | 14 ++++++++++++++
2 files changed, 16 insertions(+)
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Alexander Duyck <hidden> Date: 2016-10-24 18:07:25
This reverts commit f9d40f6a9921 ("igb: Revert support for build_skb in
igb") and adds a few changes to update it to work with the latest version
of igb. We are now able to revert the removal of this due to the fact
that with the recent changes to the page count and the use of
DMA_ATTR_SKIP_CPU_SYNC we can make the pages writable so we should not be
invalidating the additional data added when we call build_skb.
The biggest risk with this change is that we are now not able to support
full jumbo frames when using build_skb. Instead we can only support up to
2K minus the skb overhead and padding offset.
Signed-off-by: Alexander Duyck <redacted>
---
drivers/net/ethernet/intel/igb/igb.h | 29 ++++++
drivers/net/ethernet/intel/igb/igb_main.c | 130 ++++++++++++++++++++++++++---
2 files changed, 142 insertions(+), 17 deletions(-)
@@ -145,6 +145,10 @@ struct vf_data_storage {#define IGB_RX_HDR_LEN IGB_RXBUFFER_256#define IGB_RX_BUFSZ IGB_RXBUFFER_2048+#define IGB_SKB_PAD (NET_SKB_PAD + NET_IP_ALIGN)+#define IGB_MAX_BUILD_SKB_SIZE \+(SKB_WITH_OVERHEAD(IGB_RX_BUFSZ)-(IGB_SKB_PAD+IGB_TS_HDR_LEN))+/* How many Rx Buffers do we bundle into one write to the hardware ? */#define IGB_RX_BUFFER_WRITE 16 /* Must be power of 2 */
@@ -3761,6 +3761,16 @@ void igb_configure_rx_ring(struct igb_adapter *adapter,wr32(E1000_RXDCTL(reg_idx),rxdctl);}+staticvoidigb_set_rx_buffer_len(structigb_adapter*adapter,+structigb_ring*rx_ring)+{+/* set build_skb flag */+if(adapter->max_frame_size<=IGB_MAX_BUILD_SKB_SIZE)+set_ring_build_skb_enabled(rx_ring);+else+clear_ring_build_skb_enabled(rx_ring);+}+/***igb_configure_rx-ConfigurereceiveUnitafterReset*@adapter:boardprivatestructure
@@ -3778,8 +3788,12 @@ static void igb_configure_rx(struct igb_adapter *adapter)/* Setup the HW Rx Head and Tail Descriptor Pointers and*theBaseandLengthoftheRxDescriptorRing*/-for(i=0;i<adapter->num_rx_queues;i++)-igb_configure_rx_ring(adapter,adapter->rx_ring[i]);+for(i=0;i<adapter->num_rx_queues;i++){+structigb_ring*rx_ring=adapter->rx_ring[i];++igb_set_rx_buffer_len(adapter,rx_ring);+igb_configure_rx_ring(adapter,rx_ring);+}}/**
@@ -4238,7 +4252,7 @@ static void igb_set_rx_mode(struct net_device *netdev)structigb_adapter*adapter=netdev_priv(netdev);structe1000_hw*hw=&adapter->hw;unsignedintvfn=adapter->vfs_allocated_count;-u32rctl=0,vmolr=0;+u32rctl=0,vmolr=0,rlpml=MAX_JUMBO_FRAME_SIZE;intcount;/* Check for Promiscuous and All Multicast modes */
@@ -4310,12 +4324,18 @@ static void igb_set_rx_mode(struct net_device *netdev)vmolr|=rd32(E1000_VMOLR(vfn))&~(E1000_VMOLR_ROPE|E1000_VMOLR_MPME|E1000_VMOLR_ROMPE);-/* enable Rx jumbo frames, no need for restriction */+/* enable Rx jumbo frames, restrict as needed to support build_skb */vmolr&=~E1000_VMOLR_RLPML_MASK;-vmolr|=MAX_JUMBO_FRAME_SIZE|E1000_VMOLR_LPE;+vmolr|=E1000_VMOLR_LPE;+vmolr|=(adapter->max_frame_size<=IGB_MAX_BUILD_SKB_SIZE)?+IGB_MAX_BUILD_SKB_SIZE:MAX_JUMBO_FRAME_SIZE;++if(!adapter->vfs_allocated_count&&+(adapter->max_frame_size<=IGB_MAX_BUILD_SKB_SIZE))+rlpml=IGB_MAX_BUILD_SKB_SIZE;wr32(E1000_VMOLR(vfn),vmolr);-wr32(E1000_RLPML,MAX_JUMBO_FRAME_SIZE);+wr32(E1000_RLPML,rlpml);igb_restore_vf_multicasts(adapter);}
@@ -6933,6 +6953,78 @@ static bool igb_add_rx_frag(struct igb_ring *rx_ring,returnigb_can_reuse_rx_page(rx_buffer,page,truesize);}+staticstructsk_buff*igb_build_rx_buffer(structigb_ring*rx_ring,+unione1000_adv_rx_desc*rx_desc)+{+unsignedintsize=le16_to_cpu(rx_desc->wb.upper.length);+structigb_rx_buffer*rx_buffer;+structsk_buff*skb;+structpage*page;+void*va;+#if (PAGE_SIZE < 8192)+constunsignedinttruesize=IGB_RX_BUFSZ;+#else+unsignedinttruesize=SKB_DATA_ALIGN(sizeof(structskb_shared_info))++SKB_DATA_ALIGN(NET_SKB_PAD++NET_IP_ALIGN++size);+#endif++rx_buffer=&rx_ring->rx_buffer_info[rx_ring->next_to_clean];+page=rx_buffer->page;+prefetchw(page);++/* we are reusing so sync this buffer for CPU use */+dma_sync_single_range_for_cpu(rx_ring->dev,+rx_buffer->dma,+rx_buffer->page_offset+IGB_SKB_PAD,+size,+DMA_FROM_DEVICE);++va=page_address(page)+rx_buffer->page_offset;++/* prefetch first cache line of first page */+prefetch(va+IGB_SKB_PAD);+#if L1_CACHE_BYTES < 128+prefetch(va+L1_CACHE_BYTES+IGB_SKB_PAD);+#endif++/* build an skb to around the page buffer */+skb=build_skb(va,truesize);+if(unlikely(!skb)){+rx_ring->rx_stats.alloc_failed++;+returnNULL;+}++/* update pointers within the skb to store the data */+skb_reserve(skb,IGB_SKB_PAD);+__skb_put(skb,size);++/* pull timestamp out of packet data */+if(igb_test_staterr(rx_desc,E1000_RXDADV_STAT_TSIP)){+igb_ptp_rx_pktstamp(rx_ring->q_vector,skb->data,skb);+__skb_pull(skb,IGB_TS_HDR_LEN);+}++if(igb_can_reuse_rx_page(rx_buffer,page,truesize)){+/* hand second half of page back to the ring */+igb_reuse_rx_page(rx_ring,rx_buffer);+}else{+/* We are not reusing the buffer so unmap it and free+*anyreferencesweareholdingtoit+*/+dma_unmap_page_attrs(rx_ring->dev,rx_buffer->dma,+PAGE_SIZE,DMA_FROM_DEVICE,+DMA_ATTR_SKIP_CPU_SYNC);+__page_frag_drain(page,0,rx_buffer->pagecnt_bias);+}++/* clear contents of rx_buffer */+rx_buffer->page=NULL;++returnskb;+}+staticstructsk_buff*igb_fetch_rx_buffer(structigb_ring*rx_ring,unione1000_adv_rx_desc*rx_desc,structsk_buff*skb)
@@ -7178,7 +7270,10 @@ static int igb_clean_rx_irq(struct igb_q_vector *q_vector, const int budget)dma_rmb();/* retrieve a buffer from the ring */-skb=igb_fetch_rx_buffer(rx_ring,rx_desc,skb);+if(ring_uses_build_skb(rx_ring))+skb=igb_build_rx_buffer(rx_ring,rx_desc);+else+skb=igb_fetch_rx_buffer(rx_ring,rx_desc,skb);/* exit if we failed to retrieve a buffer */if(!skb)
From: Alexander Duyck <hidden> Date: 2016-10-24 18:08:05
This patch updates the driver code so that we do bulk updates of the page
reference count instead of just incrementing it by one reference at a time.
The advantage to doing this is that we cut down on atomic operations and
this in turn should give us a slight improvement in cycles per packet. In
addition if we eventually move this over to using build_skb the gains will
be more noticeable.
Signed-off-by: Alexander Duyck <redacted>
---
drivers/net/ethernet/intel/igb/igb.h | 7 ++++++-
drivers/net/ethernet/intel/igb/igb_main.c | 24 +++++++++++++++++-------
2 files changed, 23 insertions(+), 8 deletions(-)
@@ -6830,13 +6831,15 @@ static bool igb_can_reuse_rx_page(struct igb_rx_buffer *rx_buffer,structpage*page,unsignedinttruesize){+unsignedintpagecnt_bias=rx_buffer->pagecnt_bias--;+/* avoid re-using remote pages */if(unlikely(igb_page_is_reserved(page)))returnfalse;#if (PAGE_SIZE < 8192)/* if we are only owner of page we can reuse it */-if(unlikely(page_count(page)!=1))+if(unlikely(page_ref_count(page)!=pagecnt_bias))returnfalse;/* flip page offset to other buffer */
@@ -6849,10 +6852,14 @@ static bool igb_can_reuse_rx_page(struct igb_rx_buffer *rx_buffer,returnfalse;#endif-/* Even if we own the page, we are not allowed to use atomic_set()-*Thiswouldbreakget_page_unless_zero()users.+/* If we have drained the page fragment pool we need to update+*thepagecnt_biasandpagecountsothatwefullyrestockthe+*numberofreferencesthedriverholds.*/-page_ref_inc(page);+if(unlikely(!rx_buffer->pagecnt_bias)){+page_ref_add(page,USHRT_MAX);+rx_buffer->pagecnt_bias=USHRT_MAX;+}returntrue;}
@@ -6904,7 +6911,6 @@ static bool igb_add_rx_frag(struct igb_ring *rx_ring,returntrue;/* this page cannot be reused so discard it */-__free_page(page);returnfalse;}
@@ -6975,10 +6981,13 @@ static struct sk_buff *igb_fetch_rx_buffer(struct igb_ring *rx_ring,/* hand second half of page back to the ring */igb_reuse_rx_page(rx_ring,rx_buffer);}else{-/* we are not reusing the buffer so unmap it */+/* We are not reusing the buffer so unmap it and free+*anyreferencesweareholdingtoit+*/dma_unmap_page_attrs(rx_ring->dev,rx_buffer->dma,PAGE_SIZE,DMA_FROM_DEVICE,DMA_ATTR_SKIP_CPU_SYNC);+__page_frag_drain(page,0,rx_buffer->pagecnt_bias);}/* clear contents of rx_buffer */
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Konrad Rzeszutek Wilk <hidden> Date: 2016-10-24 18:09:51
On Mon, Oct 24, 2016 at 08:04:37AM -0400, Alexander Duyck wrote:
quoted hunk
As a first step to making DMA_ATTR_SKIP_CPU_SYNC apply to architectures
beyond just ARM I need to make it so that the swiotlb will respect the
flag. In order to do that I also need to update the swiotlb-xen since it
heavily makes use of the functionality.
Cc: Konrad Rzeszutek Wilk <redacted>
Signed-off-by: Alexander Duyck <redacted>
---
drivers/xen/swiotlb-xen.c | 40 ++++++++++++++++++++++----------------
include/linux/swiotlb.h | 6 ++++--
lib/swiotlb.c | 48 +++++++++++++++++++++++++++------------------
3 files changed, 56 insertions(+), 38 deletions(-)
Why? This change (re-ordering the code - and returning DMA_ERROR_CODE instead
of 0) does not have anything to do with the title.
If you really feel strongly about it - then please send it as a seperate patch.
quoted hunk
}
EXPORT_SYMBOL_GPL(xen_swiotlb_map_page);
@@ -444,7 +447,7 @@ static void xen_unmap_single(struct device *hwdev, dma_addr_t dev_addr, /* NOTE: We use dev_addr here, not paddr! */ if (is_xen_swiotlb_buffer(dev_addr)) {- swiotlb_tbl_unmap_single(hwdev, paddr, size, dir);+ swiotlb_tbl_unmap_single(hwdev, paddr, size, dir, attrs); return; }
@@ -557,16 +560,9 @@ void xen_swiotlb_unmap_page(struct device *hwdev, dma_addr_t dev_addr, start_dma_addr, sg_phys(sg), sg->length,- dir);- if (map == SWIOTLB_MAP_ERROR) {- dev_warn(hwdev, "swiotlb buffer is full\n");- /* Don't panic here, we expect map_sg users- to do proper error handling. */- xen_swiotlb_unmap_sg_attrs(hwdev, sgl, i, dir,- attrs);- sg_dma_len(sgl) = 0;- return 0;- }+ dir, attrs);+ if (map == SWIOTLB_MAP_ERROR)+ goto map_error; xen_dma_map_page(hwdev, pfn_to_page(map >> PAGE_SHIFT), dev_addr, map & ~PAGE_MASK,
@@ -589,6 +585,16 @@ void xen_swiotlb_unmap_page(struct device *hwdev, dma_addr_t dev_addr, sg_dma_len(sg) = sg->length; } return nelems;+map_error:+ dev_warn(hwdev, "swiotlb buffer is full\n");+ /*+ * Don't panic here, we expect map_sg users+ * to do proper error handling.+ */+ xen_swiotlb_unmap_sg_attrs(hwdev, sgl, i, dir,+ attrs | DMA_ATTR_SKIP_CPU_SYNC);+ sg_dma_len(sgl) = 0;+ return 0; }
This too. Why can't that be part of the existing code that was there?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Alexander Duyck <hidden> Date: 2016-10-24 18:10:25
This change allows us to pass DMA_ATTR_SKIP_CPU_SYNC which allows us to
avoid invoking cache line invalidation if the driver will just handle it
via a sync_for_cpu or sync_for_device call.
Cc: Jonas Bonn <jonas@southpole.se>
Signed-off-by: Alexander Duyck <redacted>
---
arch/openrisc/kernel/dma.c | 3 +++
1 file changed, 3 insertions(+)
@@ -141,6 +141,9 @@unsignedlongcl;dma_addr_taddr=page_to_phys(page)+offset;+if(attrs&DMA_ATTR_SKIP_CPU_SYNC)+returnaddr;+switch(dir){caseDMA_TO_DEVICE:/* Flush the dcache for the requested range */--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Konrad Rzeszutek Wilk <hidden> Date: 2016-10-24 18:11:26
On Mon, Oct 24, 2016 at 08:04:31AM -0400, Alexander Duyck wrote:
There are no users for swiotlb_map_sg so we might as well just drop it.
Cc: Konrad Rzeszutek Wilk <redacted>
Acked-by: Konrad Rzeszutek Wilk <redacted>
Thought I swear I saw a familiar patch by Christopher Hellwig at some point..
but maybe that patchset had been dropped.
From: Alexander Duyck <hidden> Date: 2016-10-24 18:12:52
The use of DMA_ATTR_SKIP_CPU_SYNC was not consistent across all of the DMA
APIs in the arch/arm folder. This change is meant to correct that so that
we get consistent behavior.
Signed-off-by: Alexander Duyck <redacted>
---
arch/frv/mb93090-mb00/pci-dma-nommu.c | 16 +++++++++++-----
arch/frv/mb93090-mb00/pci-dma.c | 7 ++++++-
2 files changed, 17 insertions(+), 6 deletions(-)
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Alexander Duyck <hidden> Date: 2016-10-24 18:12:54
This change allows us to pass DMA_ATTR_SKIP_CPU_SYNC which allows us to
avoid invoking cache line invalidation if the driver will just handle it
later via a sync_for_cpu or sync_for_device call.
Cc: Richard Kuo <redacted>
Cc: linux-hexagon@vger.kernel.org
Signed-off-by: Alexander Duyck <redacted>
---
arch/hexagon/kernel/dma.c | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: David Miller <davem@davemloft.net> Date: 2016-10-24 18:27:34
From: Alexander Duyck <redacted>
Date: Mon, 24 Oct 2016 08:06:07 -0400
This change allows us to pass DMA_ATTR_SKIP_CPU_SYNC which allows us to
avoid invoking cache line invalidation if the driver will just handle it
via a sync_for_cpu or sync_for_device call.
Cc: "David S. Miller" <davem@davemloft.net>
Cc: sparclinux@vger.kernel.org
Signed-off-by: Alexander Duyck <redacted>
This is fine for avoiding the flush for performance reasons, but the
chip isn't going to write anything back unless the device wrote into
the area.
Around Mon 24 Oct 2016 08:04:53 -0400 or thereabout, Alexander Duyck wrote:
The use of DMA_ATTR_SKIP_CPU_SYNC was not consistent across all of the DMA
APIs in the arch/arm folder. This change is meant to correct that so that
we get consistent behavior.
From: Alexander Duyck <hidden> Date: 2016-10-24 19:18:02
On Mon, Oct 24, 2016 at 11:09 AM, Konrad Rzeszutek Wilk
[off-list ref] wrote:
On Mon, Oct 24, 2016 at 08:04:37AM -0400, Alexander Duyck wrote:
quoted
As a first step to making DMA_ATTR_SKIP_CPU_SYNC apply to architectures
beyond just ARM I need to make it so that the swiotlb will respect the
flag. In order to do that I also need to update the swiotlb-xen since it
heavily makes use of the functionality.
Cc: Konrad Rzeszutek Wilk <redacted>
Signed-off-by: Alexander Duyck <redacted>
---
drivers/xen/swiotlb-xen.c | 40 ++++++++++++++++++++++----------------
include/linux/swiotlb.h | 6 ++++--
lib/swiotlb.c | 48 +++++++++++++++++++++++++++------------------
3 files changed, 56 insertions(+), 38 deletions(-)
Why? This change (re-ordering the code - and returning DMA_ERROR_CODE instead
of 0) does not have anything to do with the title.
If you really feel strongly about it - then please send it as a seperate patch.
Okay I can do that. This was mostly just to clean up the formatting
because I was over 80 characters when I added the attribute. Changing
the return value to DMA_ERROR_CODE from 0 was based on the fact that
earlier in the function that is the value you return if there is a
mapping error.
quoted
}
EXPORT_SYMBOL_GPL(xen_swiotlb_map_page);
@@ -444,7 +447,7 @@ static void xen_unmap_single(struct device *hwdev, dma_addr_t dev_addr, /* NOTE: We use dev_addr here, not paddr! */ if (is_xen_swiotlb_buffer(dev_addr)) {- swiotlb_tbl_unmap_single(hwdev, paddr, size, dir);+ swiotlb_tbl_unmap_single(hwdev, paddr, size, dir, attrs); return; }
@@ -557,16 +560,9 @@ void xen_swiotlb_unmap_page(struct device *hwdev, dma_addr_t dev_addr, start_dma_addr, sg_phys(sg), sg->length,- dir);- if (map == SWIOTLB_MAP_ERROR) {- dev_warn(hwdev, "swiotlb buffer is full\n");- /* Don't panic here, we expect map_sg users- to do proper error handling. */- xen_swiotlb_unmap_sg_attrs(hwdev, sgl, i, dir,- attrs);- sg_dma_len(sgl) = 0;- return 0;- }+ dir, attrs);+ if (map == SWIOTLB_MAP_ERROR)+ goto map_error; xen_dma_map_page(hwdev, pfn_to_page(map >> PAGE_SHIFT), dev_addr, map & ~PAGE_MASK,
@@ -589,6 +585,16 @@ void xen_swiotlb_unmap_page(struct device *hwdev, dma_addr_t dev_addr, sg_dma_len(sg) = sg->length; } return nelems;+map_error:+ dev_warn(hwdev, "swiotlb buffer is full\n");+ /*+ * Don't panic here, we expect map_sg users+ * to do proper error handling.+ */+ xen_swiotlb_unmap_sg_attrs(hwdev, sgl, i, dir,+ attrs | DMA_ATTR_SKIP_CPU_SYNC);+ sg_dma_len(sgl) = 0;+ return 0; }
This too. Why can't that be part of the existing code that was there?
Once again it was a formatting thing. I was indented too far and
adding the attribute pushed me over 80 characters so I broke it out to
a label to avoid the problem.
- Alex
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Alexander Duyck <hidden> Date: 2016-10-24 19:39:11
On Mon, Oct 24, 2016 at 11:27 AM, David Miller [off-list ref] wrote:
From: Alexander Duyck <redacted>
Date: Mon, 24 Oct 2016 08:06:07 -0400
quoted
This change allows us to pass DMA_ATTR_SKIP_CPU_SYNC which allows us to
avoid invoking cache line invalidation if the driver will just handle it
via a sync_for_cpu or sync_for_device call.
Cc: "David S. Miller" <davem@davemloft.net>
Cc: sparclinux@vger.kernel.org
Signed-off-by: Alexander Duyck <redacted>
This is fine for avoiding the flush for performance reasons, but the
chip isn't going to write anything back unless the device wrote into
the area.
That is mostly what I am doing here. The original implementation was
mostly for performance. I am trying to take the attribute that was
already in place for ARM and apply it to all the other architectures.
So what will be happening now is that we call the map function with
this attribute set and then use the sync functions to map it to the
device and then pull the mapping later.
The idea is that if Jesper does his page pool stuff it would be
calling the map/unmap functions and then the drivers would be doing
the sync_for_cpu/sync_for_device. I want to make sure the map is
cheap and we will have to call sync_for_cpu from the drivers anyway
since there is no guarantee if we will have a new page or be reusing
an existing one.
- Alex
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Konrad Rzeszutek Wilk <hidden> Date: 2016-10-25 01:22:55
quoted
This too. Why can't that be part of the existing code that was there?
Once again it was a formatting thing. I was indented too far and
adding the attribute pushed me over 80 characters so I broke it out to
a label to avoid the problem.
Aah. It is OK to go over the 80 characters. I am not that nit picky.
From: Mark Salter <hidden> Date: 2016-10-28 14:59:04
On Mon, 2016-10-24 at 08:05 -0400, Alexander Duyck wrote:
This change allows us to pass DMA_ATTR_SKIP_CPU_SYNC which allows us to
avoid invoking cache line invalidation if the driver will just handle it
later via a sync_for_cpu or sync_for_device call.
Cc: Mark Salter <redacted>
Cc: Aurelien Jacquiot <redacted>
Cc: linux-c6x-dev@linux-c6x.org
Signed-off-by: Alexander Duyck <redacted>
---
From: Russell King - ARM Linux <linux@armlinux.org.uk> Date: 2016-10-31 10:21:12
On Mon, Oct 24, 2016 at 08:04:47AM -0400, Alexander Duyck wrote:
The use of DMA_ATTR_SKIP_CPU_SYNC was not consistent across all of the DMA
APIs in the arch/arm folder. This change is meant to correct that so that
we get consistent behavior.
I'm really not convinced that this is anywhere close to correct behaviour.
If we're DMA-ing to a buffer, and we unmap it or sync_for_cpu, then we
will want to access the DMA'd data - especially in the sync_for_cpu case,
it's pointless to call sync_for_cpu if we're not going to access the
data.
So the idea of skipping the CPU copy when DMA_ATTR_SKIP_CPU_SYNC is set
seems to be completely wrong - it means we end up reading the stale data
that was in the buffer, completely ignoring whatever was DMA'd to it.
What's the use case for DMA_ATTR_SKIP_CPU_SYNC ?
--
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Alexander Duyck <hidden> Date: 2016-10-31 15:26:36
On Mon, Oct 31, 2016 at 3:20 AM, Russell King - ARM Linux
[off-list ref] wrote:
On Mon, Oct 24, 2016 at 08:04:47AM -0400, Alexander Duyck wrote:
quoted
The use of DMA_ATTR_SKIP_CPU_SYNC was not consistent across all of the DMA
APIs in the arch/arm folder. This change is meant to correct that so that
we get consistent behavior.
I'm really not convinced that this is anywhere close to correct behaviour.
If we're DMA-ing to a buffer, and we unmap it or sync_for_cpu, then we
will want to access the DMA'd data - especially in the sync_for_cpu case,
it's pointless to call sync_for_cpu if we're not going to access the
data.
First, let me clarify. The sync_for_cpu call will still work the
same. This only effects the map/unmap calls.
So the idea of skipping the CPU copy when DMA_ATTR_SKIP_CPU_SYNC is set
seems to be completely wrong - it means we end up reading the stale data
that was in the buffer, completely ignoring whatever was DMA'd to it.
I agree. However this is meant to be used in the dma_unmap call only
if sync_for_cpu has already been called for the regions that could
have been updated by the device.
What's the use case for DMA_ATTR_SKIP_CPU_SYNC ?
The main use case I have in mind is to allow for pseudo-static DMA
mappings where we can share them between the network stack and the
device driver. I use igb as an example.
1 allocate page, reset page_offset to 0
2 map page while passing DMA_ATTR_SKIP_CPU_SYNC
3 dma_sync_single_range_for_device starting at page_offset, length
2K (largest possible write by device)
4 device performs Rx DMA and updates Rx descriptor
5 read length from Rx descriptor
6 dma_sync_single_range_for_cpu starting at page_offset, length
reported by descriptor
7 if page_count == 1
7.1 update page_offset with xor 2K
7.2 hand page up to network stack
7.3 goto 3
8 unmap page with DMA_ATTR_SKIP_CPU_SYNC
9 hand page up to network stack
10 goto 1
The idea is we want to be able to have a page be accessible to the
device, but be able to share it with the network stack which might try
to write to the page. By letting the driver handle the
synchronization we get two main advantages. First we end up looping
over fewer cache lines as we only have to invalidate the region
updated by the device in steps 3 and 6 instead of the entire page.
The other advantage is that the pages are writable by the network
stack since step 8 will not invalidate the entire mapping.
I am just as concerned about the possibility of stale data. That is
why I have gone through and made sure that any path in the igb driver
called sync for the region held by the device before we might call
unmap. It isn't that I don't want the data to be kept fresh, it is a
matter of wanting control over what we are invalidating. Here is a
link to the igb patch I have that was a part of this set.
https://patchwork.ozlabs.org/patch/686747/
Thanks.
- Alex
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>