Hi Jean-Philippe,

On Fri, Jan 06, 2017 at 05:48:33PM +0000, Jean-Philippe Brucker wrote:

On 20/12/16 15:14, Will Deacon wrote:

quoted

Booting Linux on an ARM fastmodel containing an SMMU emulation results
in an unexpected I/O page fault from the legacy virtio-blk PCI device:

[    1.211721] arm-smmu-v3 2b400000.smmu: event 0x10 received:
[    1.211800] arm-smmu-v3 2b400000.smmu:	0x00000000fffff010
[    1.211880] arm-smmu-v3 2b400000.smmu:	0x0000020800000000
[    1.211959] arm-smmu-v3 2b400000.smmu:	0x00000008fa081002
[    1.212075] arm-smmu-v3 2b400000.smmu:	0x0000000000000000
[    1.212155] arm-smmu-v3 2b400000.smmu: event 0x10 received:
[    1.212234] arm-smmu-v3 2b400000.smmu:	0x00000000fffff010
[    1.212314] arm-smmu-v3 2b400000.smmu:	0x0000020800000000
[    1.212394] arm-smmu-v3 2b400000.smmu:	0x00000008fa081000
[    1.212471] arm-smmu-v3 2b400000.smmu:	0x0000000000000000

<system hangs failing to read partition table>

This is because the virtio-blk is behind an SMMU, so we have consequently
swizzled its DMA ops and configured the SMMU to translate accesses. This
then requires the vring code to use the DMA API to establish translations,
otherwise all transactions will result in fatal faults and termination.

Given that ARM-based systems only see an SMMU if one is really present
(the topology is all described by firmware tables such as device-tree or
IORT), then we can safely use the DMA API for all virtio devices.

There is a problem with the platform block device on that same model.
Since it's not behind the SMMU, the DMA ops fall back to swiotlb, which
limits the number of mappings.

It used to work with 4.9, but since 9491ae4 ("mm: don't cap request size
based on read-ahead setting") unlocked read-ahead, we quickly run into
the limit of swiotlb and panic:

[    5.382359] virtio-mmio 1c130000.virtio_block: swiotlb buffer is full
(sz: 491520 bytes)
[    5.382452] virtio-mmio 1c130000.virtio_block: DMA: Out of SW-IOMMU
space for 491520 bytes
[    5.382531] Kernel panic - not syncing: DMA: Random memory could be
DMA written
...
[    5.383148] [<ffff0000083ad754>] swiotlb_map_page+0x194/0x1a0
[    5.383226] [<ffff000008096bb8>] __swiotlb_map_page+0x20/0x88
[    5.383320] [<ffff0000084bf738>] vring_map_one_sg.isra.1+0x70/0x88
[    5.383417] [<ffff0000084c04fc>] virtqueue_add_sgs+0x2ec/0x4e8
[    5.383505] [<ffff00000856d99c>] __virtblk_add_req+0x9c/0x1a8
...
[    5.384449] [<ffff0000081829c4>] ondemand_readahead+0xfc/0x2b8

Oh, lovely!

Commit 9491ae4 caps the read-ahead request to a limit set by the backing
device. For virtio-blk, it is infinite (as set by the call to
blk_queue_max_hw_sectors in virtblk_probe).

I'm not sure how to fix this. Setting an arbitrary sector limit in the
virtio-blk driver seems unfair to other users. Maybe we should check if
the device is behind a hardware IOMMU before using the DMA API?

Couldn't the same issue potentially occur with a hardware IOMMU, where
we run out of IOVA space due to unlimited readahead? I think it might be
best to enforce a finite limit for virtio devices when the DMA API is in
use.

Do any drivers for physical (i.e. non-virtual) hardware make use of
unlimited readahead?

Will