Re: arm-smmu-v3 sharing SID

On Thu, May 18, 2023 at 11:44:56AM +0000, Peng Fan wrote:

quoted

Subject: Re: arm-smmu-v3 sharing SID

On 2023-05-18 11:46, Peng Fan wrote:

quoted

Hi all,

Current arm-smmu-v3 driver does not support sharing SID, If there are
two devices sharing one SID, the smmu-v3 driver will report "stream x
already in tree".

We have an SOC that using smmu-v3, only supports limited SIDs. From my
understanding, the smmu-v3 hardware supports SID sharing, but linux
driver not support that.

Ugh, we've always hoped that nobody would build such a thing, since
SMMUv3 allows for plenty of stream IDs for any reasonable system (Arm's
implementations support at least 24 bits), and aliasing at the StreamID level
does rather compromise the usefulness.

We only support max 64 SIDs. So I am thinking to share SID, since we
have lots of dma channels.

quoted

quoted

I would like know is it ok to add support for sharing SID?
Something like to use common rb_add/find, not
smmu-v3 driver local rb tree add/find.

Not sure what you're thinking there - the StreamID tree is still private to the
SMMU instance either way, and the existing arm_smmu_find_master()
function is ideal for arm_smmu_device_group() to detect aliasing devices
and group them appropriately. Then you also need some way to skip
updating STEs that have already been touched for a previous device in the
group (basically generalising what arm_smmu_install_ste_for_dev()
currently does for duplicate StreamIDs owned by one device).

Thanks for sharing insights. BTW, would you plan to add the support?

quoted

I'm fairly confident in reasoning about this for basic .attach_dev purposes,
since I did implement the equivalent for SMMUv2, but I'm not sure I even
want to think about what it would mean for SVA, PASIDs and nesting...

We should disable all these things for multi-device groups, it's not worth
the headache. I think we used to but I can't find it anywhere. That said,
I guess only PCIe PRI (not upstream) absolutely won't work with multiple
devices sharing a SID, since page responses are routed back by RID to the
endpoint.

Stall and by extension SVA can't really be supported with multiple devices
per SID either, but that's mainly a software complexity issue (we'd need
to guess which device it comes from, or report the fault for all devices).
PASID and nesting might work transparently due to iommu_group, though I'd
rather not think about that either.

It may boil down to updating the smmu->streams structure to support
multiple masters per stream (maybe simplify the driver first by moving to
a xarray [1]). That would provide a refcount for each SID and allow to
only write the STE once on setup and teardown. Since arm_smmu_find_master()
is only used to handle I/O page faults (stall/PRI which we won't support
in this case), it could return NULL for multi-master streams.

Thanks,
Jean

[1] https://lore.kernel.org/linux-iommu/ecb3725c-27c4-944b-b42c-f4e293521f94@arm.com/ (local)

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel