Re: IIO, dmabuf, io_uring

Hi Paul,

Am 13.08.21 um 13:41 schrieb Paul Cercueil:

Hi,

A few months ago we (ADI) tried to upstream the interface we use with 
our high-speed ADCs and DACs. It is a system with custom ioctls on the 
iio device node to dequeue and enqueue buffers (allocated with 
dma_alloc_coherent), that can then be mmap'd by userspace 
applications. Anyway, it was ultimately denied entry [1]; this API was 
okay in ~2014 when it was designed but it feels like re-inventing the 
wheel in 2021.

Back to the drawing table, and we'd like to design something that we 
can actually upstream. This high-speed interface looks awfully similar 
to DMABUF, so we may try to implement a DMABUF interface for IIO, 
unless someone has a better idea.

Yeah, that sounds a lot like a DMABUF use case.

Our first usecase is, we want userspace applications to be able to 
dequeue buffers of samples (from ADCs), and/or enqueue buffers of 
samples (for DACs), and to be able to manipulate them (mmapped 
buffers). With a DMABUF interface, I guess the userspace application 
would dequeue a dma buffer from the driver, mmap it, read/write the 
data, unmap it, then enqueue it to the IIO driver again so that it can 
be disposed of. Does that sound sane?

Well it's pretty close. Doing the map/unmap dance all the time is 
usually a bad idea since flushing the CPU TLB all the time totally kills 
your performance.

What you do instead is to implement the CPU synchronize callbacks in 
your DMA-BUF implementation and flush caches as necessary.

Our second usecase is - and that's where things get tricky - to be 
able to stream the samples to another computer for processing, over 
Ethernet or USB. Our typical setup is a high-speed ADC/DAC on a dev 
board with a FPGA and a weak soft-core or low-power CPU; processing 
the data in-situ is not an option. Copying the data from one buffer to 
another is not an option either (way too slow), so we absolutely want 
zero-copy.

Usual userspace zero-copy techniques (vmsplice+splice, MSG_ZEROCOPY 
etc) don't really work with mmapped kernel buffers allocated for DMA 
[2] and/or have a huge overhead, so the way I see it, we would also 
need DMABUF support in both the Ethernet stack and USB (functionfs) 
stack. However, as far as I understood, DMABUF is mostly a DRM/V4L2 
thing, so I am really not sure we have the right idea here.

Well two possibilities here: Either implement DMA-BUF support in the 
Ethernet/USB subsystem or get splice working efficiently with DMA-BUF 
mappings.

The first one is certainly a lot of work and no idea if the second is 
even doable and if yes also in a non-hacky way which you can get upstream.

And finally, there is the new kid in town, io_uring. I am not very 
literate about the topic, but it does not seem to be able to handle 
DMA buffers (yet?). The idea that we could dequeue a buffer of samples 
from the IIO device and send it over the network in one single syscall 
is appealing, though.

As far as I know this is orthogonal to DMA-BUF. Christoph's answer 
sounds like my understand is correct, but there are certainly people 
which know that better than I do.

Regards,
Christian.

Any thoughts? Feedback would be greatly appreciated.

Cheers,
-Paul

[1]: 
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.kernel.org%2Flinux-iio%2F20210217073638.21681-1-alexandru.ardelean%40analog.com%2FT%2F%23m6b853addb77959c55e078fbb06828db33d4bf3d7&data=04%7C01%7Cchristian.koenig%40amd.com%7C2c62025e34b644b98e2508d95e4f4dcb%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637644516997743314%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=vZfslxljjWcXi1RccZcsnKTD8x1CixRN%2Ftk4FMsWN3U%3D&reserved=0
[2]: 
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnewbedev.com%2Fzero-copy-user-space-tcp-send-of-dma-mmap-coherent-mapped-memory&data=04%7C01%7Cchristian.koenig%40amd.com%7C2c62025e34b644b98e2508d95e4f4dcb%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637644516997753306%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Fn%2B3dO%2B%2F3r0ZpC5oKsQaPN2DREZKVWdVPahYgt2bsSw%3D&reserved=0