[PATCH/RFC v2.1 0/2] Mem-to-mem device framework

-----Original Message-----
From: Pawel Osciak [mailto:p.osciak at samsung.com]
Sent: Monday, December 28, 2009 8:19 PM
To: 'Hans Verkuil'
Cc: linux-media at vger.kernel.org; linux-samsung-soc at vger.kernel.org;
linux-arm-kernel at lists.infradead.org; Marek Szyprowski;
kyungmin.park at samsung.com; Hiremath, Vaibhav; Karicheri,
Muralidharan; 'Guru Raj'; 'Xiaolin Zhang'; 'Magnus Damm'; 'Sakari
Ailus'
Subject: RE: [PATCH/RFC v2.1 0/2] Mem-to-mem device framework

Hello Hans,


On Wednesday 23 December 2009 16:06:18 Hans Verkuil wrote:

quoted

Thank you for working on this! It's much appreciated. Now I've

noticed that

quoted

patches regarding memory-to-memory and memory pool tend to get

very few comments.

quoted

I suspect that the main reason is that these are SoC-specific

features that do

quoted

not occur in consumer-type products. So most v4l developers do not

have the

quoted

interest and motivation (and time!) to look into this.

Thank you very much for your response. We were a bit surprised with
the lack of
responses as there seemed to be a good number of people interested
in this area.

I'm hoping that everybody interested would take a look at the test
device posted
along with the patches. It's virtual, no specific hardware required,
but it
demonstrates the concepts behind the framework, including
transactions.

[Hiremath, Vaibhav] I was on vacation and resumed today itself, I will go through these patch series this weekend and will get back to you.

I just had cursory look and I would say it should be really good starting point for us to support mem-to-mem devices.

Thanks,
Vaibhav

quoted

One thing that I am missing is a high-level overview of what we

want. Currently

quoted

there are patches/RFCs floating around for memory-to-memory

support, multiplanar

quoted

support and memory-pool support.

What I would like to see is a RFC that ties this all together from

the point of

quoted

view of the public API. I.e. what are the requirements? Possibly

solutions? Open

quoted

questions? Forget about how to implement it for the moment, that

will follow

quoted

from the chosen solutions.

Yes, that's true, sorry about that. We've been so into it after the
memory pool
discussion and the V4L2 mini-summit that I neglected describing the
big picture
behind this.

So to give a more high-level description, from the point of view of
applications
and the V4L2 API:

---------------
Requirements:
---------------
(Some of the following were first posted by Laurent in:
http://thread.gmane.org/gmane.linux.drivers.video-input-
infrastructure/10204).

1. Support for devices that take input data in a source buffer, take
a separate
destination buffer, process the source data and put it in the
destination buffer.

2. Allow sharing buffers between devices, effectively chaining them
to form
video pipelines. An example of this could be a video decoder, fed
with video
stream which returns raw frames, which then have to be postprocessed
by another
device and displayed. This is the main scenario we need to have for
our S3C/S5P
series SoCs. Of course, we'd like zero-copy.

3. Allow using more than one buffer by the device at the same time.
This is not
supported by videobuffer (e.g. we have to choose on which buffer
we'd like
to sleep, and we do not always know that). This is not really a
requirement
from the V4L2 API point of view, but has direct influence on how
poll() and
blocking I/O works.

4. Multiplanar buffers. Our devices require them (see the RFC for
more details:
http://article.gmane.org/gmane.linux.drivers.video-input-
infrastructure/11212).

5. Solve problems with cache coherency on non-x86 architectures,
especially in
videobuf for OUTPUT buffers. We need to flush the cache before
starting the
transaction.

6. Reduce buffer queuing latency, e.g.: move operations such as
locking, out
of qbuf.
Applications would like to queue a buffer and be able to fire up the
device
as fast as possible.

7. Large buffer allocations, buffer preallocation, etc.


---------------
Solutions:
---------------
1. After a detailed discussion, we agreed in:
http://thread.gmane.org/gmane.linux.drivers.video-input-
infrastructure/10668,
that we'd like the application to be able to queue/dequeue both
OUTPUT (as source)
and CAPTURE (as destination) buffers on one video node. Activating
the device
(after streamon) would take effect only if there are both types of
buffers
available. The application would put source data into OUTPUT buffers
and expect
to find it processed in dequeued CAPTURE buffers. Addressed by
mem2mem framework.

2. I don't see anything to do here from the API's point of view. The
application
would open two video nodes, e.g. video decoder and video
postprocessor and queue
buffers dequeued from decoder on the postprocessor. To get the best
performance,
this requires the buffers to be marked as non cached somehow to
avoid unneeded
cache syncs.

3. Mem2mem addresses this partially by adding a "transaction"
concept. It's
not bullet-proof though, as it assumes the buffers will be returned
in the same
order as passed. Some videobuffer limitations will have to be
addressed here.

4. See my RFC. Patches in progress.

5. We have narrowed it down to an additional sync() before the
operation
(i.e. in qbuf), but more issues may exist here. I have already added
sync()
support for qbuf with minimal changes to videobuf and will be
posting the
proposal soon. This also requires identifying the direction of the
sync, but
we have found a way to do this without adding anything new (videobuf
flags
are enough).

6. Later. We haven't done anything in this field.

7. We use our own allocator (see
http://thread.gmane.org/gmane.linux.ports.arm.kernel/56879), but we
have a new
concept for that which we'd like to discuss separately later.

quoted

Note that I would suggest though that the memory-pool part is

split into two

quoted

parts: how to actually allocate the memory is pretty much separate

from how v4l

quoted

will use it. The actual allocation part is probably quite complex

and might

quoted

even be hardware dependent and should be discussed separately. But

how to use

quoted

it is something that can be discussed without needing to know how

it was

quoted

allocated.

Exactly, this is the approach we have assumed right now. We'd like
to introduce
each part of the infrastructure incrementally. The plan was to do
exactly as
you said: leave the allocator-specific parts for later and for a
separate
discussion.
We intend to follow with multi-planar buffers and then to focus on
dma-contig,
as this is what our hardware requires.

quoted

BTW, what is the status of the multiplanar RFC? I later realized

that that RFC

quoted

might be very useful for adding meta-data to buffers. There are

several cases

quoted

where that is useful: sensors that provide meta-data when

capturing a frame and

quoted

imagepipelines (particularly in memory-to-memory cases) that want

to have all

quoted

parameters as part of the meta-data associated with the image.

There may well

quoted

be more of those.

This got pushed back but now after m2m, it's become next task on my
list. I
expect to be posting patches in a week or two, hopefully.
I understand that you'd like to make the pointer in the union and
the helper
struct more generic to use it to pass different types of
information?


Best regards
--
Pawel Osciak
Linux Platform Group
Samsung Poland R&D Center