On 04/08/2023 14:19, Shengjiu Wang wrote:

On Wed, Aug 2, 2023 at 8:28 PM Hans Verkuil [off-list ref] wrote:

quoted

On 02/08/2023 14:02, Shengjiu Wang wrote:

quoted

On Wed, Aug 2, 2023 at 7:22 PM Takashi Iwai [off-list ref] wrote:

quoted

On Wed, 02 Aug 2023 09:32:37 +0200,
Hans Verkuil wrote:

quoted

Hi all,

On 25/07/2023 08:12, Shengjiu Wang wrote:

quoted

Audio signal processing has the requirement for memory to
memory similar as Video.

This patch is to add this support in v4l2 framework, defined
new buffer type V4L2_BUF_TYPE_AUDIO_CAPTURE and
V4L2_BUF_TYPE_AUDIO_OUTPUT, defined new format v4l2_audio_format
for audio case usage.

The created audio device is named "/dev/audioX".

And add memory to memory support for two kinds of i.MX ASRC
module

Before I spend time on this: are the audio maintainers OK with doing
this in V4L2?

I do want to have a clear statement on this as it is not something I
can decide.

Well, I personally don't mind to have some audio capability in v4l2
layer.  But, the only uncertain thing for now is whether this is a
must-have or not.

Thanks,  I am also not sure about this.  I am also confused that why
there is no m2m implementation for audio in the kernel.  Audio also
has similar decoder encoder post-processing as video.

quoted

IIRC, the implementation in the sound driver side was never done just
because there was no similar implementation?  If so, and if the
extension to the v4l2 core layer is needed, shouldn't it be more
considered for the possible other route?

Actually I'd like someone could point me to the other route. I'd like to
try.

The reason why I select to extend v4l2 for such audio usage is that v4l2
looks best for this audio m2m implementation.  v4l2 is designed for m2m
usage.  if we need implement another 'route',  I don't think it can do better
that v4l2.

I appreciate that someone can share his ideas or doable solutions.
And please don't ignore my request, ignore my patch.

To give a bit more background: if it is decided to use the v4l API for this
(and I have no objection to this from my side since API/framework-wise it is a
good fit for this), then there are a number of things that need to be done to
get this into the media subsystem:

- documentation for the new uAPI
- add support for this to v4l2-ctl
- add v4l2-compliance tests for the new device
- highly desirable: have a virtual driver (similar to vim2m) that supports this:
  it could be as simple as just copy input to output. This helps regression
  testing.
- it might need media controller support as well. TBD.

None of this is particularly complex, but taken all together it is a fair
amount of work that also needs a lot of review time from our side.

I want to add one more option to the mix: drivers/media/core/v4l2-mem2mem.c is
the main m2m framework, but it relies heavily on the videobuf2 framework for
the capture and output queues.

The core vb2 implementation in drivers/media/common/videobuf2/videobuf2-core.c
is independent of V4L2 and can be used by other subsystems (in our case, it is
also used by the DVB API). It is a possibility to create an alsa version of
v4l2-mem2mem.c that uses the core vb2 code with an ALSA uAPI on top.

So in drivers/media/common/videobuf2/ you would have a videobuf2-alsa.c besides
the already existing videobuf2-v4l2.c and -dvb.c.

Perhaps parts of v4l2-mem2mem.c can be reused as well in that case, but I am
not sure if it is worth the effort. I suspect copying it to an alsa-mem2mem.c
and adapting it for alsa is easiest if you want to go that way.

Thanks.

Does this means that videobuf2-v4l2.c and v4l2-mem2mem.c are dedicate
for video device? if audio want to use v4l2 framework,  need to create
videobuf2-alsa.c and alsa-mem2mem.c, but it may cause a lot of function
duplicate.

The videobuf2-v4l2.c sits on top of videobuf2-core.c and provides the V4L2
uAPI for the streaming functionality. If you don't want to use the V4L2
uAPI for this, then you would need a videobuf2-alsa.c that provides a
(possibly new) ALSA uAPI. Whether that makes sense is something I cannot
decide.

v4l2-mem2mem.c uses videobuf2-v4l2.c, so if you need a ALSA version, then
you probably need to create an alsa-mem2mem.c (possibly some functionality
can be shared).

It's just a third option, and it can be useful if there is a strong desire
to keep the uAPI for this functionality entirely within the ALSA subsystem,
but you want to reuse the streaming I/O functionality that the videobuf2
core provides.

If the decision is that it is fine to use the V4L2 uAPI for this type
of audio functionality through a /dev/v4l-audioX device, then just ignore
this option and use V4L2.

Regards,

	Hans