Hi Mauro,

On 07/29/2011 06:02 AM, Mauro Carvalho Chehab wrote:

Em 28-07-2011 19:57, Sylwester Nawrocki escreveu:

quoted

On 07/28/2011 03:20 PM, Mauro Carvalho Chehab wrote:

quoted

quoted

Accumulating sub-dev controls at the video node is the right thing to do.

An MC-aware application will need to handle with that, but that doesn't sound to
be hard. All such application would need to do is to first probe the subdev controls,
and, when parsing the videodev controls, not register controls with duplicated ID's,
or to mark them with some special attribute.

IMHO it's not a big issue in general. Still, both subdev and the host device may
support same control id. And then even though the control ids are same on the subdev
and the host they could mean physically different controls (since when registering
a subdev at the host driver the host's controls take precedence and doubling subdev
controls are skipped).

True, but, except for specific usecases, the host control is enough.

For some ones those "specific use cases" could be most important (or all) the use 
cases. :-)

quoted

Also there might be some preference at user space, at which stage of the pipeline
to apply some controls. This is where the subdev API helps, and plain video node
API does not.

Again, this is for specific usecases. On such cases, what is expected is that the more
generic control will be exported via V4L2 API.

IMHO it's not about use cases, just about which hardware can be covered by the API
and which can not. When the signal processing systems are becoming distributed and
more interactively reconfigurable it's getting impossible to reasonably control them
through a centralized interface.

quoted

quoted

quoted

Thus it's a bit hard to imagine that we could do something like "optionally
not to inherit controls" as the subdev/MC API is optional. :)

This was actually implemented. There are some cases at ivtv/cx18 driver where both
the bridge and a subdev provides the same control (audio volume, for example). The
idea is to allow the bridge driver to touch at the subdev control without exposing
it to userspace, since the desire was that the bridge driver itself would expose
such control, using a logic that combines changing the subdev and the bridge registers
for volume.

This seem like hard coding a policy in the driver;) Then there is no way (it might not
be worth the effort though) to play with volume level at both devices, e.g. to obtain
optimal S/N ratio.

In general, playing with just one control is enough. Andy had a different opinion
when this issue were discussed, and he thinks that playing with both is better.
At the end, this is a developers decision, depending on how much information
(and bug reports) he had.

quoted

This is a hack...sorry, just joking ;-) Seriously, I think the
situation with the userspace subdevs is a bit different. Because with one API we
directly expose some functionality for applications, with other we code it in the
kernel, to make the devices appear uniform at user space.

Not sure if I understood you. V4L2 export drivers functionality to userspace in an
uniform way. MC api is for special applications that might need to access some
internal functions on embedded devices.

Yes, that's what I meant.

Of course, there are some cases where it doesn't make sense to export a subdev control
via V4L2 API.

quoted

quoted

quoted

Also, the sensor subdev can be configured in the video node driver as well as
through the subdev device node. Both APIs can do the same thing but in order
to let the subdev API work as expected the video node driver must be forbidden
to configure the subdev.

Why? For the sensor, a V4L2 API call will look just like a bridge driver call.
The subdev will need a mutex anyway, as two MC applications may be opening it
simultaneously. I can't see why it should forbid changing the control from the
bridge driver call.

Please do not forget there might be more than one subdev to configure and that
the bridge itself is also a subdev (which exposes a scaler interface, for instance).
A situation pretty much like in Figure 4.4 [1] (after the scaler there is also
a video node to configure, but we may assume that pixel resolution at the scaler
pad 1 is same as at the video node). Assuming the format and crop configuration
flow is from sensor to host scaler direction, if we have tried to configure _all_
subdevs when the last stage of the pipeline is configured (i.e. video node)
the whole scaler and crop/composition configuration we have been destroyed at
that time. And there is more to configure than VIDIOC_S_FMT can do.

Think from users perspective: all user wants is to see a video of a given resolution.
S_FMT (and a few other VIDIOC_* calls) have everything that the user wants: the
desired resolution, framerate and format.

Specialized applications indeed need more, in order to get the best images for
certain types of usages. So, MC is there.

Such applications will probably need to know exactly what's the sensor, what are
their bugs, how it is connected, what are the DSP blocks in the patch, how the
DSP algorithms are implemented, etc, in order to obtain the the perfect image.

I suspect we might need more the specialized, H/W aware libraries rather than 
applications.

It's also about efficient hardware utilisation. It's often important to get most
out of the device, i.e. be able to precisely configure all the bits and pieces,
depending on the use case. This could be done by having as thin kernel driver
as possible and a H/W dedicated library.
 

Even on embedded devices like smartphones and tablets, I predict that both
types of applications will be developed and used: people may use a generic
application like flash player, and an specialized application provided by
the manufacturer. Users can even develop their own applications generic
apps using V4L2 directly, at the devices that allow that.

As I said before: both application types are welcome. We just need to warrant
that a pure V4L application will work reasonably well.

Is Laurent pointed out, more complex devices could be supported through libv4l2.

quoted

Allowing the bridge driver to configure subdevs at all times would prevent
the subdev/MC API to work.

Well, then we need to think on an alternative for that. It seems an interesting
theme for the media workshop at the Kernel Summit/2011.

Sure. But isn't it that simple as letting the driver implement MC API only and
abstract the hardware details in dedicated library/libv4l2 plugin ?
I just don't see a point in moving the adaptation layer into the kernel.

quoted

quoted

quoted

There is a conflict there that in order to use
'optional' API the 'main' API behaviour must be affected....

It is optional from userspace perspective. A V4L2-only application should be able
to work with all drivers. However, a MC-aware application will likely be specific
for some hardware, as it will need to know some device-specific stuff.

Both kinds of applications are welcome, but dropping support for V4L2-only applications
is the wrong thing to do.

quoted

And I really cant use V4L2 API only as is because it's too limited.

Why?

For instance there is really yet no support for scaler and composition onto
a target buffer in the Video Capture Interface (we also use sensors with
built in scalers). It's difficult to efficiently manage capture/preview pipelines.
It is impossible to discover the system topology.

Scaler were always supported by V4L2: if the resolution specified by S_FMT is not
what the sensor provides, then scale. All non-embedded drivers with sensor or bridge
supports scale does that.

This is not really a level of support I have been thinking about. I thought about user
space knowing what exactly is scaled and where.

Composition is not properly supported yet. It could make sense to add it to V4L. How do you
think MC API would help with composite?

Yes, the new S_SELECTION ioctl replacing S_CROP is meant to add composition support.
With MC you can support composition through cropping on a source pad, however the ioctl 
naming might yet need to be revisited.

Managing capture/preview pipelines will require some support at V4L2 level. This is
a problem that needs to be addressed there anyway, as buffers for preview/capture
need to be allocated. There's an RFC about that, but I don't think it covers the
pipelines for it.

It requires fairly complex hardware configuration in hardware assisted cases.
So it's better done at MC API level. Nevertheless there is much that can be improved
at V4L2 level too.

Discovering the system topology indeed is not part of V4L2 API and will never be.
This is MC API business. There's no overlap with V4L2.

True, however I was referring to the V4L2 API (video node) _only_.

quoted

quoted

quoted

Might be that's why I see more and more often migration to OpenMAX recently.

I don't think so. People may be adopting OpenMAX just because of some marketing strategy
from the OpenMAX forum. We don't spend money to announce V4L2 ;)

:)

quoted

I think that writing a pure OpenMAX driver is the wrong thing to do, as, at the long
term, it will cost _a_lot_ for the vendors to maintain something that will never be
merged upstream.

In general it depends on priorities. For some chip vendors it might be more important
to provide a solution in short time frame. Getting something in mainline isn't effortless
and spending lot's of time on this for some parties is unacceptable.

Adding something at a forum like OpenMAX is probably not an easy task ;) It generally
takes a long time to change something on open forum specifications. Also, participating
on those forums require lots of money, with membership and travel expenses.

Adding a new driver that follows the API's is not a long-term effort. The delay is
basically one kernel cycle, e. g. about 3-4 months.

Most of the delays on merging drivers for embedded systems, however, take a
way longer than that, unfortunately. From what I see, what is delaying driver
submission is due to:
	- the lack of how to work with the Linux community. Some developers take a
long time to get the 'modus operandi';
	- the need of API changes. It is still generally faster to add a new API
addition at the Kernel than on an open forum;
	- the discussions inside the various teams (generally from the same company,
or the company and their customers) about the better way to implement some feature.

All the above also happens when developing an OpenMAX driver: companies need to
learn how to work with the OpenMax forums, API changes may be required, so they
need to participate at the forums, the several internal teams and customers
will be discussing the requirements.

I bet that there's also one additional step: to submit the driver to some company
that will check the driver compliance with the official API. Such certification
process is generally expensive and takes a long time.

At the end of the day, they'll also spend a lot of time to have the driver done,
or they'll end by releasing a "not-quite-openmax" driver, and then needing to
rework on that, due to customers complains, or to certification-compliance,
loosing time and money.

From what I can see many of the big players in the graphics devices world are already

a _contributor_ members of the Khronos Group. And the membership costs are probably
a piece of cake for such a big companies.

OMX is different from V4L2 in that there is no need to discuss every trivial detail
in the API, specific for particular H/W. Adopters just add proprietary extensions
when needed, not every detail is conservatively fixed in the API.
This can't really be done in video4linux. But we also shouldn't be forcing users
to use an API that limits H/W capabilities, _if_ we are interested in supporting
modern complex devices in V4L2.

--
Regards,
Sylwester