[MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

Early in my Linux career there was palpable concern around Linux being
locked out of future computing platforms by hardware vendors who did not
provide open drivers, or even documentation for their hardware. For the
hardware vendors that did participate upstream, maintainers used code
acceptance to influence them towards common Linux commands and
cross-vendor cooperation.

The internalized lesson from those days was: "Be wary of vendors pushing
'do anything you want and get away with it' passthrough tunnels. Demand
open documentation of all interfaces."

Present day realities and discussions merit revisiting that lesson:

1/ The truth of the matter is that until the Kernel Lockdown facility
   arrived, device vendors *had* an unfettered passthrough tunnel via
   userspace driver mechanisms like /dev/mem and pci-sysfs. The presence of
   those facilities did not appear to injure the ascension of Linux.

2/ Device passthrough, kernel passing opaque payloads, is already taken
   for granted in many subsystems. USB and HID have "raw" interfaces, EFI
   variables provide platform-specific configuration, and the oft-cited
   examples of SCSI and NVME that provide facilities to marshal any command
   payload whether mainline maintainers think the functionality is a good
   idea or not. In the case of NVME, the specification continues to evolve
   despite this Linux bypass.

3/ The practice of requiring Linux commands to wrap all device commands
   does not appear to have accelerated upstream participation in the CXL
   subsystem. I.e. CXL, in contrast to NVME, relegates passthrough to a
   build-time debug option. Some vendors are even shipping vendor
   specific firmware update facilities even though mainline has support for
   the CXL standard firmware update mechanism.

   With the impending arrival of CXL switch devices wanting to share
   mailbox handling code with the CXL core, the prohibition of
   device-specific commands is going to generate significant upstream work
   to wrap all that in Linux commands with little perceivable long term
   benefit to the subsystem.

CXL and RDMA are also foreshadowing conflicts across subsystems. It is
not difficult to imagine a future CXL or RDMA device that supports mem,
block, net, and drm/accel functionality. Which subsystem's
device-command policy applies to such a thing?

Enter the fwctl proposal [1]. From the CXL subsystem perspective it
looks like a long-term solution to the problem of managing expectations
between hardware vendors and mainline subsystems. It disclaims support
for the fast-path (data-plane) and is targeted at the long tail of
slow-path (config/debug plane) device-specific operations that are often
uninteresting to mainline. It sets expectations that the device must
advertise the effect of all commands so that the kernel can deploy
reasonable Kernel Lockdown policy, or otherwise require CAP_SYS_RAWIO
for commands that may affect user-data. It sets common expectations for
device designers, distribution maintainers, and kernel developers. It is
complimentary to the Linux-command path for operations that need deeper
kernel coordination.

The upstream discussion has yielded the full spectrum of positions on
device specific functionality, and it is a topic that needs cross-kernel
consensus as hardware increasingly spans cross-subsystem concerns.
Please consider it for a Maintainers Summit discussion.

[1]: http://lore.kernel.org/0-v2-940e479ceba9+3821-fwctl_jgg@nvidia.com

Passthrough in general is useful, but the problem is that protocols
aren't designed with it in mind.  Look at the list of command that
affect too much state and we have to block in SCSI and NVMe.  And
in NVMe I'm fighting a constant fight in the technical working group
to not add new command that instantly disable the access control we've
built into NVMe passthrough.   So IMHO passthrough can be good idea,
but only if the protocol is designed for it, and protocol designer
generally have a hard time how software works at all, never mind
the futuristic concepts of layering, abstraction and access control.

On Mon, Jul 08, 2024 at 03:26:43PM -0700, Dan Williams wrote:

Enter the fwctl proposal [1]. From the CXL subsystem perspective it
looks like a long-term solution to the problem of managing expectations
between hardware vendors and mainline subsystems. It disclaims support
for the fast-path (data-plane) and is targeted at the long tail of
slow-path (config/debug plane) device-specific operations that are often
uninteresting to mainline.

That's not true at all, device-specific operations are very interesting
to mainline, and I would argue that the "slow-path" is the most
important thing for the kernel to manage as that is where the security
and unification layers can be properly enforced.

Vendors that think the control plane should just be allowed to be
accessed by userspace "blindly" are not saying outloud that they just
want to circumvent the security layer entirely like they previously were
doing by directly accessing /dev/mem which is one of the strongest
reasons to keep enforcing this through the kernel as Christoph points
out.

It's the cumulation of multiple vendors of semi-alike config paths that
allow us to standardize them to provide the most important thing of all,
a unified view to userspace where a user does not care about what type
of hardware is running, which is really the goal of Linux as well, but
directly goes against what a vendor wants to have happen.

It sets expectations that the device must
advertise the effect of all commands so that the kernel can deploy
reasonable Kernel Lockdown policy, or otherwise require CAP_SYS_RAWIO
for commands that may affect user-data.

Yes, this is a good start, but it might still be too "vendor-specific"
at this point in time.

It sets common expectations for
device designers, distribution maintainers, and kernel developers. It is
complimentary to the Linux-command path for operations that need deeper
kernel coordination.

Yes, it's a good start, BUT by circumventing the network control plane,
the network driver maintainers rightfully are worried about this as
their review comments seem to be ignored here.  The rest of us
maintainers can't ignore that objection, sorry.

The upstream discussion has yielded the full spectrum of positions on
device specific functionality, and it is a topic that needs cross-kernel
consensus as hardware increasingly spans cross-subsystem concerns.
Please consider it for a Maintainers Summit discussion.

SHould be a fun discussion, thanks for proposing this.

greg k-h

On Tue, Jul 09, 2024 at 12:01:06PM +0200, Greg KH wrote:

On Mon, Jul 08, 2024 at 03:26:43PM -0700, Dan Williams wrote:

<...>

quoted

It sets common expectations for
device designers, distribution maintainers, and kernel developers. It is
complimentary to the Linux-command path for operations that need deeper
kernel coordination.

Yes, it's a good start, BUT by circumventing the network control plane,
the network driver maintainers rightfully are worried about this as
their review comments seem to be ignored here.  The rest of us
maintainers can't ignore that objection, sorry.

Can you please point to the TECHNICAL review comments that were
presented and later ignored?

I don't see any, but I and probably Jonathan who posted in-depth
articles in LWN might have missed them.

Thanks

On Tue, Jul 09, 2024 at 03:25:47PM +0300, Leon Romanovsky wrote:

On Tue, Jul 09, 2024 at 12:01:06PM +0200, Greg KH wrote:

quoted

On Mon, Jul 08, 2024 at 03:26:43PM -0700, Dan Williams wrote:

<...>

quoted

quoted

It sets common expectations for
device designers, distribution maintainers, and kernel developers. It is
complimentary to the Linux-command path for operations that need deeper
kernel coordination.

Yes, it's a good start, BUT by circumventing the network control plane,
the network driver maintainers rightfully are worried about this as
their review comments seem to be ignored here.  The rest of us
maintainers can't ignore that objection, sorry.

Can you please point to the TECHNICAL review comments that were
presented and later ignored?

I can't remember review comments that were made yesterday, let alone
months ago, sorry.

greg k-h

On Tue, Jul 09, 2024 at 02:33:17PM +0200, Greg KH wrote:

On Tue, Jul 09, 2024 at 03:25:47PM +0300, Leon Romanovsky wrote:

quoted

On Tue, Jul 09, 2024 at 12:01:06PM +0200, Greg KH wrote:

quoted

On Mon, Jul 08, 2024 at 03:26:43PM -0700, Dan Williams wrote:

<...>

quoted

quoted

It sets common expectations for
device designers, distribution maintainers, and kernel developers. It is
complimentary to the Linux-command path for operations that need deeper
kernel coordination.

Yes, it's a good start, BUT by circumventing the network control plane,
the network driver maintainers rightfully are worried about this as
their review comments seem to be ignored here.  The rest of us
maintainers can't ignore that objection, sorry.

Can you please point to the TECHNICAL review comments that were
presented and later ignored?

I can't remember review comments that were made yesterday, let alone
months ago, sorry.

So I will summarize the situation for you. There are NO technical review
comments from netdev maintainer (not plural maintainers). The difference
is philosophical and not technical.

And yes, "rest of us maintainers" can ignore philosophical objections.
At the end, Linux kernel is distributed open-source project with
different people who have different opinions.

Thanks

greg k-h

On Mon, 2024-07-08 at 15:26 -0700, Dan Williams wrote:

Early in my Linux career there was palpable concern around Linux
being locked out of future computing platforms by hardware vendors
who did not provide open drivers, or even documentation for their
hardware. For the hardware vendors that did participate upstream,
maintainers used code acceptance to influence them towards common
Linux commands and cross-vendor cooperation.

Firstly could I say "passthrough" seems to be the wrong word here. 
When I see it I think of device pass through from host to VM (SRIOV and
the like), which is becoming the bedrock of the virtualization world. 
However, this proposal seems to be more about user space device drivers
and fat firmware cards.

The internalized lesson from those days was: "Be wary of vendors
pushing 'do anything you want and get away with it' passthrough
tunnels. Demand open documentation of all interfaces."

Present day realities and discussions merit revisiting that lesson:

1/ The truth of the matter is that until the Kernel Lockdown facility
   arrived, device vendors *had* an unfettered passthrough tunnel via
   userspace driver mechanisms like /dev/mem and pci-sysfs. The
presence of
   those facilities did not appear to injure the ascension of Linux.

2/ Device passthrough, kernel passing opaque payloads, is already
taken
   for granted in many subsystems. USB and HID have "raw" interfaces,
EFI
   variables provide platform-specific configuration, and the oft-
cited
   examples of SCSI and NVME that provide facilities to marshal any
command
   payload whether mainline maintainers think the functionality is a
good
   idea or not. In the case of NVME, the specification continues to
evolve
   despite this Linux bypass.

Time was decades ago Oracle demanded raw access to the SCSI device
because they claimed it was easier for customers and faster for them if
they just talked to devices in their native protocol and got all of the
annoying kernel filesystems and page cache out of the way.  Fast
fowards to today and database vendors largely use filesystems thanks to
the evolution of interfaces (direct I/O) that support what they want to
do and the huge annoyance for customers of having to manage huge
numbers of unidentifiable raw devices.

For NVMe and net we do have SPDK and DPDK.  What I find is that people
tend to use them for niche use cases (like the NVMe KV command set) or
obscure network routers.  Even though the claim they both make is to
get the kernel out of the way and do stuff "way faster" the difficulty
they create by bypassing everything is quite a high burden.

For USB security tokens in the early days we had the huge problem of
everyone inventing their own interface, then they realised this was
unsustainable and came up with CTAP, but it's just a unified way for
user space applications to talk to FIDO tokens over raw USB ... is this
a problem?

3/ The practice of requiring Linux commands to wrap all device
commands
   does not appear to have accelerated upstream participation in the
CXL
   subsystem. I.e. CXL, in contrast to NVME, relegates passthrough to
a
   build-time debug option. Some vendors are even shipping vendor
   specific firmware update facilities even though mainline has
support for
   the CXL standard firmware update mechanism.

   With the impending arrival of CXL switch devices wanting to share
   mailbox handling code with the CXL core, the prohibition of
   device-specific commands is going to generate significant upstream
work
   to wrap all that in Linux commands with little perceivable long
term
   benefit to the subsystem.

CXL and RDMA are also foreshadowing conflicts across subsystems. It
is not difficult to imagine a future CXL or RDMA device that supports
mem, block, net, and drm/accel functionality. Which subsystem's
device-command policy applies to such a thing?

We already have that today: pretty much every device protocol looks a
bit network like and has an Over Ethernet or Over RDMA equivalent.

What all of the prior pass through's taught us is that if the use case
is big enough it will get pulled into the kernel and the kernel will
usually manage it better (DB users).  If it remains a niche use case it
will likely remain out of the kernel, but we won't be hurt by it (NVME
KV protocol) and sometimes it doesn't really matter and the device
manufacturers will sort it out on their own (USB tokens).

Enter the fwctl proposal [1]. From the CXL subsystem perspective it
looks like a long-term solution to the problem of managing
expectations between hardware vendors and mainline subsystems. It
disclaims support for the fast-path (data-plane) and is targeted at
the long tail of slow-path (config/debug plane) device-specific
operations that are often uninteresting to mainline. It sets
expectations that the device must advertise the effect of all
commands so that the kernel can deploy reasonable Kernel Lockdown
policy, or otherwise require CAP_SYS_RAWIO for commands that may
affect user-data. It sets common expectations for device designers,
distribution maintainers, and kernel developers. It is complimentary
to the Linux-command path for operations that need
deeper kernel coordination.

This proposal does look to me more like a tool for configuring highly
malleable fat firmware (or really mini-os) offload type devices (like
intelligent network cards) to interact correctly and be easier to
debug.  Every cloud vendor effectively has their own one of these, so I
think the problem isn't going away, so trying to bring some order to it
looks like a potentially good idea.

The upstream discussion has yielded the full spectrum of positions on
device specific functionality, and it is a topic that needs cross-
kernel consensus as hardware increasingly spans cross-subsystem
concerns. Please consider it for a Maintainers Summit discussion.

I'm with Greg on this ... can you point to some of the contrary
positions?

Regards,

James

Christoph Hellwig wrote:

Passthrough in general is useful, but the problem is that protocols
aren't designed with it in mind.  Look at the list of command that
affect too much state and we have to block in SCSI and NVMe.  And
in NVMe I'm fighting a constant fight in the technical working group
to not add new command that instantly disable the access control we've
built into NVMe passthrough.   So IMHO passthrough can be good idea,
but only if the protocol is designed for it, and protocol designer
generally have a hard time how software works at all, never mind
the futuristic concepts of layering, abstraction and access control.

I think this protocol feedback from the kernel community to hardware
designers is top on the list of benefits for this discussion.

How to draw that "needs kernel mediation" line takes
kernel-developer-practitioner expertise.

The positive takeaway from the NVME experience in my mind is that no
doubt the NVME subsystem has blocked commands that hardware developers
prefer that Linux not filter. That filter is likely trivially bypassed
by repurposing some other non-filtered command to have the desired side
effect. Something like "write to a magic LBA == run filtered command".
The fact that those cynical hacks are not deployed and the technical
working group is still holding discussions on the proper protocol means
that hardware designers have a higher incentive to get it right than to
get it deployed.

The conversation seems to break down when it comes to "trust us this
passthrough is only for device-specific functionality with low cross
vendor or kernel appeal." How can we design protocols that have a
high-level of trust?

A "Command Effects Log" seems like that starting point, with trust that
cynical abuses of that contract have a higher cost than benefit, and
trust that the protocol limits the potential damage of such abuse.

Greg KH wrote:

On Mon, Jul 08, 2024 at 03:26:43PM -0700, Dan Williams wrote:

quoted

Enter the fwctl proposal [1]. From the CXL subsystem perspective it
looks like a long-term solution to the problem of managing expectations
between hardware vendors and mainline subsystems. It disclaims support
for the fast-path (data-plane) and is targeted at the long tail of
slow-path (config/debug plane) device-specific operations that are often
uninteresting to mainline.

That's not true at all, device-specific operations are very interesting
to mainline, and I would argue that the "slow-path" is the most
important thing for the kernel to manage as that is where the security
and unification layers can be properly enforced.

Vendors that think the control plane should just be allowed to be
accessed by userspace "blindly" are not saying outloud that they just
want to circumvent the security layer entirely like they previously were
doing by directly accessing /dev/mem which is one of the strongest
reasons to keep enforcing this through the kernel as Christoph points
out.

It's the cumulation of multiple vendors of semi-alike config paths that
allow us to standardize them to provide the most important thing of all,
a unified view to userspace where a user does not care about what type
of hardware is running, which is really the goal of Linux as well, but
directly goes against what a vendor wants to have happen.

100% agree and this is the argument I made for creating
tsm-report-configfs. What I meant by "config-plane" is something more
along the lines of an FPGA bitstream redefining device parameters that
get realized after a reset, less the in-band live-config-plane of a
device that is likely similar across a class of devices where the kernel
has a clear interest in mediating.

A fuzzier example is that CXL has commands like "read vendor debug log"
and I can see a device having specific control toggles to modify the
data the device dumps into that log. The expectation is the device
designer will put anything of general/kernel interest into other formal
telemetry commands. That "vendor debug log" command is trusted to not
have other kernel relevant side effects, it just facilitates debug
between end users and device manufacturers.

quoted

It sets expectations that the device must
advertise the effect of all commands so that the kernel can deploy
reasonable Kernel Lockdown policy, or otherwise require CAP_SYS_RAWIO
for commands that may affect user-data.

Yes, this is a good start, but it might still be too "vendor-specific"
at this point in time.

This gets back to the trusted protocols point that Christoph raised. The
community has examples where this tension is resolved with negotiated
protocol concessions. The fwctl discussion has not progressed to the
protocol concessions phase.

quoted

It sets common expectations for
device designers, distribution maintainers, and kernel developers. It is
complimentary to the Linux-command path for operations that need deeper
kernel coordination.

Yes, it's a good start, BUT by circumventing the network control plane,
the network driver maintainers rightfully are worried about this as
their review comments seem to be ignored here.  The rest of us
maintainers can't ignore that objection, sorry.

Hence a mediated discussion proposal because the on list debate has a
visible deficit of trust that email is unlikely to overcome.

James Bottomley wrote:

quoted

The upstream discussion has yielded the full spectrum of positions on
device specific functionality, and it is a topic that needs cross-
kernel consensus as hardware increasingly spans cross-subsystem
concerns. Please consider it for a Maintainers Summit discussion.

I'm with Greg on this ... can you point to some of the contrary
positions?

This thread has that discussion:

http://lore.kernel.org/0-v1-9912f1a11620+2a-fwctl_jgg@nvidia.com

I do not want to speak for others on the saliency of their points, all I
can say is that the contrary positions have so far not moved me to drop
consideration of fwctl for CXL.

Where CXL has a Command Effects Log that is a reasonable protocol for
making decisions about opaque command codes, and that CXL already has a
few years of experience with the commands that *do* need a Linux-command
wrapper.

Some open questions from that thread are: what does it mean for the fate
of a proposal if one subsystem Acks the ABI and another Naks it for a
device that crosses subsystem functionality? Would a cynical hardware
response just lead to plumbing an NVME admin queue, or CXL mailbox to
get device-specific commands past another subsystem's objection?

My reconsideration of the "debug-build only" policy for CXL
device-specific commands was influenced by a conversation with a distro
developer where they asserted, paraphrasing: "at what point is a device
vendor incentivized to ship an out-of-tree module just to restore their
passthrough functionality?. At that point upstream has lost out on
collaboration and distro kernel ABI has gained another out-of-tree
consumer."

So the tension is healthy, but it has diminishing returns past a certain
point.

On Tue, Jul 09, 2024 at 12:43:50PM -0700, Dan Williams wrote:

A "Command Effects Log" seems like that starting point, with trust that
cynical abuses of that contract have a higher cost than benefit, and
trust that the protocol limits the potential damage of such abuse.

I've taken the view that companies are now very vigilant about
security and often have their own internal incentives and procedures
to do secure things.

If someone does a cynical security breaking thing and deploys it to a
wide user base they are likely to be caught by a security researcher
and embarassed with a CVE and a web site with a snappy name.

Not 100% of course, but it is certainly not a wild west of people just
doing whatever they want.

The other half of this bargin is we have to be much clearer about what
the security model is and what is security breaking. Like Christoph I
often have conversations with people who don't understand the basics
of how the Linux security models should work and are doing device-side
work that has to fit into it.

Jason

On Tue, 9 Jul 2024 15:15:13 -0700
Dan Williams [off-list ref] wrote:

James Bottomley wrote:

quoted

quoted

The upstream discussion has yielded the full spectrum of positions on
device specific functionality, and it is a topic that needs cross-
kernel consensus as hardware increasingly spans cross-subsystem
concerns. Please consider it for a Maintainers Summit discussion.  

I'm with Greg on this ... can you point to some of the contrary
positions?  

This thread has that discussion:

http://lore.kernel.org/0-v1-9912f1a11620+2a-fwctl_jgg@nvidia.com

I do not want to speak for others on the saliency of their points, all I
can say is that the contrary positions have so far not moved me to drop
consideration of fwctl for CXL.

I was resisting rat holing. Oh well...

For a 'subset' of CXL.  There are a wide range of controls that are highly
destructive, potentially to other hosts (simplest one is a command that
will surprise remove someone else's memory). For those I'm not sure
fwctl gets us anywhere - but we still need a solution (Subject to
config gates etc as typically this is BMCs not hosts).
Maybe fwctl eventually ends up with levels of 'safety' (beyond the
current read vs write vs write_full, or maybe those are enough).

Complexities such as message tunneling to multiple components are also
going to be fun, but we want the non destructive bits of those to work
as part of the safe set, so we can get telemetry from downstream devices.

Good to cover the debug and telemetry usecase, but it still leaves us with
gaping holes were we need to solve the permissions problem, perhaps that
is layered on top of fwctl, perhaps something else is needed.

So if fwctl is adopted, I do want the means to use it for the highly
destructive stuff as well!  Maybe that's a future discussion.

Where CXL has a Command Effects Log that is a reasonable protocol for
making decisions about opaque command codes, and that CXL already has a
few years of experience with the commands that *do* need a Linux-command
wrapper.

Worth asking if this will incorporate unknown but not vendor defined
commands.  There is a long tail of stuff in the spec we haven't caught up
with yet.  Or you thinking keep this for the strictly vendor defined stuff?

Some open questions from that thread are: what does it mean for the fate
of a proposal if one subsystem Acks the ABI and another Naks it for a
device that crosses subsystem functionality? Would a cynical hardware
response just lead to plumbing an NVME admin queue, or CXL mailbox to
get device-specific commands past another subsystem's objection?

My reconsideration of the "debug-build only" policy for CXL
device-specific commands was influenced by a conversation with a distro
developer where they asserted, paraphrasing: "at what point is a device
vendor incentivized to ship an out-of-tree module just to restore their
passthrough functionality?. At that point upstream has lost out on
collaboration and distro kernel ABI has gained another out-of-tree
consumer."

So the tension is healthy, but it has diminishing returns past a certain
point.

On Wed, 10 Jul 2024 14:22:38 +0100
Jonathan Cameron [off-list ref] wrote:

On Tue, 9 Jul 2024 15:15:13 -0700
Dan Williams [off-list ref] wrote:

quoted

James Bottomley wrote:  

quoted

quoted

The upstream discussion has yielded the full spectrum of positions on
device specific functionality, and it is a topic that needs cross-
kernel consensus as hardware increasingly spans cross-subsystem
concerns. Please consider it for a Maintainers Summit discussion.    

I'm with Greg on this ... can you point to some of the contrary
positions?    

This thread has that discussion:

http://lore.kernel.org/0-v1-9912f1a11620+2a-fwctl_jgg@nvidia.com

I do not want to speak for others on the saliency of their points, all I
can say is that the contrary positions have so far not moved me to drop
consideration of fwctl for CXL.  

I was resisting rat holing. Oh well...

To throw another 'fun' one in there.  For anything integrated with the host
there is a proposal to provide a MCTP via PCC (ACPI described mailbox). [1]
I don't think it makes sense to rule that out as it's logically no
different from MCTP in general (e.g. a host controller for PCI VDM, or
I2C etc)

Anyone who has a suitable firmware can do whatever they like with that
and the interfaces is exposed directly to userspace. Adam, perhaps you can
describe your use case a little?  Is it applicable to general server distros?

We might suggest distributions don't enable MCTP but does that
actually get us anywhere?  Anyhow, I suspect there are other similar routes, but
this one happens to be under review at the moment.

[1] https://lore.kernel.org/all/20240702225845.322234-1-admiyo@os.amperecomputing.com/

For a 'subset' of CXL.  There are a wide range of controls that are highly
destructive, potentially to other hosts (simplest one is a command that
will surprise remove someone else's memory). For those I'm not sure
fwctl gets us anywhere - but we still need a solution (Subject to
config gates etc as typically this is BMCs not hosts).
Maybe fwctl eventually ends up with levels of 'safety' (beyond the
current read vs write vs write_full, or maybe those are enough).

Complexities such as message tunneling to multiple components are also
going to be fun, but we want the non destructive bits of those to work
as part of the safe set, so we can get telemetry from downstream devices.

Good to cover the debug and telemetry usecase, but it still leaves us with
gaping holes were we need to solve the permissions problem, perhaps that
is layered on top of fwctl, perhaps something else is needed.

So if fwctl is adopted, I do want the means to use it for the highly
destructive stuff as well!  Maybe that's a future discussion.

quoted

Where CXL has a Command Effects Log that is a reasonable protocol for
making decisions about opaque command codes, and that CXL already has a
few years of experience with the commands that *do* need a Linux-command
wrapper.  

Worth asking if this will incorporate unknown but not vendor defined
commands.  There is a long tail of stuff in the spec we haven't caught up
with yet.  Or you thinking keep this for the strictly vendor defined stuff?

quoted

Some open questions from that thread are: what does it mean for the fate
of a proposal if one subsystem Acks the ABI and another Naks it for a
device that crosses subsystem functionality? Would a cynical hardware
response just lead to plumbing an NVME admin queue, or CXL mailbox to
get device-specific commands past another subsystem's objection?

My reconsideration of the "debug-build only" policy for CXL
device-specific commands was influenced by a conversation with a distro
developer where they asserted, paraphrasing: "at what point is a device
vendor incentivized to ship an out-of-tree module just to restore their
passthrough functionality?. At that point upstream has lost out on
collaboration and distro kernel ABI has gained another out-of-tree
consumer."

So the tension is healthy, but it has diminishing returns past a certain
point.
  

On Tue, Jul 09, 2024 at 09:02:25AM -0700, James Bottomley wrote:

For NVMe and net we do have SPDK and DPDK.  What I find is that people
tend to use them for niche use cases (like the NVMe KV command set) or
obscure network routers.  Even though the claim they both make is to
get the kernel out of the way and do stuff "way faster" the difficulty
they create by bypassing everything is quite a high burden.

[..]
 

What all of the prior pass through's taught us is that if the use case
is big enough it will get pulled into the kernel and the kernel will
usually manage it better (DB users).  If it remains a niche use case it
will likely remain out of the kernel, but we won't be hurt by it (NVME
KV protocol) and sometimes it doesn't really matter and the device
manufacturers will sort it out on their own (USB tokens).

I don't see it as being linked to big enough use case at all.

The kernel gets involved if there are good technical reasons to do
so. Databases running over real filesystems with O_DIRECT is really
technically better than raw block devices.

While DPDK shows the opposite, userspace is the technically better
option. This is now shown at scale. DPDK is not some niche. A big
chunk of internet traffic is going through DPDKs, especially for
mobile. Many ORAN solutions include DPDK on Linux.

What has been improved kernel-side is the intergation. DPDK
deployments now often use RDMA raw queue pairs instead of VFIO, which
laregly eliminates the "high burden".

There are many other cases, like DPDK, where the right answer is to
reduce the kernel involvement. It is not so simple that things always
get pulled into the kernel.

Jason

On Tue, Jul 09, 2024 at 03:47:23PM +0300, Leon Romanovsky wrote:

On Tue, Jul 09, 2024 at 02:33:17PM +0200, Greg KH wrote:

quoted

On Tue, Jul 09, 2024 at 03:25:47PM +0300, Leon Romanovsky wrote:

quoted

On Tue, Jul 09, 2024 at 12:01:06PM +0200, Greg KH wrote:

quoted

On Mon, Jul 08, 2024 at 03:26:43PM -0700, Dan Williams wrote:

<...>

quoted

quoted

It sets common expectations for
device designers, distribution maintainers, and kernel developers. It is
complimentary to the Linux-command path for operations that need deeper
kernel coordination.

Yes, it's a good start, BUT by circumventing the network control plane,
the network driver maintainers rightfully are worried about this as
their review comments seem to be ignored here.  The rest of us
maintainers can't ignore that objection, sorry.

Can you please point to the TECHNICAL review comments that were
presented and later ignored?

I can't remember review comments that were made yesterday, let alone
months ago, sorry.

So I will summarize the situation for you. There are NO technical review
comments from netdev maintainer (not plural maintainers). The difference
is philosophical and not technical.

Yes, to my knowledge no technical comments where given against fwctl
that have not been addressed.

Also, please, can we understand that networking in Linux has diverse
maintainership?

These days a huge amount of networking hardware is being deployed
where the software netdev networking stack is not the primary software
driving the HW. Many sites principally use RDMA, DPDK, and other
stacks on some devices.

For instance I previously shared a paper from Azure indicating that
over 70% of traffic on some of their networks was RDMA focused. [1]

We need to *share* the responsibility to support this complex HW. I'm
certain we won't always agree on the right way to do that.

If there is no real technical harm, let's not leap to vetoing ideas
from other networking-related subsystem maintainers please.

Jason

[1] https://www.usenix.org/system/files/nsdi23-bai.pdf

On Wed, Jul 10, 2024 at 02:22:38PM +0100, Jonathan Cameron wrote:

On Tue, 9 Jul 2024 15:15:13 -0700
Dan Williams [off-list ref] wrote:

quoted

James Bottomley wrote:

quoted

quoted

The upstream discussion has yielded the full spectrum of positions on
device specific functionality, and it is a topic that needs cross-
kernel consensus as hardware increasingly spans cross-subsystem
concerns. Please consider it for a Maintainers Summit discussion.  

I'm with Greg on this ... can you point to some of the contrary
positions?  

This thread has that discussion:

http://lore.kernel.org/0-v1-9912f1a11620+2a-fwctl_jgg@nvidia.com

I do not want to speak for others on the saliency of their points, all I
can say is that the contrary positions have so far not moved me to drop
consideration of fwctl for CXL.

I was resisting rat holing. Oh well...

For a 'subset' of CXL.  There are a wide range of controls that are highly
destructive, potentially to other hosts (simplest one is a command that
will surprise remove someone else's memory).

I don't know alot of CXL, but from talking with Dan and reading these
posts it seems to me that CXL turn into a network, with switches and
multi-node and then somehow hid some kind of 'raw packet' interface to
communicate node-to-node. But never added any kind of node level
authorization? ie trust the nodes not to hurt each other?

Sounds sketchy to me :)

So if fwctl is adopted, I do want the means to use it for the highly
destructive stuff as well!  Maybe that's a future discussion.

With that kind of security model you probably have to trust the
userspace, even in a lockdown kernel.

ie can userspace replace the CXL HW that has the command interface
with VFIO and then do anything with nothing more than CAP_SYS_ADMIN
and root?

If so it is not unreasonable that a fwctl interface has a similar
level of protection.

quoted

Where CXL has a Command Effects Log that is a reasonable protocol for
making decisions about opaque command codes, and that CXL already has a
few years of experience with the commands that *do* need a Linux-command
wrapper.

Worth asking if this will incorporate unknown but not vendor defined
commands.  There is a long tail of stuff in the spec we haven't caught up
with yet.  Or you thinking keep this for the strictly vendor defined stuff?

I would allow as much as possible in fwctl that meets the defined
functional limitations and security model.

There is security merit in saying userspace will run, parse and
convert to output complex commands if it can safely do so. From an end
user perspective running a common tool to view the output is generally
always preferred anyhow, and the typical user doesn't really care if
the tool trundles through sysfs or does something else.

Jason

On Thu, 2024-07-11 at 10:50 -0300, Jason Gunthorpe wrote:

On Tue, Jul 09, 2024 at 09:02:25AM -0700, James Bottomley wrote:

quoted

For NVMe and net we do have SPDK and DPDK.  What I find is that
people tend to use them for niche use cases (like the NVMe KV
command set) or obscure network routers.  Even though the claim
they both make is to get the kernel out of the way and do stuff
"way faster" the difficulty they create by bypassing everything is
quite a high burden.

[..]
 

quoted

What all of the prior pass through's taught us is that if the use
case is big enough it will get pulled into the kernel and the
kernel will usually manage it better (DB users).  If it remains a
niche use case it will likely remain out of the kernel, but we
won't be hurt by it (NVME KV protocol) and sometimes it doesn't
really matter and the device manufacturers will sort it out on
their own (USB tokens).

I don't see it as being linked to big enough use case at all.

'it' being fwctl?  I'm happy to take a wait and see approach with that.
I'm in the camp that doesn't squash a novel proposal because the kernel
should be controlling it.  I'm confident that if the use case becomes
big enough the kernel will likely do it in the end.

The kernel gets involved if there are good technical reasons to do
so. Databases running over real filesystems with O_DIRECT is really
technically better than raw block devices.

Exactly for a whole host of performance and more particularly
management reasons.

While DPDK shows the opposite, userspace is the technically better
option. This is now shown at scale. DPDK is not some niche. A big
chunk of internet traffic is going through DPDKs, especially for
mobile. Many ORAN solutions include DPDK on Linux.

ORAN being Open Radio Access Network?  But that's a case in point: the
kernel doesn't have a LTE stack or APN handling for networking.  RAN
hardware is not very widespread outside of cell providers, meaning it
doesn't get a lot of widespread exposure.  However, I believe Osmocom
is trying to change this (giving linux a native stack instead of DPDK)
and they're on record as referring to DPDK as the "Rabbit Hole".

To look at a counter example: how many linux based router boxes (i.e.
hardware based not cloud based) actually use DPDK?  I have a huge list
of cloud projects that overran and got cancelled because they decided
to use DPDK to replace a function the kernel was already doing (because
faster) and then found that if you replace function X in the kernel
generally the rest of the alphabet needs replacing as well, which blows
your project deadlines.

That's not to say there aren't a whole host of uses for DPDK: novel
protocols, traffic classification experiments, etc.  It's just that
it's like this honeytrap for the unwary and any project that comes
along with DPDK somewhere in its spec immediately gets extra scrutiny
(it's not that we don't do them, just that we make sure there's a
genuine use case that isn't reinventing what the kernel already does).

What has been improved kernel-side is the intergation. DPDK
deployments now often use RDMA raw queue pairs instead of VFIO, which
laregly eliminates the "high burden".

There are many other cases, like DPDK, where the right answer is to
reduce the kernel involvement. It is not so simple that things always
get pulled into the kernel.

I don't disagree: there are many novel protocols and other use cases
that will never make it into the kernel simply because they won't get
the adoption; they're all ideal candidates for DPDK.  However, I do
take issue with "reduce kernel involvement" that's what gives rise to
projects that start by rewriting a piece of kernel networking in DPDK
and get sucked down the Rabbit Hole and never come out the other side.

James

On Thu, Jul 11, 2024 at 08:16:55AM -0700, James Bottomley wrote:

quoted

quoted

What all of the prior pass through's taught us is that if the use
case is big enough it will get pulled into the kernel and the
kernel will usually manage it better (DB users).  If it remains a
niche use case it will likely remain out of the kernel, but we
won't be hurt by it (NVME KV protocol) and sometimes it doesn't
really matter and the device manufacturers will sort it out on
their own (USB tokens).

I don't see it as being linked to big enough use case at all.

'it' being fwctl?  

Sorry I ment "it" as "if the use case is big enough it will get pulled
into the kernel"

I'm in the camp that doesn't squash a novel proposal because the kernel
should be controlling it.  I'm confident that if the use case becomes
big enough the kernel will likely do it in the end.

Yes, I agree as well. If there is technical excellence to draw
someting into the kernel then it will definitely happen regardless if
other options are available.

quoted

While DPDK shows the opposite, userspace is the technically better
option. This is now shown at scale. DPDK is not some niche. A big
chunk of internet traffic is going through DPDKs, especially for
mobile. Many ORAN solutions include DPDK on Linux.

ORAN being Open Radio Access Network?  

Yes

But that's a case in point: the kernel doesn't have a LTE stack or
APN handling for networking.

That isn't really the main point of ORAN, a native LTE stack is
interesting separately but is a different problem space than ORAN.

ORAN is about doing all the complex maths for RF signal processing in
software disaggregated from the antennae hardware, in realtime. It is
some bonkers combination of >= 100G networking and hard realtime
packet processing with incredible math using dedicated accelerator HW.

I don't disagree: there are many novel protocols and other use cases
that will never make it into the kernel simply because they won't get
the adoption;

My point is that prefering userspace/kernel isn't about NOVEL, it is
about technical excellence.

DPDK running a >= 100G NIC with extensive HW packet processing
offloads is technically excellent at doing proprietary traffic
processing work in mobile and cloud networks.

We are already running these DPDK things at near perfect performance
with minimal integration hurdles, there is not a lot of room to
actually be *better* with an in-kernel alternative.

This is what I mean, technically good displaces technically bad, it
has nothing to do with kernel=good userspace=bad.

The design of DPDK, with reduced kernel involvment, is a technically
correct approach to achiving it's goal, and it's goal is legitimate.

To expand a bit, Mellanox has supported both paths for a long time, a
lot of work has been done to improve the netstack and mlx5 to give
matching performance compared to DPDK in some use cases. Yet, I think
the market has spoken and DPDK seems to have much more popularity in
some very big verticals.

Yes, using DPDK carries a heavy development cost and people should
avoid it if they don't see a 5-10x improvment, IMHO.

Jason

Jonathan Cameron wrote:

On Tue, 9 Jul 2024 15:15:13 -0700
Dan Williams [off-list ref] wrote:

quoted

James Bottomley wrote:

quoted

quoted

The upstream discussion has yielded the full spectrum of positions on
device specific functionality, and it is a topic that needs cross-
kernel consensus as hardware increasingly spans cross-subsystem
concerns. Please consider it for a Maintainers Summit discussion.  

I'm with Greg on this ... can you point to some of the contrary
positions?  

This thread has that discussion:

http://lore.kernel.org/0-v1-9912f1a11620+2a-fwctl_jgg@nvidia.com

I do not want to speak for others on the saliency of their points, all I
can say is that the contrary positions have so far not moved me to drop
consideration of fwctl for CXL.

I was resisting rat holing. Oh well...

For a 'subset' of CXL.  There are a wide range of controls that are highly
destructive, potentially to other hosts (simplest one is a command that
will surprise remove someone else's memory). For those I'm not sure
fwctl gets us anywhere - but we still need a solution (Subject to
config gates etc as typically this is BMCs not hosts).
Maybe fwctl eventually ends up with levels of 'safety' (beyond the
current read vs write vs write_full, or maybe those are enough).

It is not clear to me that fwctl needs more levels of safety vs the
local subsystem config options controlling what can and can not be sent
over the channel. The CXL backend for fwctl adds the local "command
effects" level of safety.

For the "Linux as BMC" case the security model is external to the
kernel, right? Which means it does not present a protocol that the
kernel can reason about.

Unless and until someone develops an authorization model for BMC nodes
to join a network topology I think that use case is orthogonal to the
primary in-band use case for fwctl.

It is still useful there to avoid defining yet another transport, but a
node that has unfettered access to wreak havoc on the network is not the
kernel's problem.

Complexities such as message tunneling to multiple components are also
going to be fun, but we want the non destructive bits of those to work
as part of the safe set, so we can get telemetry from downstream devices.

Good to cover the debug and telemetry usecase, but it still leaves us with
gaping holes were we need to solve the permissions problem, perhaps that
is layered on top of fwctl, perhaps something else is needed.

But that's more a CXL switch-management command security protocol
problem than fwctl, right? In other words, as far as I understand, there
is no spec provided permission model for switch management that Linux
could enforce, so it's more in the category of build a kernel that can
pass any payload and hope someone else has solved the problem of
limiting what damage that node can inflict.

So if fwctl is adopted, I do want the means to use it for the highly
destructive stuff as well!  Maybe that's a future discussion.

quoted

Where CXL has a Command Effects Log that is a reasonable protocol for
making decisions about opaque command codes, and that CXL already has a
few years of experience with the commands that *do* need a Linux-command
wrapper.

Worth asking if this will incorporate unknown but not vendor defined
commands.  There is a long tail of stuff in the spec we haven't caught up
with yet.  Or you thinking keep this for the strictly vendor defined stuff?

Long term, yes, it should be able to expand to any command code family.
Short term, to get started, the CXL "Feature" facility at least conveys
whether opcodes are reads or writes, independent of their side effects,
and are scoped to be "settings".

There is still the matter of background commands need to support
cancellation to avoid indefinite background-command-slot monopolization,
and there are still commands that need kernel coordination. So, I see
fwctl command support arriving in stages.

On Thu, 11 Jul 2024 12:05:59 -0300
Jason Gunthorpe [off-list ref] wrote:

On Wed, Jul 10, 2024 at 02:22:38PM +0100, Jonathan Cameron wrote:

quoted

On Tue, 9 Jul 2024 15:15:13 -0700
Dan Williams [off-list ref] wrote:
  

quoted

James Bottomley wrote:  

quoted

quoted

The upstream discussion has yielded the full spectrum of positions on
device specific functionality, and it is a topic that needs cross-
kernel consensus as hardware increasingly spans cross-subsystem
concerns. Please consider it for a Maintainers Summit discussion.    

I'm with Greg on this ... can you point to some of the contrary
positions?    

This thread has that discussion:

http://lore.kernel.org/0-v1-9912f1a11620+2a-fwctl_jgg@nvidia.com

I do not want to speak for others on the saliency of their points, all I
can say is that the contrary positions have so far not moved me to drop
consideration of fwctl for CXL.  

I was resisting rat holing. Oh well...

For a 'subset' of CXL.  There are a wide range of controls that are highly
destructive, potentially to other hosts (simplest one is a command that
will surprise remove someone else's memory).  

I don't know alot of CXL, but from talking with Dan and reading these
posts it seems to me that CXL turn into a network, with switches and
multi-node and then somehow hid some kind of 'raw packet' interface to
communicate node-to-node. But never added any kind of node level
authorization? ie trust the nodes not to hurt each other?

You can't actually communicate node to node in the sense of host
to host.  You can just unplug stuff on another other host
(I guess that's a low bandwith comms channel...)
Data access should not be possible in general (ignoring shared
memory which is unrelated to the control path).
Control plane for the nasty stuff should all be in control
of one entity in the system - termed a fabric manager.

Sounds sketchy to me :)

Yes. The model is with the intent that this is only exposed by
hardware to a BMC / Fabric Manager - so security is by wiring.
The reason it's exposed on a PCI upstream port on a switch is that
there are designs where the Fabric manager is 'just another host'.
It's probably not running general software but that Fabric Manager
is running Linux too.  This isn't hugely different to not wiring
your MCTP management interface directly to the host such that the
OS can mess with it.

quoted

So if fwctl is adopted, I do want the means to use it for the highly
destructive stuff as well!  Maybe that's a future discussion.  

With that kind of security model you probably have to trust the
userspace, even in a lockdown kernel.

Agreed - but if we put infrastructure in place I want it to support
this as well.

ie can userspace replace the CXL HW that has the command interface
with VFIO and then do anything with nothing more than CAP_SYS_ADMIN
and root?

Yes if the wiring put that special PCI function on your PCI hierarchy.

If so it is not unreasonable that a fwctl interface has a similar
level of protection.

quoted

quoted

Where CXL has a Command Effects Log that is a reasonable protocol for
making decisions about opaque command codes, and that CXL already has a
few years of experience with the commands that *do* need a Linux-command
wrapper.  

Worth asking if this will incorporate unknown but not vendor defined
commands.  There is a long tail of stuff in the spec we haven't caught up
with yet.  Or you thinking keep this for the strictly vendor defined stuff?  

I would allow as much as possible in fwctl that meets the defined
functional limitations and security model.

There is security merit in saying userspace will run, parse and
convert to output complex commands if it can safely do so. From an end
user perspective running a common tool to view the output is generally
always preferred anyhow, and the typical user doesn't really care if
the tool trundles through sysfs or does something else.

Fair enough.  A bit of potential duplication won't be too painful
(fwctl vs stuff that we know is safe enough for a 'normal' interface).

Thanks,

Jonathan

Jason

On Thu, Jul 11, 2024 at 06:01:00PM +0100, Jonathan Cameron wrote:

Control plane for the nasty stuff should all be in control
of one entity in the system - termed a fabric manager.

quoted

Sounds sketchy to me :)

Yes. The model is with the intent that this is only exposed by
hardware to a BMC / Fabric Manager - so security is by wiring.

If you rely on physical seperation then I'd say that the Linux who
gets access to that physical HW should have rights to operate it, even
from userspace. You may say only root/user/label should have those
special rights, but it is kind of baked into the model that the
special physical connection lets you harm other nodes too.

It is important to think what things should be in fwctl, if I were to
take a similar situation for IB, the fabric manager plugs into
/dev/infiniband/umad* and that is the special interface for exchanging
packets between nodes to do fabric management.

But IB has a well defined container for fabric management and it is
easy to steer things into proper places.

Jason

On Thu, 11 Jul 2024 09:36:27 -0700
Dan Williams [off-list ref] wrote:

Jonathan Cameron wrote:

quoted

On Tue, 9 Jul 2024 15:15:13 -0700
Dan Williams [off-list ref] wrote:
  

quoted

James Bottomley wrote:  

quoted

quoted

The upstream discussion has yielded the full spectrum of positions on
device specific functionality, and it is a topic that needs cross-
kernel consensus as hardware increasingly spans cross-subsystem
concerns. Please consider it for a Maintainers Summit discussion.    

I'm with Greg on this ... can you point to some of the contrary
positions?    

This thread has that discussion:

http://lore.kernel.org/0-v1-9912f1a11620+2a-fwctl_jgg@nvidia.com

I do not want to speak for others on the saliency of their points, all I
can say is that the contrary positions have so far not moved me to drop
consideration of fwctl for CXL.  

I was resisting rat holing. Oh well...

For a 'subset' of CXL.  There are a wide range of controls that are highly
destructive, potentially to other hosts (simplest one is a command that
will surprise remove someone else's memory). For those I'm not sure
fwctl gets us anywhere - but we still need a solution (Subject to
config gates etc as typically this is BMCs not hosts).
Maybe fwctl eventually ends up with levels of 'safety' (beyond the
current read vs write vs write_full, or maybe those are enough).  

It is not clear to me that fwctl needs more levels of safety vs the
local subsystem config options controlling what can and can not be sent
over the channel. The CXL backend for fwctl adds the local "command
effects" level of safety.

For the "Linux as BMC" case the security model is external to the
kernel, right? Which means it does not present a protocol that the
kernel can reason about.

The security model is indeed external, but I'd like a Linux BMC
config to allow turning off the protections but still using the
same fundamental interfaces as we normally use for the safe stuff.
I don't want
1) The CXL IOCTLs
2) FWCTL
3) Yet another interface.

Unless and until someone develops an authorization model for BMC nodes
to join a network topology I think that use case is orthogonal to the
primary in-band use case for fwctl.

Use case wise I agree this isn't the current primary in-band use case
for fwctl, hence the rat hole introductory comment.

It is still useful there to avoid defining yet another transport, but a
node that has unfettered access to wreak havoc on the network is not the
kernel's problem.

As long as I can enable it via a sensible interface (and don't need to
spin another) that is fine by me.

quoted

Complexities such as message tunneling to multiple components are also
going to be fun, but we want the non destructive bits of those to work
as part of the safe set, so we can get telemetry from downstream devices.

Good to cover the debug and telemetry usecase, but it still leaves us with
gaping holes were we need to solve the permissions problem, perhaps that
is layered on top of fwctl, perhaps something else is needed.  

But that's more a CXL switch-management command security protocol
problem than fwctl, right? In other words, as far as I understand, there
is no spec provided permission model for switch management that Linux
could enforce, so it's more in the category of build a kernel that can
pass any payload and hope someone else has solved the problem of
limiting what damage that node can inflict.

Two separate things here.

For tunneling, there is plenty that will map to fwctl because it's just
a transport question.  The tunnel command itself has a CEL that says
it might eat babies so we'd need to check the relevant CEL for the
destination to make sure they were just as safe as non tunneled version.
So it's just an implementation detail, be it a fiddly one.

For destructive options sure it's a config problem. But I do want
to be able to lock down the kernel on the BMC but still allow the
discructive command. Lock down is protecting and restricting the BMC
not the other hosts in this use case. 

quoted

So if fwctl is adopted, I do want the means to use it for the highly
destructive stuff as well!  Maybe that's a future discussion.
  

quoted

Where CXL has a Command Effects Log that is a reasonable protocol for
making decisions about opaque command codes, and that CXL already has a
few years of experience with the commands that *do* need a Linux-command
wrapper.  

Worth asking if this will incorporate unknown but not vendor defined
commands.  There is a long tail of stuff in the spec we haven't caught up
with yet.  Or you thinking keep this for the strictly vendor defined stuff?  

Long term, yes, it should be able to expand to any command code family.
Short term, to get started, the CXL "Feature" facility at least conveys
whether opcodes are reads or writes, independent of their side effects,
and are scoped to be "settings".

There is still the matter of background commands need to support
cancellation to avoid indefinite background-command-slot monopolization,
and there are still commands that need kernel coordination. So, I see
fwctl command support arriving in stages.

Makes sense.  Tunneled access to CXL features should be an a good explorative
feature to do reasonably soon.

Jonathan

On Wed, Jul 10, 2024 at 10:05:14AM -0300, Jason Gunthorpe wrote:

On Tue, Jul 09, 2024 at 12:43:50PM -0700, Dan Williams wrote:

quoted

A "Command Effects Log" seems like that starting point, with trust that
cynical abuses of that contract have a higher cost than benefit, and
trust that the protocol limits the potential damage of such abuse.

I've taken the view that companies are now very vigilant about
security and often have their own internal incentives and procedures
to do secure things.

If someone does a cynical security breaking thing and deploys it to a
wide user base they are likely to be caught by a security researcher
and embarassed with a CVE and a web site with a snappy name.

That may be the case in the server world, and for protocols such as
NVMe. My experience in the media world differs. I've seen too many
horrors to list them all here, so I'll only mention one of the worst
examples coming to my mind, of an (BSP) driver taking a physical address
from unpriviledged userspace and giving it to a DMA engine without any
filtering. I think this was mostly to be blamed on the developer not
knowing better, there was no malicious intent.

In general, can we trust closed-source firmwares when they document the
side effects of pass-through commands ? Again, I think the answer
differs between different classes of devices, the security culture is
not uniform across the whole IT industry.

Not 100% of course, but it is certainly not a wild west of people just
doing whatever they want.

The other half of this bargin is we have to be much clearer about what
the security model is and what is security breaking. Like Christoph I
often have conversations with people who don't understand the basics
of how the Linux security models should work and are doing device-side
work that has to fit into it.

-- 
Regards,

Laurent Pinchart

On Tue, Jul 09, 2024 at 03:15:13PM -0700, Dan Williams wrote:

James Bottomley wrote:

quoted

quoted

The upstream discussion has yielded the full spectrum of positions on
device specific functionality, and it is a topic that needs cross-
kernel consensus as hardware increasingly spans cross-subsystem
concerns. Please consider it for a Maintainers Summit discussion.

I'm with Greg on this ... can you point to some of the contrary
positions?

This thread has that discussion:

http://lore.kernel.org/0-v1-9912f1a11620+2a-fwctl_jgg@nvidia.com

I do not want to speak for others on the saliency of their points, all I
can say is that the contrary positions have so far not moved me to drop
consideration of fwctl for CXL.

Where CXL has a Command Effects Log that is a reasonable protocol for
making decisions about opaque command codes, and that CXL already has a
few years of experience with the commands that *do* need a Linux-command
wrapper.

Some open questions from that thread are: what does it mean for the fate
of a proposal if one subsystem Acks the ABI and another Naks it for a
device that crosses subsystem functionality? Would a cynical hardware
response just lead to plumbing an NVME admin queue, or CXL mailbox to
get device-specific commands past another subsystem's objection?

My default answer would be to trust the maintainers of the relevant
subsystems (or try to convince them when you disagree :-)). Not only
should they know the technical implications best, they should also have
a good view of the whole vertical stack, and the implications of
pass-through for their ecosystem. This may result in a single NAK
overriding ACKs, but we could also try to find technical solutions when
we'll face such issues, to enforce different sets of rules for the
different functions of a device.

Subsystem hopping is something we're recently noticed for camera ISPs,
where a vendor wanted to move from V4L2 to DRM. Technical reasons for
doing so were given, and they were (in my opinion) rather excuses. The
unspoken real (again in my opinion) reason was to avoid documenting the
firmware interface and ship userspace binary blobs with no way for free
software to use all the device's features. That's something we have been
fighting against for years, trying to convince vendors that they can
provide better and more open camera support without the world
collapsing, with increasing success recently. Saying amen to
pass-through in this case would be a huge step back that would hurt
users and the whole ecosystem in the short and long term.

My reconsideration of the "debug-build only" policy for CXL
device-specific commands was influenced by a conversation with a distro
developer where they asserted, paraphrasing: "at what point is a device
vendor incentivized to ship an out-of-tree module just to restore their
passthrough functionality?. At that point upstream has lost out on
collaboration and distro kernel ABI has gained another out-of-tree
consumer."

So the tension is healthy, but it has diminishing returns past a certain
point.

-- 
Regards,

Laurent Pinchart

On Sun, Jul 21, 2024 at 10:25:30PM +0300, Laurent Pinchart wrote:

On Tue, Jul 09, 2024 at 03:15:13PM -0700, Dan Williams wrote:

quoted

James Bottomley wrote:

quoted

quoted

The upstream discussion has yielded the full spectrum of positions on
device specific functionality, and it is a topic that needs cross-
kernel consensus as hardware increasingly spans cross-subsystem
concerns. Please consider it for a Maintainers Summit discussion.

I'm with Greg on this ... can you point to some of the contrary
positions?

This thread has that discussion:

http://lore.kernel.org/0-v1-9912f1a11620+2a-fwctl_jgg@nvidia.com

I do not want to speak for others on the saliency of their points, all I
can say is that the contrary positions have so far not moved me to drop
consideration of fwctl for CXL.

Where CXL has a Command Effects Log that is a reasonable protocol for
making decisions about opaque command codes, and that CXL already has a
few years of experience with the commands that *do* need a Linux-command
wrapper.

Some open questions from that thread are: what does it mean for the fate
of a proposal if one subsystem Acks the ABI and another Naks it for a
device that crosses subsystem functionality? Would a cynical hardware
response just lead to plumbing an NVME admin queue, or CXL mailbox to
get device-specific commands past another subsystem's objection?

My default answer would be to trust the maintainers of the relevant
subsystems (or try to convince them when you disagree :-)).

You know, trust is a two-way street. If you want to trust maintainers,
they need to trust others as well. The situation where one maintainer
says "I don't trust you, so I will not allow you and other X maintainers
to do Y" is not a healthy situation.

Not only should they know the technical implications best, they should also have
a good view of the whole vertical stack, and the implications of
pass-through for their ecosystem. 

It is wishful thinking. It is clearly not true for large subsystems
and/or complex devices.

Thanks

On Mon, Jul 22, 2024 at 10:31:19AM +0300, Leon Romanovsky wrote:

On Sun, Jul 21, 2024 at 10:25:30PM +0300, Laurent Pinchart wrote:

quoted

On Tue, Jul 09, 2024 at 03:15:13PM -0700, Dan Williams wrote:

quoted

James Bottomley wrote:

quoted

quoted

The upstream discussion has yielded the full spectrum of positions on
device specific functionality, and it is a topic that needs cross-
kernel consensus as hardware increasingly spans cross-subsystem
concerns. Please consider it for a Maintainers Summit discussion.

I'm with Greg on this ... can you point to some of the contrary
positions?

This thread has that discussion:

http://lore.kernel.org/0-v1-9912f1a11620+2a-fwctl_jgg@nvidia.com

I do not want to speak for others on the saliency of their points, all I
can say is that the contrary positions have so far not moved me to drop
consideration of fwctl for CXL.

Where CXL has a Command Effects Log that is a reasonable protocol for
making decisions about opaque command codes, and that CXL already has a
few years of experience with the commands that *do* need a Linux-command
wrapper.

Some open questions from that thread are: what does it mean for the fate
of a proposal if one subsystem Acks the ABI and another Naks it for a
device that crosses subsystem functionality? Would a cynical hardware
response just lead to plumbing an NVME admin queue, or CXL mailbox to
get device-specific commands past another subsystem's objection?

My default answer would be to trust the maintainers of the relevant
subsystems (or try to convince them when you disagree :-)).

You know, trust is a two-way street. If you want to trust maintainers,
they need to trust others as well. The situation where one maintainer
says "I don't trust you, so I will not allow you and other X maintainers
to do Y" is not a healthy situation.

quoted

Not only should they know the technical implications best, they should also have
a good view of the whole vertical stack, and the implications of
pass-through for their ecosystem. 

It is wishful thinking. It is clearly not true for large subsystems
and/or complex devices.

Are you saying that kernel communities behind large subsystems for
complex devices generally have no idea about what they're doing ? Or
that in a small number of particular cases those communities are
clueless ? Or does that apply to just the maintainer, not the whole
subsystem core developers ? I'd like to better understand the scale of
your claim here.

-- 
Regards,

Laurent Pinchart

On Sun, Jul 21, 2024 at 9:25 PM Laurent Pinchart
[off-list ref] wrote:

On Tue, Jul 09, 2024 at 03:15:13PM -0700, Dan Williams wrote:

quoted

James Bottomley wrote:

quoted

quoted

The upstream discussion has yielded the full spectrum of positions on
device specific functionality, and it is a topic that needs cross-
kernel consensus as hardware increasingly spans cross-subsystem
concerns. Please consider it for a Maintainers Summit discussion.

I'm with Greg on this ... can you point to some of the contrary
positions?

This thread has that discussion:

http://lore.kernel.org/0-v1-9912f1a11620+2a-fwctl_jgg@nvidia.com

I do not want to speak for others on the saliency of their points, all I
can say is that the contrary positions have so far not moved me to drop
consideration of fwctl for CXL.

Where CXL has a Command Effects Log that is a reasonable protocol for
making decisions about opaque command codes, and that CXL already has a
few years of experience with the commands that *do* need a Linux-command
wrapper.

Some open questions from that thread are: what does it mean for the fate
of a proposal if one subsystem Acks the ABI and another Naks it for a
device that crosses subsystem functionality? Would a cynical hardware
response just lead to plumbing an NVME admin queue, or CXL mailbox to
get device-specific commands past another subsystem's objection?

My default answer would be to trust the maintainers of the relevant
subsystems (or try to convince them when you disagree :-)). Not only
should they know the technical implications best, they should also have
a good view of the whole vertical stack, and the implications of
pass-through for their ecosystem. This may result in a single NAK
overriding ACKs, but we could also try to find technical solutions when
we'll face such issues, to enforce different sets of rules for the
different functions of a device.

Subsystem hopping is something we're recently noticed for camera ISPs,
where a vendor wanted to move from V4L2 to DRM. Technical reasons for
doing so were given, and they were (in my opinion) rather excuses. The
unspoken real (again in my opinion) reason was to avoid documenting the
firmware interface and ship userspace binary blobs with no way for free
software to use all the device's features. That's something we have been
fighting against for years, trying to convince vendors that they can
provide better and more open camera support without the world
collapsing, with increasing success recently. Saying amen to
pass-through in this case would be a huge step back that would hurt
users and the whole ecosystem in the short and long term.

In my view, DRM is a more suitable model for complex ISPs than V4L2:

- Userspace Complexity: ISPs demand a highly complex and evolving API,
similar to Vulkan or OpenGL. Applications typically need a framework
like libcamera to utilize ISPs effectively, much like Mesa for
graphics cards.

- Lack of Standardization: There's no universal standard for ISPs;
each vendor implements unique features and usage patterns. DRM
addresses this through vendor-specific IOCTLs

- Proprietary Architectures: Vendors often don't fully disclose their
hardware architectures. DRM cleverly only necessitates a Mesa
implementation, not comprehensive documentation.


Our current approach of pushing back against vendors, instead of
seeking compromise, has resulted in the vast majority of the market
(99% if not more) relying on out-of-tree drivers. This leaves users
with no options for utilizing their cameras outside of Android.

DRM allows a hybrid model, where:
- Open Source Foundation: Standard use cases are covered by a fully
open-source stack.
- Vendor Differentiation: Vendors retain the freedom to implement
proprietary features (e.g., automatic makeup) as closed source.

This approach would allow billions of users to access their hardware
more securely and with in-tree driver support. Our current stubborn
pursuit of an idealistic goal has already negatively impacted both
users and the ecosystem.

The late wins, in my opinion, cannot scale to the consumer market, and
Linux will remain a niche market for ISPs.


If such a hybrid model goes against Linux goals, this is something
that should be agreed upon by the whole community, so we have the same
criteria for all subsystems.



--
Ricardo Ribalda

On Mon, Jul 22, 2024 at 11:53:17AM +0300, Laurent Pinchart wrote:

On Mon, Jul 22, 2024 at 10:31:19AM +0300, Leon Romanovsky wrote:

quoted

On Sun, Jul 21, 2024 at 10:25:30PM +0300, Laurent Pinchart wrote:

quoted

On Tue, Jul 09, 2024 at 03:15:13PM -0700, Dan Williams wrote:

quoted

James Bottomley wrote:

quoted

quoted

The upstream discussion has yielded the full spectrum of positions on
device specific functionality, and it is a topic that needs cross-
kernel consensus as hardware increasingly spans cross-subsystem
concerns. Please consider it for a Maintainers Summit discussion.

I'm with Greg on this ... can you point to some of the contrary
positions?

This thread has that discussion:

http://lore.kernel.org/0-v1-9912f1a11620+2a-fwctl_jgg@nvidia.com

I do not want to speak for others on the saliency of their points, all I
can say is that the contrary positions have so far not moved me to drop
consideration of fwctl for CXL.

Where CXL has a Command Effects Log that is a reasonable protocol for
making decisions about opaque command codes, and that CXL already has a
few years of experience with the commands that *do* need a Linux-command
wrapper.

Some open questions from that thread are: what does it mean for the fate
of a proposal if one subsystem Acks the ABI and another Naks it for a
device that crosses subsystem functionality? Would a cynical hardware
response just lead to plumbing an NVME admin queue, or CXL mailbox to
get device-specific commands past another subsystem's objection?

My default answer would be to trust the maintainers of the relevant
subsystems (or try to convince them when you disagree :-)).

You know, trust is a two-way street. If you want to trust maintainers,
they need to trust others as well. The situation where one maintainer
says "I don't trust you, so I will not allow you and other X maintainers
to do Y" is not a healthy situation.

quoted

Not only should they know the technical implications best, they should also have
a good view of the whole vertical stack, and the implications of
pass-through for their ecosystem. 

It is wishful thinking. It is clearly not true for large subsystems
and/or complex devices.

Are you saying that kernel communities behind large subsystems for
complex devices generally have no idea about what they're doing ? Or
that in a small number of particular cases those communities are
clueless ? Or does that apply to just the maintainer, not the whole
subsystem core developers ? I'd like to better understand the scale of
your claim here.

I don't know how you jumped from saying "the maintainers of the relevant
subsystems" to "kernel communities". I'm talking about maintainers, not
communities.

There is no way to know everything about everything. In large subsystems,
the stack above kernel is so vast, which makes it impossible to know all
use cases. This is why some words (... good ... whole ...) in your sentence
are not accurate.

So the idea that one maintainer somehow equal to the whole community and
this person can block something for other members of the larger community
is overreaching.

Thanks

-- 
Regards,

Laurent Pinchart

Hi Leon,

On Mon, Jul 22, 2024 at 01:44:07PM +0300, Leon Romanovsky wrote:

On Mon, Jul 22, 2024 at 11:53:17AM +0300, Laurent Pinchart wrote:

quoted

On Mon, Jul 22, 2024 at 10:31:19AM +0300, Leon Romanovsky wrote:

quoted

On Sun, Jul 21, 2024 at 10:25:30PM +0300, Laurent Pinchart wrote:

quoted

On Tue, Jul 09, 2024 at 03:15:13PM -0700, Dan Williams wrote:

quoted

James Bottomley wrote:

quoted

quoted

The upstream discussion has yielded the full spectrum of positions on
device specific functionality, and it is a topic that needs cross-
kernel consensus as hardware increasingly spans cross-subsystem
concerns. Please consider it for a Maintainers Summit discussion.

I'm with Greg on this ... can you point to some of the contrary
positions?

This thread has that discussion:

http://lore.kernel.org/0-v1-9912f1a11620+2a-fwctl_jgg@nvidia.com

I do not want to speak for others on the saliency of their points, all I
can say is that the contrary positions have so far not moved me to drop
consideration of fwctl for CXL.

Where CXL has a Command Effects Log that is a reasonable protocol for
making decisions about opaque command codes, and that CXL already has a
few years of experience with the commands that *do* need a Linux-command
wrapper.

Some open questions from that thread are: what does it mean for the fate
of a proposal if one subsystem Acks the ABI and another Naks it for a
device that crosses subsystem functionality? Would a cynical hardware
response just lead to plumbing an NVME admin queue, or CXL mailbox to
get device-specific commands past another subsystem's objection?

My default answer would be to trust the maintainers of the relevant
subsystems (or try to convince them when you disagree :-)).

You know, trust is a two-way street. If you want to trust maintainers,
they need to trust others as well. The situation where one maintainer
says "I don't trust you, so I will not allow you and other X maintainers
to do Y" is not a healthy situation.

quoted

Not only should they know the technical implications best, they should also have
a good view of the whole vertical stack, and the implications of
pass-through for their ecosystem. 

It is wishful thinking. It is clearly not true for large subsystems
and/or complex devices.

Are you saying that kernel communities behind large subsystems for
complex devices generally have no idea about what they're doing ? Or
that in a small number of particular cases those communities are
clueless ? Or does that apply to just the maintainer, not the whole
subsystem core developers ? I'd like to better understand the scale of
your claim here.

I don't know how you jumped from saying "the maintainers of the relevant
subsystems" to "kernel communities". I'm talking about maintainers, not
communities.

I wasn't too sure, so that's why I asked. I have also not been very
precise in my previous e-mails. When I mentioned trusting maintainers, I
meant trusting the combined knowledge of the relevant maintainer(s) and
core developer(s) for a subsystem. The number of people that this
covers, and how they collectively reach agreements, very much depends on
subsystems.

There is no way to know everything about everything. In large subsystems,
the stack above kernel is so vast, which makes it impossible to know all
use cases. This is why some words (... good ... whole ...) in your sentence
are not accurate.

So the idea that one maintainer somehow equal to the whole community and
this person can block something for other members of the larger community
is overreaching.

-- 
Regards,

Laurent Pinchart

Hi Ricardo,

On Mon, Jul 22, 2024 at 12:42:52PM +0200, Ricardo Ribalda Delgado wrote:

On Sun, Jul 21, 2024 at 9:25 PM Laurent Pinchart wrote:

quoted

On Tue, Jul 09, 2024 at 03:15:13PM -0700, Dan Williams wrote:

quoted

James Bottomley wrote:

quoted

quoted

The upstream discussion has yielded the full spectrum of positions on
device specific functionality, and it is a topic that needs cross-
kernel consensus as hardware increasingly spans cross-subsystem
concerns. Please consider it for a Maintainers Summit discussion.

I'm with Greg on this ... can you point to some of the contrary
positions?

This thread has that discussion:

http://lore.kernel.org/0-v1-9912f1a11620+2a-fwctl_jgg@nvidia.com

I do not want to speak for others on the saliency of their points, all I
can say is that the contrary positions have so far not moved me to drop
consideration of fwctl for CXL.

Where CXL has a Command Effects Log that is a reasonable protocol for
making decisions about opaque command codes, and that CXL already has a
few years of experience with the commands that *do* need a Linux-command
wrapper.

Some open questions from that thread are: what does it mean for the fate
of a proposal if one subsystem Acks the ABI and another Naks it for a
device that crosses subsystem functionality? Would a cynical hardware
response just lead to plumbing an NVME admin queue, or CXL mailbox to
get device-specific commands past another subsystem's objection?

My default answer would be to trust the maintainers of the relevant
subsystems (or try to convince them when you disagree :-)). Not only
should they know the technical implications best, they should also have
a good view of the whole vertical stack, and the implications of
pass-through for their ecosystem. This may result in a single NAK
overriding ACKs, but we could also try to find technical solutions when
we'll face such issues, to enforce different sets of rules for the
different functions of a device.

Subsystem hopping is something we're recently noticed for camera ISPs,
where a vendor wanted to move from V4L2 to DRM. Technical reasons for
doing so were given, and they were (in my opinion) rather excuses. The
unspoken real (again in my opinion) reason was to avoid documenting the
firmware interface and ship userspace binary blobs with no way for free
software to use all the device's features. That's something we have been
fighting against for years, trying to convince vendors that they can
provide better and more open camera support without the world
collapsing, with increasing success recently. Saying amen to
pass-through in this case would be a huge step back that would hurt
users and the whole ecosystem in the short and long term.

In my view, DRM is a more suitable model for complex ISPs than V4L2:

I know we disagree on this topic :-) I'm sure we'll continue the
conversation, but I think the technical discussion likely belongs to a
different mail thread.

- Userspace Complexity: ISPs demand a highly complex and evolving API,
similar to Vulkan or OpenGL. Applications typically need a framework
like libcamera to utilize ISPs effectively, much like Mesa for
graphics cards.

- Lack of Standardization: There's no universal standard for ISPs;
each vendor implements unique features and usage patterns. DRM
addresses this through vendor-specific IOCTLs

- Proprietary Architectures: Vendors often don't fully disclose their
hardware architectures. DRM cleverly only necessitates a Mesa
implementation, not comprehensive documentation.

This point isn't technical and is more on-topic for this mail thread.

V4L2 doesn't require hundreds of pages of comprehensive documentation in
text form. An open-source userspace implementation that covers the
feature set exposed by the driver is acceptable in place of
documentation (provided, of course, that the userspace code wouldn't be
deliberately obfuscated). This is similar in spirit to the rule for GPU
DRM drivers.

Our current approach of pushing back against vendors, instead of
seeking compromise, has resulted in the vast majority of the market
(99% if not more) relying on out-of-tree drivers. This leaves users
with no options for utilizing their cameras outside of Android.

DRM allows a hybrid model, where:
- Open Source Foundation: Standard use cases are covered by a fully
open-source stack.
- Vendor Differentiation: Vendors retain the freedom to implement
proprietary features (e.g., automatic makeup) as closed source.

V4L2 does as well, you can implement all kind of closed-source ISP
control algorithms in userspace, as long as there's an open-source
implementation that exercises the same hardware features. A good analogy
for people less familiar with ISPs is shader compilers, GPU vendors are
free to ship closed-source implementations that include more
optimizations, as long as the open-source, less optimized implementation
covers the same GPU ISA, so that open-source developers can also work on
optimizing it.

Thinking that DRM would offer a free pass-through path compared to V4L2
doesn't seem realistic to me. Both subsystems will have similar rules.

This approach would allow billions of users to access their hardware
more securely and with in-tree driver support. Our current stubborn
pursuit of an idealistic goal has already negatively impacted both
users and the ecosystem.

The late wins, in my opinion, cannot scale to the consumer market, and
Linux will remain a niche market for ISPs.

If such a hybrid model goes against Linux goals, this is something
that should be agreed upon by the whole community, so we have the same
criteria for all subsystems.

-- 
Regards,

Laurent Pinchart

Hi Laurent

On Mon, Jul 22, 2024 at 1:18 PM Laurent Pinchart
[off-list ref] wrote:

Hi Ricardo,

On Mon, Jul 22, 2024 at 12:42:52PM +0200, Ricardo Ribalda Delgado wrote:

quoted

On Sun, Jul 21, 2024 at 9:25 PM Laurent Pinchart wrote:

quoted

On Tue, Jul 09, 2024 at 03:15:13PM -0700, Dan Williams wrote:

quoted

James Bottomley wrote:

quoted

quoted

The upstream discussion has yielded the full spectrum of positions on
device specific functionality, and it is a topic that needs cross-
kernel consensus as hardware increasingly spans cross-subsystem
concerns. Please consider it for a Maintainers Summit discussion.

I'm with Greg on this ... can you point to some of the contrary
positions?

This thread has that discussion:

http://lore.kernel.org/0-v1-9912f1a11620+2a-fwctl_jgg@nvidia.com

I do not want to speak for others on the saliency of their points, all I
can say is that the contrary positions have so far not moved me to drop
consideration of fwctl for CXL.

Where CXL has a Command Effects Log that is a reasonable protocol for
making decisions about opaque command codes, and that CXL already has a
few years of experience with the commands that *do* need a Linux-command
wrapper.

Some open questions from that thread are: what does it mean for the fate
of a proposal if one subsystem Acks the ABI and another Naks it for a
device that crosses subsystem functionality? Would a cynical hardware
response just lead to plumbing an NVME admin queue, or CXL mailbox to
get device-specific commands past another subsystem's objection?

My default answer would be to trust the maintainers of the relevant
subsystems (or try to convince them when you disagree :-)). Not only
should they know the technical implications best, they should also have
a good view of the whole vertical stack, and the implications of
pass-through for their ecosystem. This may result in a single NAK
overriding ACKs, but we could also try to find technical solutions when
we'll face such issues, to enforce different sets of rules for the
different functions of a device.

Subsystem hopping is something we're recently noticed for camera ISPs,
where a vendor wanted to move from V4L2 to DRM. Technical reasons for
doing so were given, and they were (in my opinion) rather excuses. The
unspoken real (again in my opinion) reason was to avoid documenting the
firmware interface and ship userspace binary blobs with no way for free
software to use all the device's features. That's something we have been
fighting against for years, trying to convince vendors that they can
provide better and more open camera support without the world
collapsing, with increasing success recently. Saying amen to
pass-through in this case would be a huge step back that would hurt
users and the whole ecosystem in the short and long term.

In my view, DRM is a more suitable model for complex ISPs than V4L2:

I know we disagree on this topic :-) I'm sure we'll continue the
conversation, but I think the technical discussion likely belongs to a
different mail thread.

quoted

- Userspace Complexity: ISPs demand a highly complex and evolving API,
similar to Vulkan or OpenGL. Applications typically need a framework
like libcamera to utilize ISPs effectively, much like Mesa for
graphics cards.

- Lack of Standardization: There's no universal standard for ISPs;
each vendor implements unique features and usage patterns. DRM
addresses this through vendor-specific IOCTLs

- Proprietary Architectures: Vendors often don't fully disclose their
hardware architectures. DRM cleverly only necessitates a Mesa
implementation, not comprehensive documentation.

This point isn't technical and is more on-topic for this mail thread.

V4L2 doesn't require hundreds of pages of comprehensive documentation in
text form. An open-source userspace implementation that covers the
feature set exposed by the driver is acceptable in place of
documentation (provided, of course, that the userspace code wouldn't be
deliberately obfuscated). This is similar in spirit to the rule for GPU
DRM drivers.

In DRM vendors typically define a custom IOCTL per driver to pass
command buffers.
Only the command buffer structure, and a mesa implementation using
that command buffer to support the standard features is required.

In V4l2 custom IOCTLs are discouraged. Random command buffers cannot
be passed from userspace, they are typically formed in the driver from
a strictly checked struct.

quoted

Our current approach of pushing back against vendors, instead of
seeking compromise, has resulted in the vast majority of the market
(99% if not more) relying on out-of-tree drivers. This leaves users
with no options for utilizing their cameras outside of Android.

DRM allows a hybrid model, where:
- Open Source Foundation: Standard use cases are covered by a fully
open-source stack.
- Vendor Differentiation: Vendors retain the freedom to implement
proprietary features (e.g., automatic makeup) as closed source.

V4L2 does as well, you can implement all kind of closed-source ISP
control algorithms in userspace, as long as there's an open-source
implementation that exercises the same hardware features. A good analogy

Is it really mandatory to have an open-source 3A algorithm? I thought
defining the input and output from the algorithm was good enough.
AFAIK for some time there was no ipu3 open source algorithm, and the
driver has been upstream.

for people less familiar with ISPs is shader compilers, GPU vendors are
free to ship closed-source implementations that include more
optimizations, as long as the open-source, less optimized implementation
covers the same GPU ISA, so that open-source developers can also work on
optimizing it.

I believe a more accurate description is that in v4l2 is that we
expect that all the registers, device architecture and behaviour to be
documented and accessed with standard IOCTLs. Anything not documented
cannot be accessed by userspace.

In DRM their concern is that there is a fully open source
implementation that the user can use. Vendors have custom IOCTLs and
they can offer proprietary software for some use cases.

Thinking that DRM would offer a free pass-through path compared to V4L2
doesn't seem realistic to me. Both subsystems will have similar rules.

DRM does indeed allow vendors to pass random command buffers and they
will be sent to the hardware. We cannot do that in v4l2.

I might be wrong, but GPU drivers do not deeply inspect the command
buffers to make sure that they do not use any feature not covered by
mesa.

quoted

This approach would allow billions of users to access their hardware
more securely and with in-tree driver support. Our current stubborn
pursuit of an idealistic goal has already negatively impacted both
users and the ecosystem.

The late wins, in my opinion, cannot scale to the consumer market, and
Linux will remain a niche market for ISPs.

If such a hybrid model goes against Linux goals, this is something
that should be agreed upon by the whole community, so we have the same
criteria for all subsystems.

--
Regards,

Laurent Pinchart

-- 
Ricardo Ribalda

On Sun, Jul 21, 2024 at 09:51:05PM +0300, Laurent Pinchart wrote:

That may be the case in the server world, and for protocols such as
NVMe. My experience in the media world differs. I've seen too many
horrors to list them all here, so I'll only mention one of the worst
examples coming to my mind, of an (BSP) driver taking a physical address
from unpriviledged userspace and giving it to a DMA engine without any
filtering. I think this was mostly to be blamed on the developer not
knowing better, there was no malicious intent.

In general, can we trust closed-source firmwares when they document the
side effects of pass-through commands ? Again, I think the answer
differs between different classes of devices, the security culture is
not uniform across the whole IT industry.

That does make sense to me, and I certainly don't feel the same
comfort when looking at embedded or consumer HW that has a
historically much weaker security story.

Jason

On Mon, Jul 22, 2024 at 02:10:04PM +0300, Laurent Pinchart wrote:

Hi Leon,

On Mon, Jul 22, 2024 at 01:44:07PM +0300, Leon Romanovsky wrote:

quoted

On Mon, Jul 22, 2024 at 11:53:17AM +0300, Laurent Pinchart wrote:

quoted

On Mon, Jul 22, 2024 at 10:31:19AM +0300, Leon Romanovsky wrote:

quoted

On Sun, Jul 21, 2024 at 10:25:30PM +0300, Laurent Pinchart wrote:

quoted

On Tue, Jul 09, 2024 at 03:15:13PM -0700, Dan Williams wrote:

quoted

James Bottomley wrote:

quoted

quoted

The upstream discussion has yielded the full spectrum of positions on
device specific functionality, and it is a topic that needs cross-
kernel consensus as hardware increasingly spans cross-subsystem
concerns. Please consider it for a Maintainers Summit discussion.

I'm with Greg on this ... can you point to some of the contrary
positions?

This thread has that discussion:

http://lore.kernel.org/0-v1-9912f1a11620+2a-fwctl_jgg@nvidia.com

I do not want to speak for others on the saliency of their points, all I
can say is that the contrary positions have so far not moved me to drop
consideration of fwctl for CXL.

Where CXL has a Command Effects Log that is a reasonable protocol for
making decisions about opaque command codes, and that CXL already has a
few years of experience with the commands that *do* need a Linux-command
wrapper.

Some open questions from that thread are: what does it mean for the fate
of a proposal if one subsystem Acks the ABI and another Naks it for a
device that crosses subsystem functionality? Would a cynical hardware
response just lead to plumbing an NVME admin queue, or CXL mailbox to
get device-specific commands past another subsystem's objection?

My default answer would be to trust the maintainers of the relevant
subsystems (or try to convince them when you disagree :-)).

You know, trust is a two-way street. If you want to trust maintainers,
they need to trust others as well. The situation where one maintainer
says "I don't trust you, so I will not allow you and other X maintainers
to do Y" is not a healthy situation.

quoted

Not only should they know the technical implications best, they should also have
a good view of the whole vertical stack, and the implications of
pass-through for their ecosystem. 

It is wishful thinking. It is clearly not true for large subsystems
and/or complex devices.

Are you saying that kernel communities behind large subsystems for
complex devices generally have no idea about what they're doing ? Or
that in a small number of particular cases those communities are
clueless ? Or does that apply to just the maintainer, not the whole
subsystem core developers ? I'd like to better understand the scale of
your claim here.

I don't know how you jumped from saying "the maintainers of the relevant
subsystems" to "kernel communities". I'm talking about maintainers, not
communities.

I wasn't too sure, so that's why I asked. I have also not been very
precise in my previous e-mails. When I mentioned trusting maintainers, I
meant trusting the combined knowledge of the relevant maintainer(s) and
core developer(s) for a subsystema

Unfortunately, the reason for this topic proposed for Maintainer's summit
is that the maintainer and core developers are disagree and there is no way
to resolve it, because it is not technical difference, but a philosophical one.

The number of people that this covers, and how they collectively reach
agreements, very much depends on subsystems.

quoted

There is no way to know everything about everything. In large subsystems,
the stack above kernel is so vast, which makes it impossible to know all
use cases. This is why some words (... good ... whole ...) in your sentence
are not accurate.

So the idea that one maintainer somehow equal to the whole community and
this person can block something for other members of the larger community
is overreaching.

-- 
Regards,

Laurent Pinchart

On Mon, Jul 22, 2024 at 04:28:28PM +0300, Leon Romanovsky wrote:

On Mon, Jul 22, 2024 at 02:10:04PM +0300, Laurent Pinchart wrote:

quoted

On Mon, Jul 22, 2024 at 01:44:07PM +0300, Leon Romanovsky wrote:

quoted

On Mon, Jul 22, 2024 at 11:53:17AM +0300, Laurent Pinchart wrote:

quoted

On Mon, Jul 22, 2024 at 10:31:19AM +0300, Leon Romanovsky wrote:

quoted

On Sun, Jul 21, 2024 at 10:25:30PM +0300, Laurent Pinchart wrote:

quoted

On Tue, Jul 09, 2024 at 03:15:13PM -0700, Dan Williams wrote:

quoted

James Bottomley wrote:

quoted

quoted

The upstream discussion has yielded the full spectrum of positions on
device specific functionality, and it is a topic that needs cross-
kernel consensus as hardware increasingly spans cross-subsystem
concerns. Please consider it for a Maintainers Summit discussion.

I'm with Greg on this ... can you point to some of the contrary
positions?

This thread has that discussion:

http://lore.kernel.org/0-v1-9912f1a11620+2a-fwctl_jgg@nvidia.com

I do not want to speak for others on the saliency of their points, all I
can say is that the contrary positions have so far not moved me to drop
consideration of fwctl for CXL.

Where CXL has a Command Effects Log that is a reasonable protocol for
making decisions about opaque command codes, and that CXL already has a
few years of experience with the commands that *do* need a Linux-command
wrapper.

Some open questions from that thread are: what does it mean for the fate
of a proposal if one subsystem Acks the ABI and another Naks it for a
device that crosses subsystem functionality? Would a cynical hardware
response just lead to plumbing an NVME admin queue, or CXL mailbox to
get device-specific commands past another subsystem's objection?

My default answer would be to trust the maintainers of the relevant
subsystems (or try to convince them when you disagree :-)).

You know, trust is a two-way street. If you want to trust maintainers,
they need to trust others as well. The situation where one maintainer
says "I don't trust you, so I will not allow you and other X maintainers
to do Y" is not a healthy situation.

quoted

Not only should they know the technical implications best, they should also have
a good view of the whole vertical stack, and the implications of
pass-through for their ecosystem. 

It is wishful thinking. It is clearly not true for large subsystems
and/or complex devices.

Are you saying that kernel communities behind large subsystems for
complex devices generally have no idea about what they're doing ? Or
that in a small number of particular cases those communities are
clueless ? Or does that apply to just the maintainer, not the whole
subsystem core developers ? I'd like to better understand the scale of
your claim here.

I don't know how you jumped from saying "the maintainers of the relevant
subsystems" to "kernel communities". I'm talking about maintainers, not
communities.

I wasn't too sure, so that's why I asked. I have also not been very
precise in my previous e-mails. When I mentioned trusting maintainers, I
meant trusting the combined knowledge of the relevant maintainer(s) and
core developer(s) for a subsystema

Unfortunately, the reason for this topic proposed for Maintainer's summit
is that the maintainer and core developers are disagree and there is no way
to resolve it, because it is not technical difference, but a philosophical one.

Having been involved in a similar disagreement, I'm not sure
"philosophical" is the right term. I can't talk about the fwctl issue in
particular as I have only vaguely followed the saga, and I will
therefore not take a side there, but in general I tend to use
"political" instead of "philosophical". The issues of market control,
competition and vendor lock-in vs. empowerment also play important
roles. This makes it even more difficult to discuss the disagreements
openly.

quoted

The number of people that this covers, and how they collectively reach
agreements, very much depends on subsystems.

quoted

There is no way to know everything about everything. In large subsystems,
the stack above kernel is so vast, which makes it impossible to know all
use cases. This is why some words (... good ... whole ...) in your sentence
are not accurate.

So the idea that one maintainer somehow equal to the whole community and
this person can block something for other members of the larger community
is overreaching.

-- 
Regards,

Laurent Pinchart

On Mon, Jul 22, 2024 at 05:13:43PM +0300, Laurent Pinchart wrote:

quoted

Unfortunately, the reason for this topic proposed for Maintainer's summit
is that the maintainer and core developers are disagree and there is no way
to resolve it, because it is not technical difference, but a philosophical one.

Having been involved in a similar disagreement, I'm not sure
"philosophical" is the right term. I can't talk about the fwctl issue in
particular as I have only vaguely followed the saga, and I will
therefore not take a side there, but in general I tend to use
"political" instead of "philosophical". The issues of market control,
competition and vendor lock-in vs. empowerment also play important
roles. This makes it even more difficult to discuss the disagreements
openly.

I've prefered philosophical in this case because I don't view any ill
will on any side here, while I would tend to think the word political
carries a more negative tone.

There isn't really any lock in discussion here, there is some
philosophical disagreement on how much "generalism" should be enforced
by the kernel in some topics and a fear that if things are more
permissive through the kernel then some will loose power to force
their views.

As we don't even agree on how much generalism the kernel should be
enforcing, and what value that even brings to the ecosystem, it is
very much philosophical.

This is a little different than the open shader compiler sorts of
arguments where GPU compilers really underpin a lot of value in the
software stack. We are talking about configuring flash on a device,
and debugging the FW running on the device. There is minimal value to
open source communities with these activities in the first place.

Jason

On Mon, 8 Jul 2024, Dan Williams wrote:

2/ Device passthrough, kernel passing opaque payloads, is already taken
   for granted in many subsystems. USB and HID have "raw" interfaces

Just as a completely random datapoint here: after I implemented hidraw 
inteface long time ago, I was a little bit hesitant about really merging 
it, because there was a general fear that this would shatter the HID 
driver ecosystem, making it difficult for people to find proper drivers 
for their devices, etc.

Turns out that that didn't happen. Drivers for generic devices are still 
implemented properly in the kernel, and hidraw is mostly used for rather 
specific, one-off solutions, where the vendor's business plan is "ship 
this one appliance and forget forever", which doesn't really cause any 
harm to the whole ecosystem.

-- 
Jiri Kosina
SUSE Labs

On Tue, 2024-07-23 at 13:20 +0200, Jiri Kosina wrote:

On Mon, 8 Jul 2024, Dan Williams wrote:

quoted

2/ Device passthrough, kernel passing opaque payloads, is already
taken for granted in many subsystems. USB and HID have "raw"
interfaces

Just as a completely random datapoint here: after I implemented
hidraw inteface long time ago, I was a little bit hesitant about
really merging it, because there was a general fear that this would
shatter the HID driver ecosystem, making it difficult for people to
find proper drivers  for their devices, etc.

The problem with hidraw is that userspace has to understand the device
to use it, but a lot of HID devices (keyboards, mice, serial ports,
etc.) want to fit into an existing ecosystem so they have to have a
kernel driver to avoid having to update all the user applications. 
However, entirely new devices don't have the existing ecosystem
problem.

Turns out that that didn't happen. Drivers for generic devices are
still implemented properly in the kernel, and hidraw is mostly used
for rather specific, one-off solutions, where the vendor's business
plan is "ship this one appliance and forget forever", which doesn't
really cause any harm to the whole ecosystem.

That's not entirely true.  FIDO tokens (the ones Konstantin is
recommending for kernel.org access) are an entire class of devices that
use hidraw and don't have a kernel driver.  There's an array of
manufacturers producing them, but the CTAP specification and its
conformance is what keeps a single user mode driver (which is now
present as a separate implementation in all web browsers and the
userspace libfido2) for all of them.  Fido is definitely not a one off,
but on the other hand, not having a kernel driver doesn't seem to harm
the ecosystem and they can get away with it because there was no
existing device type for them to fit into (except, as you say, an array
of incompatible and short lived USB key tokens which annoyed everyone
by having usability limits due to the oneoffness).

James

On Tue, 23 Jul 2024, James Bottomley wrote:

That's not entirely true.  FIDO tokens (the ones Konstantin is
recommending for kernel.org access) are an entire class of devices that
use hidraw and don't have a kernel driver.  There's an array of
manufacturers producing them, but the CTAP specification and its
conformance is what keeps a single user mode driver (which is now
present as a separate implementation in all web browsers and the
userspace libfido2) for all of them.  

Agreed, but that pretty much underlines my point though.

The ecosystem didn't get shattered as a result of me having created 
hidraw.

libfido2 is on pretty much everyone's machine now (at least for those who 
need it), and people are using that all the time to authenticate to 
kernel.org/Google/Okta/whatnot. No workflow got broken in the process.

-- 
Jiri Kosina
SUSE Labs

On Tue, Jul 23, 2024 at 07:36:24AM -0400, James Bottomley wrote:

On Tue, 2024-07-23 at 13:20 +0200, Jiri Kosina wrote:

quoted

On Mon, 8 Jul 2024, Dan Williams wrote:

quoted

2/ Device passthrough, kernel passing opaque payloads, is already
taken for granted in many subsystems. USB and HID have "raw"
interfaces

Just as a completely random datapoint here: after I implemented
hidraw inteface long time ago, I was a little bit hesitant about
really merging it, because there was a general fear that this would
shatter the HID driver ecosystem, making it difficult for people to
find proper drivers  for their devices, etc.

The problem with hidraw is that userspace has to understand the device
to use it, but a lot of HID devices (keyboards, mice, serial ports,
etc.) want to fit into an existing ecosystem so they have to have a
kernel driver to avoid having to update all the user applications. 
However, entirely new devices don't have the existing ecosystem
problem.

quoted

Turns out that that didn't happen. Drivers for generic devices are
still implemented properly in the kernel, and hidraw is mostly used
for rather specific, one-off solutions, where the vendor's business
plan is "ship this one appliance and forget forever", which doesn't
really cause any harm to the whole ecosystem.

That's not entirely true.  FIDO tokens (the ones Konstantin is
recommending for kernel.org access) are an entire class of devices that
use hidraw and don't have a kernel driver.  There's an array of
manufacturers producing them, but the CTAP specification and its
conformance is what keeps a single user mode driver (which is now
present as a separate implementation in all web browsers and the
userspace libfido2) for all of them.  Fido is definitely not a one off,
but on the other hand, not having a kernel driver doesn't seem to harm
the ecosystem and they can get away with it because there was no
existing device type for them to fit into (except, as you say, an array
of incompatible and short lived USB key tokens which annoyed everyone
by having usability limits due to the oneoffness).

While "userspace drivers" often cause allergic reactions, I think I
won't cause a controversy if I say that we are all used to them in
certain areas. My heart rate will increase if someone proposes replacing
a USB webcam driver with a libusb-based solution, but I don't lose sleep
over the fact that my GPU is mostly controlled by code in Mesa.

What I get from the discussions I've followed or partcipated in over the
years is that the main worry of free software communities is being
forced to use closed-source userspace components, whether that would be
to make the device usable at all, or to achieve decent level of
performance or full feature set. We've been through years of mostly
closed-source GPU support, of printer "windrivers", and quite a few
other horrors. The good news is that we've so far overcome lots (most)
of those challenges. Reverse engineering projects paid off, and so did
working hand-in-hand with industry actors in multiple ways (both openly
and behind the scenes). One could then legitimately ask why we're still
scared.

I can't fully answer that question, but there are two points that I
think are relevant. Note that due to my background and experience, this
will be heavily biased towards consumer and embedded hardware, not data
centre-grade devices. Some technologies from the latter however have a
tendency to migrate to the former over time, so the distinction isn't
necessarily as relevant as one may consider.

The first point is that hardware gets more complicated over time, and in
some markets there's also an increase in the number of vendors and
devices. There's a perceived (whether true or not) danger that we won't
be able to keep up with just reverse engineering and a development model
relying on hobyists. Getting vendors involved is important if we want to
scale.

Second, I think there's a fear of regression. For some categories of
devices, we have made slow but real progress to try and convince the
industry to be more open. This sometimes took a decade of work,
patiently building bridges and creating ecosystems brick by brick. Some
of those ecosystems are sturdy, some not so. Giving pass-through a blank
check will likely have very different effects in different areas. I
don't personally believe it will shatter everything, but I'm convinced
it carries risk in areas where cooperation with vendors is in its
infancy or is fragile for any other reason.

Finally, let's not forget that pass-through APIs are not an all or
nothing option. To cite that example only, DRM requires GPU drivers to
have an open-source userspace implementation to merge the kernel driver,
and the same subsystems strongly pushes for API standardization for
display controllers. We can set different rules for different cases.

-- 
Regards,

Laurent Pinchart

On Wed, 2024-07-24 at 01:22 +0200, Jiri Kosina wrote:

On Tue, 23 Jul 2024, James Bottomley wrote:

quoted

That's not entirely true.  FIDO tokens (the ones Konstantin is
recommending for kernel.org access) are an entire class of devices
that
use hidraw and don't have a kernel driver.  There's an array of
manufacturers producing them, but the CTAP specification and its
conformance is what keeps a single user mode driver (which is now
present as a separate implementation in all web browsers and the
userspace libfido2) for all of them.  

Agreed, but that pretty much underlines my point though.

The ecosystem didn't get shattered as a result of me having created 
hidraw.

Yes, we're in agreement on this.  I was just extrapolating to not every
bypass is inherently evil.

libfido2 is on pretty much everyone's machine now (at least for those
who need it), and people are using that all the time to authenticate
to kernel.org/Google/Okta/whatnot. No workflow got broken in the
process.

Well, there is one use case I can think of that would have the kernel
talking to a fido token (i.e. us having a kernel driver): using it as
the root for trusted and encrypted keys.  It might be very useful for
security features like encrypted device tree or kernel command line
files, or even passing in a private X.509 key to add to the kernel
trusted keyrings or for module signing.   The rush to bypass the kernel
deprived us of thinking about this as an application, but, since the
spec is open, if anyone cares enough, I'm sure it will eventually get
written.

James

On Wed, 2024-07-24 at 23:00 +0300, Laurent Pinchart wrote:
[...]

What I get from the discussions I've followed or partcipated in over
the years is that the main worry of free software communities is
being forced to use closed-source userspace components, whether that
would be to make the device usable at all, or to achieve decent level
of performance or full feature set. We've been through years of
mostly closed-source GPU support, of printer "windrivers", and quite
a few other horrors. The good news is that we've so far overcome lots
(most) of those challenges. Reverse engineering projects paid off,
and so did working hand-in-hand with industry actors in multiple ways
(both openly and behind the scenes). One could then legitimately ask
why we're still scared.

I don't think I am.  We're mostly fully capable of expounding at length
on the business rationale for being open if the thing they're hiding
isn't much of a differentiator anyway (or they're simply hiding it to
try to retain some illusion of control), so we shouldn't have any fear
of being able to make our case in language business people understand.

I also think this fear is partly a mindset problem on our part.  We
came out of the real fight for openness and we do embrace things like a
licence that forces open code (GPL) and symbols that discourage
proprietary drivers (EXPORT_SYMBOL_GPL), so we've somewhat drunk the
FSF coolaid that if we don't stand over manufacturers every second and
force them they'll slide back to their old proprietary ways.  However,
if you look at the entirely permissive ecosystem that grew up after we
did (openstack, docker, kubernetes, etc.) they don't have any such fear
and yet they still have large amounts of uncompelled openness and give
back.

I can't fully answer that question, but there are two points that I
think are relevant. Note that due to my background and experience,
this will be heavily biased towards consumer and embedded hardware,
not data centre-grade devices. Some technologies from the latter
however have a tendency to migrate to the former over time, so the
distinction isn't necessarily as relevant as one may consider.

The first point is that hardware gets more complicated over time, and
in some markets there's also an increase in the number of vendors and
devices. There's a perceived (whether true or not) danger that we
won't be able to keep up with just reverse engineering and a
development model relying on hobyists. Getting vendors involved is
important if we want to scale.

Yes, but there are lots of not very useful complex devices being
produced every day that fail to capture market share.  Not having
reverse engineered drivers for them is no real loss.  If a device does
gain market share, it gains a huge pool of users some of whom become
interested in reverse engineering, so I think market forces actually
work in our favour: we get reverse engineering mostly where the devices
are actually interesting and capture market share.  It's self scaling.

Second, I think there's a fear of regression. For some categories of
devices, we have made slow but real progress to try and convince the
industry to be more open. This sometimes took a decade of work,
patiently building bridges and creating ecosystems brick by brick.
Some of those ecosystems are sturdy, some not so. Giving pass-through
a blank check will likely have very different effects in different
areas. I don't personally believe it will shatter everything, but I'm
convinced it carries risk in areas where cooperation with vendors is
in its infancy or is fragile for any other reason.

I also think we're on the rise in this space.  Since most cloud
workloads are on Linux, there's huge market pressure on most "found in
the cloud" devices (like accelerators and GPUs) to have an easy to
consume Linux story.  Nvidia is a case in point.  When it only cared
about fast games on some other OS, we get shafted with a proprietary
graphics drivers.  Now it's under pressure to be the number one AI
accelerator provider for the cloud it's suddenly wondering about open
source drivers to make adoption easier.

Finally, let's not forget that pass-through APIs are not an all or
nothing option. To cite that example only, DRM requires GPU drivers
to have an open-source userspace implementation to merge the kernel
driver, and the same subsystems strongly pushes for API
standardization for display controllers. We can set different rules
for different cases.

I certainly think we can afford to experiment here, yes.

James

On Wed, 24 Jul 2024 16:37:21 -0400
James Bottomley [off-list ref] wrote:

quoted

The first point is that hardware gets more complicated over time, and
in some markets there's also an increase in the number of vendors and
devices. There's a perceived (whether true or not) danger that we
won't be able to keep up with just reverse engineering and a
development model relying on hobyists. Getting vendors involved is
important if we want to scale.  

Yes, but there are lots of not very useful complex devices being
produced every day that fail to capture market share.  Not having
reverse engineered drivers for them is no real loss.  If a device does
gain market share, it gains a huge pool of users some of whom become
interested in reverse engineering, so I think market forces actually
work in our favour: we get reverse engineering mostly where the devices
are actually interesting and capture market share.  It's self scaling.

I agree with this. If a small vendor with low market share has a
proprietary device where they could easily port to Linux via a pass
through, they may do that. But if they don't have that, and require
engineering resources to port to Linux, they will not bother. As they would
only care about the Windows market. So the device becomes useless for the
Linux system. There's not enough Linux users to make a small vendor care
about losing us. That just shrinks the number of devices that are available
to Linux.

My guess is that vendors want to write one piece of code. If it they only
need to modify a small portion to get it to another OS, they would do that.
But if it takes more effort than that, there's not enough cost incentive to
bother.

For devices with a larger market share, it would make it more worth their
while to open source their work, otherwise there's more incentive for us to
reverse engineer it anyway.

-- Steve

On Wed, Jul 24, 2024 at 10:02 PM Laurent Pinchart
[off-list ref] wrote:

While "userspace drivers" often cause allergic reactions, I think I
won't cause a controversy if I say that we are all used to them in
certain areas. My heart rate will increase if someone proposes replacing
a USB webcam driver with a libusb-based solution, but I don't lose sleep
over the fact that my GPU is mostly controlled by code in Mesa.

I think the key point here is that USB webcams follow a standard, and
GPUs don't.

What I get from the discussions I've followed or partcipated in over the
years is that the main worry of free software communities is being
forced to use closed-source userspace components, whether that would be
to make the device usable at all, or to achieve decent level of
performance or full feature set. We've been through years of mostly
closed-source GPU support, of printer "windrivers", and quite a few
other horrors. The good news is that we've so far overcome lots (most)
of those challenges. Reverse engineering projects paid off, and so did
working hand-in-hand with industry actors in multiple ways (both openly
and behind the scenes). One could then legitimately ask why we're still
scared.

It would be great to define what are the free software communities
here. Distros and final users are also "free software communities" and
they do not care about niche use cases covered by proprietary
software.
They only care (and should care) about normal workflows.

I can't fully answer that question, but there are two points that I
think are relevant. Note that due to my background and experience, this
will be heavily biased towards consumer and embedded hardware, not data
centre-grade devices. Some technologies from the latter however have a
tendency to migrate to the former over time, so the distinction isn't
necessarily as relevant as one may consider.

The first point is that hardware gets more complicated over time, and in
some markets there's also an increase in the number of vendors and
devices. There's a perceived (whether true or not) danger that we won't
be able to keep up with just reverse engineering and a development model
relying on hobyists. Getting vendors involved is important if we want to
scale.

If we want vendors involved, we need to build an ecosystem where they
feel invited.

We should not take as hostages our users and impose rules on how they
should build or even sell their product.

Second, I think there's a fear of regression. For some categories of
devices, we have made slow but real progress to try and convince the
industry to be more open. This sometimes took a decade of work,
patiently building bridges and creating ecosystems brick by brick. Some
of those ecosystems are sturdy, some not so. Giving pass-through a blank
check will likely have very different effects in different areas. I
don't personally believe it will shatter everything, but I'm convinced
it carries risk in areas where cooperation with vendors is in its
infancy or is fragile for any other reason.

We control what is accepted and what is not. We just need clear rules,
to avoid regressions like:
- For areas where there is a standard (NVME, UVC,...) most of the
drivers must be in-kernel, and use generic system calls.
- For areas with no standard, custom system calls are allowed, and
some part of the driver can be in userspace.
- To land a driver, there must be a full open source stack capable of
using it for standard use cases.
- If there is an established open source stack (mesa, openVINO,
libcamera...), the open source stack must be based on it.
- Vendor passthrough mechanisms are allowed for niche use cases or
development/experimentation.

I believe those rules are already in place in some subsystems. We just
have to agree what rules should apply to all the kernel by policy.

We can agree that this kind of discussion is done better face to face.

Regards!

Finally, let's not forget that pass-through APIs are not an all or
nothing option. To cite that example only, DRM requires GPU drivers to
have an open-source userspace implementation to merge the kernel driver,
and the same subsystems strongly pushes for API standardization for
display controllers. We can set different rules for different cases.

--
Regards,

Laurent Pinchart

--
Ricardo Ribalda

Hi Ricardo,

It would be great to define what are the free software communities
here. Distros and final users are also "free software communities" and
they do not care about niche use cases covered by proprietary
software.
They only care (and should care) about normal workflows.

Don't you think there are distros and final users who do care about
proprietary influences, in general. For sure, though, I think they
should not be told what they should care about?

If we want vendors involved, we need to build an ecosystem where they
feel invited.

Definitely. Invited as in "you are welcome to work with us on this",
though. Not as in "come here and do what you want". This is why we have
a coding-style etc...

We should not take as hostages our users and impose rules on how they
should build or even sell their product.

Right, it should be a plain buisness decision. I totally agree with your
"clear rules" request below. And yeah, face to face is probably best
suited.

- Vendor passthrough mechanisms are allowed for niche use cases or
development/experimentation.

Problem with "niche" is that it can grow really big. See Linux :)

All the best,

   Wolfram

On Thu, Jul 25, 2024 at 11:26:38AM +0200, Ricardo Ribalda Delgado wrote:

On Wed, Jul 24, 2024 at 10:02 PM Laurent Pinchart
[off-list ref] wrote:

<...>

It would be great to define what are the free software communities
here. Distros and final users are also "free software communities" and
they do not care about niche use cases covered by proprietary
software.

Are you certain about that?

They only care (and should care) about normal workflows.

What is a normal workflow?
Does it mean that if user bought something very expensive he
should not be able to use it with free software, because his
usage is different from yours?

Thanks

On Thu, Jul 25, 2024 at 2:23 PM Leon Romanovsky [off-list ref] wrote:

On Thu, Jul 25, 2024 at 11:26:38AM +0200, Ricardo Ribalda Delgado wrote:

quoted

On Wed, Jul 24, 2024 at 10:02 PM Laurent Pinchart
[off-list ref] wrote:

<...>

quoted

It would be great to define what are the free software communities
here. Distros and final users are also "free software communities" and
they do not care about niche use cases covered by proprietary
software.

Are you certain about that?

As a user, and as an open source Distro developer I have a small hint.
But you could also ask users what they think about not being able to
use their notebook's cameras. The last time that I could not use some
basic hardware from a notebook with Linux was 20 years ago.

quoted

They only care (and should care) about normal workflows.

What is a normal workflow?
Does it mean that if user bought something very expensive he
should not be able to use it with free software, because his
usage is different from yours?

Thanks

It means that we should not block the standard usage for 99% of the
population just because 1% of the users cannot do something fancy with
their device.

Let me give you an example. When I buy a camera I want to be able to
do Video Conferencing and take some static photos of documents. I do
not care about: automatic makeup, AI generated background, unicorn
filters, eyes recentering... But we need to give a way to vendors to
implement those things closely, without the marketing differentiators,
vendors have zero incentive to invest in Linux, and that affects all
the population.

This challenge seems to be solved for GPUs. I am using my AMD GPU
freely and my nephew can install the amdgpu-pro proprietary user space
driver to play duke nukem (or whatever kids play now) at 2000 fps.

There are other other subsystems that allow vendor passthrough and
their ecosystem has not collapsed.

Can we have some general guidance of what is acceptable? Can we define
together the "normal workflow" and focus on a *full* open source
implementation of that?

-- 
Ricardo Ribalda

On Thu, Jul 25, 2024 at 03:02:13PM +0200, Ricardo Ribalda Delgado wrote:

On Thu, Jul 25, 2024 at 2:23 PM Leon Romanovsky [off-list ref] wrote:

quoted

On Thu, Jul 25, 2024 at 11:26:38AM +0200, Ricardo Ribalda Delgado wrote:

quoted

On Wed, Jul 24, 2024 at 10:02 PM Laurent Pinchart
[off-list ref] wrote:

<...>

quoted

It would be great to define what are the free software communities
here. Distros and final users are also "free software communities" and
they do not care about niche use cases covered by proprietary
software.

Are you certain about that?

As a user, and as an open source Distro developer I have a small hint.
But you could also ask users what they think about not being able to
use their notebook's cameras. The last time that I could not use some
basic hardware from a notebook with Linux was 20 years ago.

Lucky you, I still have consumer hardware (speaker) that doesn't work
with Linux, and even now, there is basic hardware in my current
laptop (HP docking station) that doesn't work reliably in Linux.

quoted

quoted

They only care (and should care) about normal workflows.

What is a normal workflow?
Does it mean that if user bought something very expensive he
should not be able to use it with free software, because his
usage is different from yours?

Thanks

It means that we should not block the standard usage for 99% of the
population just because 1% of the users cannot do something fancy with
their device.

Right, the problem is that in some areas the statistics slightly different.
99% population is blocked because 1% of the users don't need it and
don't think that it is "normal" flow.

Let me give you an example. When I buy a camera I want to be able to
do Video Conferencing and take some static photos of documents. I do
not care about: automatic makeup, AI generated background, unicorn
filters, eyes recentering... But we need to give a way to vendors to
implement those things closely, without the marketing differentiators,
vendors have zero incentive to invest in Linux, and that affects all
the population.

This challenge seems to be solved for GPUs. I am using my AMD GPU
freely and my nephew can install the amdgpu-pro proprietary user space
driver to play duke nukem (or whatever kids play now) at 2000 fps.

There are other other subsystems that allow vendor passthrough and
their ecosystem has not collapsed.

Yes, I completely agree with you on that.

Can we have some general guidance of what is acceptable? Can we define
together the "normal workflow" and focus on a *full* open source
implementation of that?

I don't think that is possible to define "normal workflow". Requirement
to have open-source counterpart to everything exposed through UAPI is a
valid one. I'm all for that.

Thanks

-- 
Ricardo Ribalda

On Thu, Jul 25, 2024 at 04:20:35PM +0300, Leon Romanovsky wrote:

On Thu, Jul 25, 2024 at 03:02:13PM +0200, Ricardo Ribalda Delgado wrote:

quoted

As a user, and as an open source Distro developer I have a small hint.
But you could also ask users what they think about not being able to
use their notebook's cameras. The last time that I could not use some
basic hardware from a notebook with Linux was 20 years ago.

Lucky you, I still have consumer hardware (speaker) that doesn't work
with Linux, and even now, there is basic hardware in my current
laptop (HP docking station) that doesn't work reliably in Linux.

FWIW for most audio issues (especially built in stuff) with laptops if
you report it upstream it'll generally be relatively easy to quirk.
Unfortunately it's idiomatic for ACPI systems to quirk off DMI
information for almost everything which means a constant stream of per
system quirks for subsystems like audio.

On Thu, 25 Jul 2024 15:02:13 +0200
Ricardo Ribalda Delgado [off-list ref] wrote:

As a user, and as an open source Distro developer I have a small hint.
But you could also ask users what they think about not being able to
use their notebook's cameras. The last time that I could not use some
basic hardware from a notebook with Linux was 20 years ago.

FYI, I love my Dell XPS 13. But it's old. The battery died once and I had
to replace it. I would love to buy a new one, but the new one's webcam is
not supported by Linux. I'm just waiting for when this will be resolved. :-p

-- Steve

On Thu, Jul 25, 2024 at 02:29:36PM +0100, Mark Brown wrote:

On Thu, Jul 25, 2024 at 04:20:35PM +0300, Leon Romanovsky wrote:

quoted

On Thu, Jul 25, 2024 at 03:02:13PM +0200, Ricardo Ribalda Delgado wrote:

quoted

quoted

As a user, and as an open source Distro developer I have a small hint.
But you could also ask users what they think about not being able to
use their notebook's cameras. The last time that I could not use some
basic hardware from a notebook with Linux was 20 years ago.

quoted

Lucky you, I still have consumer hardware (speaker) that doesn't work
with Linux, and even now, there is basic hardware in my current
laptop (HP docking station) that doesn't work reliably in Linux.

FWIW for most audio issues (especially built in stuff) with laptops if
you report it upstream it'll generally be relatively easy to quirk.
Unfortunately it's idiomatic for ACPI systems to quirk off DMI
information for almost everything which means a constant stream of per
system quirks for subsystems like audio.

It is Jabra USB speaker. One day, I will have time to debug it :).

Thanks

On Thu, 2024-07-25 at 17:18 +0300, Leon Romanovsky wrote:

On Thu, Jul 25, 2024 at 02:29:36PM +0100, Mark Brown wrote:

quoted

On Thu, Jul 25, 2024 at 04:20:35PM +0300, Leon Romanovsky wrote:

quoted

On Thu, Jul 25, 2024 at 03:02:13PM +0200, Ricardo Ribalda Delgado
wrote:

quoted

quoted

As a user, and as an open source Distro developer I have a
small hint.  But you could also ask users what they think about
not being able to use their notebook's cameras. The last time
that I could not use some basic hardware from a notebook with
Linux was 20 years ago.

quoted

Lucky you, I still have consumer hardware (speaker) that doesn't
work with Linux, and even now, there is basic hardware in my
current laptop (HP docking station) that doesn't work reliably in
Linux.

FWIW for most audio issues (especially built in stuff) with laptops
if you report it upstream it'll generally be relatively easy to
quirk. Unfortunately it's idiomatic for ACPI systems to quirk off
DMI information for almost everything which means a constant stream
of per system quirks for subsystems like audio.

It is Jabra USB speaker. One day, I will have time to debug it :).

Those actually do work with Linux.  It's the speakers Plumbers has
provided to BoF/Hack sessions that needed remote attendees.  There was
a pulseaudio issue a while ago that any Mic/Speaker combination that
exported HFP instead of HSP didn't work, but that's fixed upstream (but
you might need to upgrade your pulseaudio).

James

On Thu, Jul 25, 2024 at 10:22:48AM -0400, James Bottomley wrote:

On Thu, 2024-07-25 at 17:18 +0300, Leon Romanovsky wrote:

quoted

On Thu, Jul 25, 2024 at 02:29:36PM +0100, Mark Brown wrote:

quoted

On Thu, Jul 25, 2024 at 04:20:35PM +0300, Leon Romanovsky wrote:

quoted

On Thu, Jul 25, 2024 at 03:02:13PM +0200, Ricardo Ribalda Delgado
wrote:

quoted

quoted

As a user, and as an open source Distro developer I have a
small hint.  But you could also ask users what they think about
not being able to use their notebook's cameras. The last time
that I could not use some basic hardware from a notebook with
Linux was 20 years ago.

quoted

Lucky you, I still have consumer hardware (speaker) that doesn't
work with Linux, and even now, there is basic hardware in my
current laptop (HP docking station) that doesn't work reliably in
Linux.

FWIW for most audio issues (especially built in stuff) with laptops
if you report it upstream it'll generally be relatively easy to
quirk. Unfortunately it's idiomatic for ACPI systems to quirk off
DMI information for almost everything which means a constant stream
of per system quirks for subsystems like audio.

It is Jabra USB speaker. One day, I will have time to debug it :).

Those actually do work with Linux.  It's the speakers Plumbers has
provided to BoF/Hack sessions that needed remote attendees.  There was
a pulseaudio issue a while ago that any Mic/Speaker combination that
exported HFP instead of HSP didn't work, but that's fixed upstream (but
you might need to upgrade your pulseaudio).

I don't want to hijack this thread, but I tried now my two Jabra
speakers (one is used and another is brand new which I got from IT)
and both don't work on my FC40 laptop.

Thanks

James

On Wed, Jul 24, 2024 at 04:37:21PM -0400, James Bottomley wrote:

On Wed, 2024-07-24 at 23:00 +0300, Laurent Pinchart wrote:
[...]

quoted

What I get from the discussions I've followed or partcipated in over
the years is that the main worry of free software communities is
being forced to use closed-source userspace components, whether that
would be to make the device usable at all, or to achieve decent level
of performance or full feature set. We've been through years of
mostly closed-source GPU support, of printer "windrivers", and quite
a few other horrors. The good news is that we've so far overcome lots
(most) of those challenges. Reverse engineering projects paid off,
and so did working hand-in-hand with industry actors in multiple ways
(both openly and behind the scenes). One could then legitimately ask
why we're still scared.

I don't think I am.  We're mostly fully capable of expounding at length
on the business rationale for being open if the thing they're hiding
isn't much of a differentiator anyway (or they're simply hiding it to
try to retain some illusion of control), so we shouldn't have any fear
of being able to make our case in language business people understand.

I also think this fear is partly a mindset problem on our part.  We
came out of the real fight for openness and we do embrace things like a
licence that forces open code (GPL) and symbols that discourage
proprietary drivers (EXPORT_SYMBOL_GPL), so we've somewhat drunk the
FSF coolaid that if we don't stand over manufacturers every second and
force them they'll slide back to their old proprietary ways.  However,
if you look at the entirely permissive ecosystem that grew up after we
did (openstack, docker, kubernetes, etc.) they don't have any such fear
and yet they still have large amounts of uncompelled openness and give
back.

I don't think those are necessarily relevant examples, as far as device
pass-through goes. Vendors have many times reverted to proprietary ways,
and they still do, at least in the areas of the kernel I'm most active
in. I've seen first hand a large SoC vendor very close to opening a
significant part of their camera stack and changing their mind at the
last minute when they heard they could possibly merge their code through
a different subsystem with a pass-through blank cheque.

I'm willing to believe it can be different in other areas, which may
partly explain why different subsystems and different developers have
different biases and have trouble understand each other's point of view.

quoted

I can't fully answer that question, but there are two points that I
think are relevant. Note that due to my background and experience,
this will be heavily biased towards consumer and embedded hardware,
not data centre-grade devices. Some technologies from the latter
however have a tendency to migrate to the former over time, so the
distinction isn't necessarily as relevant as one may consider.

The first point is that hardware gets more complicated over time, and
in some markets there's also an increase in the number of vendors and
devices. There's a perceived (whether true or not) danger that we
won't be able to keep up with just reverse engineering and a
development model relying on hobyists. Getting vendors involved is
important if we want to scale.

Yes, but there are lots of not very useful complex devices being
produced every day that fail to capture market share.  Not having
reverse engineered drivers for them is no real loss.  If a device does
gain market share, it gains a huge pool of users some of whom become
interested in reverse engineering, so I think market forces actually
work in our favour: we get reverse engineering mostly where the devices
are actually interesting and capture market share.  It's self scaling.

I can't agree with that, sorry. Not only is the difficulty to
reverse-engineer some classes of devices increasing, but saying that
only devices that make it to the top of the market share chart are worth
considering will leave many users on the side of the road.

quoted

Second, I think there's a fear of regression. For some categories of
devices, we have made slow but real progress to try and convince the
industry to be more open. This sometimes took a decade of work,
patiently building bridges and creating ecosystems brick by brick.
Some of those ecosystems are sturdy, some not so. Giving pass-through
a blank check will likely have very different effects in different
areas. I don't personally believe it will shatter everything, but I'm
convinced it carries risk in areas where cooperation with vendors is
in its infancy or is fragile for any other reason.

I also think we're on the rise in this space.  Since most cloud
workloads are on Linux, there's huge market pressure on most "found in
the cloud" devices (like accelerators and GPUs) to have an easy to
consume Linux story.  Nvidia is a case in point.  When it only cared
about fast games on some other OS, we get shafted with a proprietary
graphics drivers.  Now it's under pressure to be the number one AI
accelerator provider for the cloud it's suddenly wondering about open
source drivers to make adoption easier.

I can't comment on Nvidia and their inference engines in particular. The
server market may be in a better position that the consumer and embedded
market, and if that's the case, I'm happy for the servers. That doesn't
solve the issues in other markets though.

quoted

Finally, let's not forget that pass-through APIs are not an all or
nothing option. To cite that example only, DRM requires GPU drivers
to have an open-source userspace implementation to merge the kernel
driver, and the same subsystems strongly pushes for API
standardization for display controllers. We can set different rules
for different cases.

I certainly think we can afford to experiment here, yes.

-- 
Regards,

Laurent Pinchart

On Thu, Jul 25, 2024 at 10:31:25PM +0300, Laurent Pinchart wrote:

I don't think those are necessarily relevant examples, as far as device
pass-through goes. Vendors have many times reverted to proprietary ways,
and they still do, at least in the areas of the kernel I'm most active
in. I've seen first hand a large SoC vendor very close to opening a
significant part of their camera stack and changing their mind at the
last minute when they heard they could possibly merge their code through
a different subsystem with a pass-through blank cheque.

If someone came with a fully open source framework for (say) some
camera, with a passthrough kernel driver design, would you reject it
soley because it is passthrough based and you are scared that
something else will use it to do something not open source?

I wouldn't agree with that position, I think denying users useful open
source solutions out of fear is not what Linux should be doing.

Jason

On Thu, Jul 25, 2024 at 04:20:35PM +0300, Leon Romanovsky wrote:

On Thu, Jul 25, 2024 at 03:02:13PM +0200, Ricardo Ribalda Delgado wrote:

quoted

On Thu, Jul 25, 2024 at 2:23 PM Leon Romanovsky wrote:

quoted

On Thu, Jul 25, 2024 at 11:26:38AM +0200, Ricardo Ribalda Delgado wrote:

quoted

On Wed, Jul 24, 2024 at 10:02 PM Laurent Pinchart wrote:

<...>

quoted

It would be great to define what are the free software communities
here. Distros and final users are also "free software communities" and
they do not care about niche use cases covered by proprietary
software.

Are you certain about that?

As a user, and as an open source Distro developer I have a small hint.
But you could also ask users what they think about not being able to
use their notebook's cameras. The last time that I could not use some
basic hardware from a notebook with Linux was 20 years ago.

Lucky you, I still have consumer hardware (speaker) that doesn't work
with Linux, and even now, there is basic hardware in my current
laptop (HP docking station) that doesn't work reliably in Linux.

quoted

quoted

quoted

They only care (and should care) about normal workflows.

What is a normal workflow?
Does it mean that if user bought something very expensive he
should not be able to use it with free software, because his
usage is different from yours?

Thanks

It means that we should not block the standard usage for 99% of the
population just because 1% of the users cannot do something fancy with
their device.

Right, the problem is that in some areas the statistics slightly different.
99% population is blocked because 1% of the users don't need it and
don't think that it is "normal" flow.

quoted

Let me give you an example. When I buy a camera I want to be able to
do Video Conferencing and take some static photos of documents. I do
not care about: automatic makeup, AI generated background, unicorn
filters, eyes recentering... But we need to give a way to vendors to
implement those things closely, without the marketing differentiators,
vendors have zero incentive to invest in Linux, and that affects all
the population.

I've seen these kind of examples being repeatedly given in discussions
related to camera ISP support in Linux. They are very misleading. These
are not the kind of features that are relevant for the device
pass-through discussion these day. Those are high-level use cases
implemented in userspace, and vendors can ship any closed-source
binaries they want there. What I care about is the features exposed by
the kernel to userspace API.

quoted

This challenge seems to be solved for GPUs. I am using my AMD GPU
freely and my nephew can install the amdgpu-pro proprietary user space
driver to play duke nukem (or whatever kids play now) at 2000 fps.

There are other other subsystems that allow vendor passthrough and
their ecosystem has not collapsed.

Yes, I completely agree with you on that.

quoted

Can we have some general guidance of what is acceptable? Can we define
together the "normal workflow" and focus on a *full* open source
implementation of that?

I don't think that is possible to define "normal workflow". Requirement
to have open-source counterpart to everything exposed through UAPI is a
valid one. I'm all for that.

That's my current opinion as well, as least when it comes to the kernel
areas I mostly work with.

-- 
Regards,

Laurent Pinchart

Hi Ricardo,

On Mon, Jul 22, 2024 at 01:56:11PM +0200, Ricardo Ribalda Delgado wrote:

On Mon, Jul 22, 2024 at 1:18 PM Laurent Pinchart wrote:

quoted

On Mon, Jul 22, 2024 at 12:42:52PM +0200, Ricardo Ribalda Delgado wrote:

quoted

On Sun, Jul 21, 2024 at 9:25 PM Laurent Pinchart wrote:

quoted

On Tue, Jul 09, 2024 at 03:15:13PM -0700, Dan Williams wrote:

quoted

James Bottomley wrote:

quoted

quoted

The upstream discussion has yielded the full spectrum of positions on
device specific functionality, and it is a topic that needs cross-
kernel consensus as hardware increasingly spans cross-subsystem
concerns. Please consider it for a Maintainers Summit discussion.

I'm with Greg on this ... can you point to some of the contrary
positions?

This thread has that discussion:

http://lore.kernel.org/0-v1-9912f1a11620+2a-fwctl_jgg@nvidia.com

I do not want to speak for others on the saliency of their points, all I
can say is that the contrary positions have so far not moved me to drop
consideration of fwctl for CXL.

Where CXL has a Command Effects Log that is a reasonable protocol for
making decisions about opaque command codes, and that CXL already has a
few years of experience with the commands that *do* need a Linux-command
wrapper.

Some open questions from that thread are: what does it mean for the fate
of a proposal if one subsystem Acks the ABI and another Naks it for a
device that crosses subsystem functionality? Would a cynical hardware
response just lead to plumbing an NVME admin queue, or CXL mailbox to
get device-specific commands past another subsystem's objection?

My default answer would be to trust the maintainers of the relevant
subsystems (or try to convince them when you disagree :-)). Not only
should they know the technical implications best, they should also have
a good view of the whole vertical stack, and the implications of
pass-through for their ecosystem. This may result in a single NAK
overriding ACKs, but we could also try to find technical solutions when
we'll face such issues, to enforce different sets of rules for the
different functions of a device.

Subsystem hopping is something we're recently noticed for camera ISPs,
where a vendor wanted to move from V4L2 to DRM. Technical reasons for
doing so were given, and they were (in my opinion) rather excuses. The
unspoken real (again in my opinion) reason was to avoid documenting the
firmware interface and ship userspace binary blobs with no way for free
software to use all the device's features. That's something we have been
fighting against for years, trying to convince vendors that they can
provide better and more open camera support without the world
collapsing, with increasing success recently. Saying amen to
pass-through in this case would be a huge step back that would hurt
users and the whole ecosystem in the short and long term.

In my view, DRM is a more suitable model for complex ISPs than V4L2:

I know we disagree on this topic :-) I'm sure we'll continue the
conversation, but I think the technical discussion likely belongs to a
different mail thread.

quoted

- Userspace Complexity: ISPs demand a highly complex and evolving API,
similar to Vulkan or OpenGL. Applications typically need a framework
like libcamera to utilize ISPs effectively, much like Mesa for
graphics cards.

- Lack of Standardization: There's no universal standard for ISPs;
each vendor implements unique features and usage patterns. DRM
addresses this through vendor-specific IOCTLs

- Proprietary Architectures: Vendors often don't fully disclose their
hardware architectures. DRM cleverly only necessitates a Mesa
implementation, not comprehensive documentation.

This point isn't technical and is more on-topic for this mail thread.

V4L2 doesn't require hundreds of pages of comprehensive documentation in
text form. An open-source userspace implementation that covers the
feature set exposed by the driver is acceptable in place of
documentation (provided, of course, that the userspace code wouldn't be
deliberately obfuscated). This is similar in spirit to the rule for GPU
DRM drivers.

In DRM vendors typically define a custom IOCTL per driver to pass
command buffers.
Only the command buffer structure, and a mesa implementation using
that command buffer to support the standard features is required.

In V4l2 custom IOCTLs are discouraged. Random command buffers cannot
be passed from userspace, they are typically formed in the driver from
a strictly checked struct.

V4L2 has a mechanism to pass buffers between userspace and kernelspace,
and that mechanism is used in mainline drivers to pass camera ISP
parameters. They're not called "command buffers" but that's just a
difference in terminology. The technical means to pass command buffers
to the driver is thus there, I see no meaningful difference with DRM.
Where things can differ is in the contents of those buffers, and the
requirements for documentation or open userspace implementations, but
that's not a technical question.

quoted

quoted

Our current approach of pushing back against vendors, instead of
seeking compromise, has resulted in the vast majority of the market
(99% if not more) relying on out-of-tree drivers. This leaves users
with no options for utilizing their cameras outside of Android.

DRM allows a hybrid model, where:
- Open Source Foundation: Standard use cases are covered by a fully
open-source stack.
- Vendor Differentiation: Vendors retain the freedom to implement
proprietary features (e.g., automatic makeup) as closed source.

V4L2 does as well, you can implement all kind of closed-source ISP
control algorithms in userspace, as long as there's an open-source
implementation that exercises the same hardware features. A good analogy

Is it really mandatory to have an open-source 3A algorithm? I thought
defining the input and output from the algorithm was good enough.

What really matters is documenting the ISP parameters with enough
details to allow for the implementation of open-source userspace code.
Once you have that, 3A is quite simple. You can refine it (especially
AWB) to great length, for instance using NPUs to compute parameters, and
there's absolutely no issue with such userspace implementations being
closed.

In practice, some vendors prefer documenting the parameters by writing
an open-source userspace implementation, partly maybe because developers
are more familiar writing code than formal documentation. I would be
fine either way, as long as there's enough information to make use of
the ISP.

AFAIK for some time there was no ipu3 open source algorithm, and the
driver has been upstream.

It sneaked in before we realized we had to enforce rules :-) That's
actually a good example, when we wrote open-source userspace support, we
realized that the level of documentation included in the IPU3 kernel
header was nowhere close to what was needed to make use of the device.

quoted

for people less familiar with ISPs is shader compilers, GPU vendors are
free to ship closed-source implementations that include more
optimizations, as long as the open-source, less optimized implementation
covers the same GPU ISA, so that open-source developers can also work on
optimizing it.

I believe a more accurate description is that in v4l2 is that we
expect that all the registers, device architecture and behaviour to be
documented and accessed with standard IOCTLs. Anything not documented
cannot be accessed by userspace.

In DRM their concern is that there is a fully open source
implementation that the user can use. Vendors have custom IOCTLs and
they can offer proprietary software for some use cases.

Custom ioctls are not closed secrets in DRM, so comparing custom ioctls
vs. standard ioctls isn't very relevant to this discussion. I really
don't see how this would be about ioctls, it's about making the featured
exposed by the drivers, through whatever means a particular subsystem
allows, usable by open userspace.

quoted

Thinking that DRM would offer a free pass-through path compared to V4L2
doesn't seem realistic to me. Both subsystems will have similar rules.

DRM does indeed allow vendors to pass random command buffers and they
will be sent to the hardware. We cannot do that in v4l2.

You can pass a command buffer to a V4L2 device and have the driver send
it to the device firmware (ISPs using real command buffers usually run a
firmware). If you want the driver to be merged upstream, you have to
document the command buffer in enough details.

I might be wrong, but GPU drivers do not deeply inspect the command
buffers to make sure that they do not use any feature not covered by
mesa.

That's correct, but I don't think that's relevant. The GPU market has
GLSL and Vulkan. An open-source compliant implementation will end up
exercising a very very large part of the device ISA, command submission
mechanism and synchronization primitives, if not all of it. There's
little a vendor would keep under the hood and use in closed-source
userspace only. For cameras, there's no standard userspace API that
covers by design a very large part of what is the ISP equivalent of a
GPU ISA. Even "command buffers" are not a proper description of the
parameters the vast majority of ISPs consume.

quoted

quoted

This approach would allow billions of users to access their hardware
more securely and with in-tree driver support. Our current stubborn
pursuit of an idealistic goal has already negatively impacted both
users and the ecosystem.

The late wins, in my opinion, cannot scale to the consumer market, and
Linux will remain a niche market for ISPs.

If such a hybrid model goes against Linux goals, this is something
that should be agreed upon by the whole community, so we have the same
criteria for all subsystems.

-- 
Regards,

Laurent Pinchart

Hi Jason,

On Thu, Jul 25, 2024 at 04:43:14PM -0300, Jason Gunthorpe wrote:

On Thu, Jul 25, 2024 at 10:31:25PM +0300, Laurent Pinchart wrote:

quoted

I don't think those are necessarily relevant examples, as far as device
pass-through goes. Vendors have many times reverted to proprietary ways,
and they still do, at least in the areas of the kernel I'm most active
in. I've seen first hand a large SoC vendor very close to opening a
significant part of their camera stack and changing their mind at the
last minute when they heard they could possibly merge their code through
a different subsystem with a pass-through blank cheque.

If someone came with a fully open source framework for (say) some
camera,

We have such a framework, it's called libcamera :-) Multiple vendors are
already collaborating.

with a passthrough kernel driver design, would you reject it
soley because it is passthrough based and you are scared that
something else will use it to do something not open source?

It depends what "passthrough kernel driver design" means. If it means
accessing the PCI registers directly from userspace, yes. That's what X
used to do before KMS, and I'm glad it's now a distant past.

If it means a kernel driver that takes the majority of its runtime
parameters from a buffer blob assembled by userspace, while controlling
clocks, power domains and performing basic validation in kernelspace,
then I've already acked multiple drivers with such a design, exactly
because they have open-source userspace that doesn't try to keep many
device features proprietary and usable by closed-source userspace only.

I wouldn't agree with that position, I think denying users useful open
source solutions out of fear is not what Linux should be doing.

-- 
Regards,

Laurent Pinchart

On Thu, Jul 25, 2024 at 11:07:03PM +0300, Laurent Pinchart wrote:

If it means a kernel driver that takes the majority of its runtime
parameters from a buffer blob assembled by userspace, while controlling
clocks, power domains and performing basic validation in kernelspace,
then I've already acked multiple drivers with such a design, exactly
because they have open-source userspace that doesn't try to keep many
device features proprietary and usable by closed-source userspace only.

This also pretty much matches where we are with RDMA as well. Lots and
lots of stuff in userspace, but lots and lots is open. We've been able
to keep open kernel and userspace drivers quite well.

Thanks,
Jason

Hi Laurent

On Thu, Jul 25, 2024 at 9:44 PM Laurent Pinchart
[off-list ref] wrote:

On Thu, Jul 25, 2024 at 04:20:35PM +0300, Leon Romanovsky wrote:

quoted

On Thu, Jul 25, 2024 at 03:02:13PM +0200, Ricardo Ribalda Delgado wrote:

quoted

On Thu, Jul 25, 2024 at 2:23 PM Leon Romanovsky wrote:

quoted

On Thu, Jul 25, 2024 at 11:26:38AM +0200, Ricardo Ribalda Delgado wrote:

quoted

On Wed, Jul 24, 2024 at 10:02 PM Laurent Pinchart wrote:

<...>

quoted

It would be great to define what are the free software communities
here. Distros and final users are also "free software communities" and
they do not care about niche use cases covered by proprietary
software.

Are you certain about that?

As a user, and as an open source Distro developer I have a small hint.
But you could also ask users what they think about not being able to
use their notebook's cameras. The last time that I could not use some
basic hardware from a notebook with Linux was 20 years ago.

Lucky you, I still have consumer hardware (speaker) that doesn't work
with Linux, and even now, there is basic hardware in my current
laptop (HP docking station) that doesn't work reliably in Linux.

quoted

quoted

quoted

They only care (and should care) about normal workflows.

What is a normal workflow?
Does it mean that if user bought something very expensive he
should not be able to use it with free software, because his
usage is different from yours?

Thanks

It means that we should not block the standard usage for 99% of the
population just because 1% of the users cannot do something fancy with
their device.

Right, the problem is that in some areas the statistics slightly different.
99% population is blocked because 1% of the users don't need it and
don't think that it is "normal" flow.

quoted

Let me give you an example. When I buy a camera I want to be able to
do Video Conferencing and take some static photos of documents. I do
not care about: automatic makeup, AI generated background, unicorn
filters, eyes recentering... But we need to give a way to vendors to
implement those things closely, without the marketing differentiators,
vendors have zero incentive to invest in Linux, and that affects all
the population.

I've seen these kind of examples being repeatedly given in discussions
related to camera ISP support in Linux. They are very misleading. These
are not the kind of features that are relevant for the device
pass-through discussion these day. Those are high-level use cases
implemented in userspace, and vendors can ship any closed-source
binaries they want there. What I care about is the features exposed by
the kernel to userspace API.

The ISPs are gradually becoming programmable devices and they indeed
help during all of those examples.

Userspace needs to send/receive information from the ISP, and that is
exactly what vendors want to keep in the close.

Describing how they implement those algorithms is a patent minefield
and their differentiating factor.

quoted

quoted

This challenge seems to be solved for GPUs. I am using my AMD GPU
freely and my nephew can install the amdgpu-pro proprietary user space
driver to play duke nukem (or whatever kids play now) at 2000 fps.

There are other other subsystems that allow vendor passthrough and
their ecosystem has not collapsed.

Yes, I completely agree with you on that.

quoted

Can we have some general guidance of what is acceptable? Can we define
together the "normal workflow" and focus on a *full* open source
implementation of that?

I don't think that is possible to define "normal workflow". Requirement
to have open-source counterpart to everything exposed through UAPI is a
valid one. I'm all for that.

That's my current opinion as well, as least when it comes to the kernel
areas I mostly work with.

--
Regards,

Laurent Pinchart

-- 
Ricardo Ribalda

Hi Laurent

On Thu, Jul 25, 2024 at 9:32 PM Laurent Pinchart
[off-list ref] wrote:

On Wed, Jul 24, 2024 at 04:37:21PM -0400, James Bottomley wrote:

quoted

On Wed, 2024-07-24 at 23:00 +0300, Laurent Pinchart wrote:
[...]

quoted

What I get from the discussions I've followed or partcipated in over
the years is that the main worry of free software communities is
being forced to use closed-source userspace components, whether that
would be to make the device usable at all, or to achieve decent level
of performance or full feature set. We've been through years of
mostly closed-source GPU support, of printer "windrivers", and quite
a few other horrors. The good news is that we've so far overcome lots
(most) of those challenges. Reverse engineering projects paid off,
and so did working hand-in-hand with industry actors in multiple ways
(both openly and behind the scenes). One could then legitimately ask
why we're still scared.

I don't think I am.  We're mostly fully capable of expounding at length
on the business rationale for being open if the thing they're hiding
isn't much of a differentiator anyway (or they're simply hiding it to
try to retain some illusion of control), so we shouldn't have any fear
of being able to make our case in language business people understand.

I also think this fear is partly a mindset problem on our part.  We
came out of the real fight for openness and we do embrace things like a
licence that forces open code (GPL) and symbols that discourage
proprietary drivers (EXPORT_SYMBOL_GPL), so we've somewhat drunk the
FSF coolaid that if we don't stand over manufacturers every second and
force them they'll slide back to their old proprietary ways.  However,
if you look at the entirely permissive ecosystem that grew up after we
did (openstack, docker, kubernetes, etc.) they don't have any such fear
and yet they still have large amounts of uncompelled openness and give
back.

I don't think those are necessarily relevant examples, as far as device
pass-through goes. Vendors have many times reverted to proprietary ways,
and they still do, at least in the areas of the kernel I'm most active
in. I've seen first hand a large SoC vendor very close to opening a
significant part of their camera stack and changing their mind at the
last minute when they heard they could possibly merge their code through
a different subsystem with a pass-through blank cheque.

Without knowing who that large SoC vendor is, and what they will be
willing to open, it is difficult to know what opportunity has been
lost. I would argue that if they have cancelled their open plans based
on an hypothesis, their willingness to open was not that high.

It would be more healthy for the ecosystem, if those discussions were
done more openly, at least all the core maintainers should be
involved.

I'm willing to believe it can be different in other areas, which may
partly explain why different subsystems and different developers have
different biases and have trouble understand each other's point of view.

It is not different in other areas, it is the same area. At the end of
the day it is the same chip manufacturers, with the same legal teams.
It is our attitude that is different.

quoted

quoted

I can't fully answer that question, but there are two points that I
think are relevant. Note that due to my background and experience,
this will be heavily biased towards consumer and embedded hardware,
not data centre-grade devices. Some technologies from the latter
however have a tendency to migrate to the former over time, so the
distinction isn't necessarily as relevant as one may consider.

The first point is that hardware gets more complicated over time, and
in some markets there's also an increase in the number of vendors and
devices. There's a perceived (whether true or not) danger that we
won't be able to keep up with just reverse engineering and a
development model relying on hobyists. Getting vendors involved is
important if we want to scale.

Yes, but there are lots of not very useful complex devices being
produced every day that fail to capture market share.  Not having
reverse engineered drivers for them is no real loss.  If a device does
gain market share, it gains a huge pool of users some of whom become
interested in reverse engineering, so I think market forces actually
work in our favour: we get reverse engineering mostly where the devices
are actually interesting and capture market share.  It's self scaling.

I can't agree with that, sorry. Not only is the difficulty to
reverse-engineer some classes of devices increasing, but saying that
only devices that make it to the top of the market share chart are worth
considering will leave many users on the side of the road.

Today we have left BILLIONS of users at the other side of the road.

quoted

quoted

Second, I think there's a fear of regression. For some categories of
devices, we have made slow but real progress to try and convince the
industry to be more open. This sometimes took a decade of work,
patiently building bridges and creating ecosystems brick by brick.
Some of those ecosystems are sturdy, some not so. Giving pass-through
a blank check will likely have very different effects in different
areas. I don't personally believe it will shatter everything, but I'm
convinced it carries risk in areas where cooperation with vendors is
in its infancy or is fragile for any other reason.

I also think we're on the rise in this space.  Since most cloud
workloads are on Linux, there's huge market pressure on most "found in
the cloud" devices (like accelerators and GPUs) to have an easy to
consume Linux story.  Nvidia is a case in point.  When it only cared
about fast games on some other OS, we get shafted with a proprietary
graphics drivers.  Now it's under pressure to be the number one AI
accelerator provider for the cloud it's suddenly wondering about open
source drivers to make adoption easier.

I can't comment on Nvidia and their inference engines in particular. The
server market may be in a better position that the consumer and embedded
market, and if that's the case, I'm happy for the servers. That doesn't
solve the issues in other markets though.

quoted

quoted

Finally, let's not forget that pass-through APIs are not an all or
nothing option. To cite that example only, DRM requires GPU drivers
to have an open-source userspace implementation to merge the kernel
driver, and the same subsystems strongly pushes for API
standardization for display controllers. We can set different rules
for different cases.

I certainly think we can afford to experiment here, yes.

--
Regards,

Laurent Pinchart

--
Ricardo Ribalda

On Thu, Jul 25, 2024 at 10:02 PM Laurent Pinchart
[off-list ref] wrote:

Hi Ricardo,

On Mon, Jul 22, 2024 at 01:56:11PM +0200, Ricardo Ribalda Delgado wrote:

quoted

On Mon, Jul 22, 2024 at 1:18 PM Laurent Pinchart wrote:

quoted

On Mon, Jul 22, 2024 at 12:42:52PM +0200, Ricardo Ribalda Delgado wrote:

quoted

On Sun, Jul 21, 2024 at 9:25 PM Laurent Pinchart wrote:

quoted

On Tue, Jul 09, 2024 at 03:15:13PM -0700, Dan Williams wrote:

quoted

James Bottomley wrote:

quoted

quoted

The upstream discussion has yielded the full spectrum of positions on
device specific functionality, and it is a topic that needs cross-
kernel consensus as hardware increasingly spans cross-subsystem
concerns. Please consider it for a Maintainers Summit discussion.

I'm with Greg on this ... can you point to some of the contrary
positions?

This thread has that discussion:

http://lore.kernel.org/0-v1-9912f1a11620+2a-fwctl_jgg@nvidia.com

I do not want to speak for others on the saliency of their points, all I
can say is that the contrary positions have so far not moved me to drop
consideration of fwctl for CXL.

Where CXL has a Command Effects Log that is a reasonable protocol for
making decisions about opaque command codes, and that CXL already has a
few years of experience with the commands that *do* need a Linux-command
wrapper.

Some open questions from that thread are: what does it mean for the fate
of a proposal if one subsystem Acks the ABI and another Naks it for a
device that crosses subsystem functionality? Would a cynical hardware
response just lead to plumbing an NVME admin queue, or CXL mailbox to
get device-specific commands past another subsystem's objection?

My default answer would be to trust the maintainers of the relevant
subsystems (or try to convince them when you disagree :-)). Not only
should they know the technical implications best, they should also have
a good view of the whole vertical stack, and the implications of
pass-through for their ecosystem. This may result in a single NAK
overriding ACKs, but we could also try to find technical solutions when
we'll face such issues, to enforce different sets of rules for the
different functions of a device.

Subsystem hopping is something we're recently noticed for camera ISPs,
where a vendor wanted to move from V4L2 to DRM. Technical reasons for
doing so were given, and they were (in my opinion) rather excuses. The
unspoken real (again in my opinion) reason was to avoid documenting the
firmware interface and ship userspace binary blobs with no way for free
software to use all the device's features. That's something we have been
fighting against for years, trying to convince vendors that they can
provide better and more open camera support without the world
collapsing, with increasing success recently. Saying amen to
pass-through in this case would be a huge step back that would hurt
users and the whole ecosystem in the short and long term.

In my view, DRM is a more suitable model for complex ISPs than V4L2:

I know we disagree on this topic :-) I'm sure we'll continue the
conversation, but I think the technical discussion likely belongs to a
different mail thread.

quoted

- Userspace Complexity: ISPs demand a highly complex and evolving API,
similar to Vulkan or OpenGL. Applications typically need a framework
like libcamera to utilize ISPs effectively, much like Mesa for
graphics cards.

- Lack of Standardization: There's no universal standard for ISPs;
each vendor implements unique features and usage patterns. DRM
addresses this through vendor-specific IOCTLs

- Proprietary Architectures: Vendors often don't fully disclose their
hardware architectures. DRM cleverly only necessitates a Mesa
implementation, not comprehensive documentation.

This point isn't technical and is more on-topic for this mail thread.

V4L2 doesn't require hundreds of pages of comprehensive documentation in
text form. An open-source userspace implementation that covers the
feature set exposed by the driver is acceptable in place of
documentation (provided, of course, that the userspace code wouldn't be
deliberately obfuscated). This is similar in spirit to the rule for GPU
DRM drivers.

In DRM vendors typically define a custom IOCTL per driver to pass
command buffers.
Only the command buffer structure, and a mesa implementation using
that command buffer to support the standard features is required.

In V4l2 custom IOCTLs are discouraged. Random command buffers cannot
be passed from userspace, they are typically formed in the driver from
a strictly checked struct.

V4L2 has a mechanism to pass buffers between userspace and kernelspace,
and that mechanism is used in mainline drivers to pass camera ISP
parameters. They're not called "command buffers" but that's just a
difference in terminology. The technical means to pass command buffers
to the driver is thus there, I see no meaningful difference with DRM.
Where things can differ is in the contents of those buffers, and the
requirements for documentation or open userspace implementations, but
that's not a technical question.

There are two things here:

- The political/strategic/philosophical/religious aspect: The industry
definitely prefers the strategic requirements imposed by DRM. In fact
some vendors had some huge legal troubles when they had tried to
follow v4l2 requirements.
- The technical aspect: DRM is more mature when it comes to
sending/receiving buffers to the hardware, and an ISP looks *much*
more similar to an accel device or a GPU than a UVC camera.

quoted

quoted

quoted

Our current approach of pushing back against vendors, instead of
seeking compromise, has resulted in the vast majority of the market
(99% if not more) relying on out-of-tree drivers. This leaves users
with no options for utilizing their cameras outside of Android.

DRM allows a hybrid model, where:
- Open Source Foundation: Standard use cases are covered by a fully
open-source stack.
- Vendor Differentiation: Vendors retain the freedom to implement
proprietary features (e.g., automatic makeup) as closed source.

V4L2 does as well, you can implement all kind of closed-source ISP
control algorithms in userspace, as long as there's an open-source
implementation that exercises the same hardware features. A good analogy

Is it really mandatory to have an open-source 3A algorithm? I thought
defining the input and output from the algorithm was good enough.

What really matters is documenting the ISP parameters with enough
details to allow for the implementation of open-source userspace code.
Once you have that, 3A is quite simple. You can refine it (especially
AWB) to great length, for instance using NPUs to compute parameters, and
there's absolutely no issue with such userspace implementations being
closed.

In practice, some vendors prefer documenting the parameters by writing
an open-source userspace implementation, partly maybe because developers
are more familiar writing code than formal documentation. I would be
fine either way, as long as there's enough information to make use of
the ISP.

Even with vendor passthrough there is still a need to provide a full
open source implementation (probably based on libcamera).

So you will have enough information to use all the common
functionality of a camera.

quoted

AFAIK for some time there was no ipu3 open source algorithm, and the
driver has been upstream.

It sneaked in before we realized we had to enforce rules :-) That's
actually a good example, when we wrote open-source userspace support, we
realized that the level of documentation included in the IPU3 kernel
header was nowhere close to what was needed to make use of the device.

quoted

quoted

for people less familiar with ISPs is shader compilers, GPU vendors are
free to ship closed-source implementations that include more
optimizations, as long as the open-source, less optimized implementation
covers the same GPU ISA, so that open-source developers can also work on
optimizing it.

I believe a more accurate description is that in v4l2 is that we
expect that all the registers, device architecture and behaviour to be
documented and accessed with standard IOCTLs. Anything not documented
cannot be accessed by userspace.

In DRM their concern is that there is a fully open source
implementation that the user can use. Vendors have custom IOCTLs and
they can offer proprietary software for some use cases.

Custom ioctls are not closed secrets in DRM, so comparing custom ioctls
vs. standard ioctls isn't very relevant to this discussion. I really
don't see how this would be about ioctls, it's about making the featured
exposed by the drivers, through whatever means a particular subsystem
allows, usable by open userspace.

quoted

quoted

Thinking that DRM would offer a free pass-through path compared to V4L2
doesn't seem realistic to me. Both subsystems will have similar rules.

DRM does indeed allow vendors to pass random command buffers and they
will be sent to the hardware. We cannot do that in v4l2.

You can pass a command buffer to a V4L2 device and have the driver send
it to the device firmware (ISPs using real command buffers usually run a
firmware). If you want the driver to be merged upstream, you have to
document the command buffer in enough details.

Documenting with "enough details" is not enough. In V4L2, we have to
deeply inspect every single buffer to make sure that it is not sending
an unknown combination of command+arguments, or in other situations we
construct the command buffer in the driver.

quoted

I might be wrong, but GPU drivers do not deeply inspect the command
buffers to make sure that they do not use any feature not covered by
mesa.

That's correct, but I don't think that's relevant. The GPU market has
GLSL and Vulkan. An open-source compliant implementation will end up
exercising a very very large part of the device ISA, command submission
mechanism and synchronization primitives, if not all of it. There's
little a vendor would keep under the hood and use in closed-source
userspace only. For cameras, there's no standard userspace API that
covers by design a very large part of what is the ISP equivalent of a

Libcamera supports Camera HAL3, gstreamer, v4l2, pipewire...

If we are afraid of vendors providing "toy" implemententations to pass
the openness requirements, we can add more features/tests to
libcamera.

And at the end of the day, there will be humans deciding if what a
vendor has provided is good enough or not.

GPU ISA. Even "command buffers" are not a proper description of the
parameters the vast majority of ISPs consume.

Modern ISPs are definitely going in the direction of "command buffers"

quoted

quoted

quoted

This approach would allow billions of users to access their hardware
more securely and with in-tree driver support. Our current stubborn
pursuit of an idealistic goal has already negatively impacted both
users and the ecosystem.

The late wins, in my opinion, cannot scale to the consumer market, and
Linux will remain a niche market for ISPs.

If such a hybrid model goes against Linux goals, this is something
that should be agreed upon by the whole community, so we have the same
criteria for all subsystems.

--
Regards,

Laurent Pinchart

--
Ricardo Ribalda

Hi Laurent

On Thu, Jul 25, 2024 at 10:07 PM Laurent Pinchart
[off-list ref] wrote:

Hi Jason,

On Thu, Jul 25, 2024 at 04:43:14PM -0300, Jason Gunthorpe wrote:

quoted

On Thu, Jul 25, 2024 at 10:31:25PM +0300, Laurent Pinchart wrote:

quoted

I don't think those are necessarily relevant examples, as far as device
pass-through goes. Vendors have many times reverted to proprietary ways,
and they still do, at least in the areas of the kernel I'm most active
in. I've seen first hand a large SoC vendor very close to opening a
significant part of their camera stack and changing their mind at the
last minute when they heard they could possibly merge their code through
a different subsystem with a pass-through blank cheque.

If someone came with a fully open source framework for (say) some
camera,

We have such a framework, it's called libcamera :-) Multiple vendors are
already collaborating.

quoted

with a passthrough kernel driver design, would you reject it
soley because it is passthrough based and you are scared that
something else will use it to do something not open source?

It depends what "passthrough kernel driver design" means. If it means
accessing the PCI registers directly from userspace, yes. That's what X
used to do before KMS, and I'm glad it's now a distant past.

Nobody has suggested giving PCI register access to userspace.

If it means a kernel driver that takes the majority of its runtime
parameters from a buffer blob assembled by userspace, while controlling
clocks, power domains and performing basic validation in kernelspace,
then I've already acked multiple drivers with such a design, exactly
because they have open-source userspace that doesn't try to keep many
device features proprietary and usable by closed-source userspace only.

If that was an option we would not be having this discussion.

Vendors cannot have vendor access in v4l2.
"""
It is not an option to upstream a driver that has support for
undocumented closed features. Basically maintainers can't put their
name on something that contains unverifiable (for them) and unusable
(by all except the vendor) features.
"""
https://linuxtv.org/news.php?entry=2022-11-14-1.hverkuil

quoted

I wouldn't agree with that position, I think denying users useful open
source solutions out of fear is not what Linux should be doing.

--
Regards,

Laurent Pinchart

--
Ricardo Ribalda

On Fri, Jul 26, 2024 at 10:04:33AM +0200, Ricardo Ribalda Delgado wrote:

On Thu, Jul 25, 2024 at 10:02 PM Laurent Pinchart wrote:

quoted

On Mon, Jul 22, 2024 at 01:56:11PM +0200, Ricardo Ribalda Delgado wrote:

quoted

On Mon, Jul 22, 2024 at 1:18 PM Laurent Pinchart wrote:

quoted

On Mon, Jul 22, 2024 at 12:42:52PM +0200, Ricardo Ribalda Delgado wrote:

quoted

On Sun, Jul 21, 2024 at 9:25 PM Laurent Pinchart wrote:

quoted

On Tue, Jul 09, 2024 at 03:15:13PM -0700, Dan Williams wrote:

quoted

James Bottomley wrote:

quoted

quoted

The upstream discussion has yielded the full spectrum of positions on
device specific functionality, and it is a topic that needs cross-
kernel consensus as hardware increasingly spans cross-subsystem
concerns. Please consider it for a Maintainers Summit discussion.

I'm with Greg on this ... can you point to some of the contrary
positions?

This thread has that discussion:

http://lore.kernel.org/0-v1-9912f1a11620+2a-fwctl_jgg@nvidia.com

I do not want to speak for others on the saliency of their points, all I
can say is that the contrary positions have so far not moved me to drop
consideration of fwctl for CXL.

Where CXL has a Command Effects Log that is a reasonable protocol for
making decisions about opaque command codes, and that CXL already has a
few years of experience with the commands that *do* need a Linux-command
wrapper.

Some open questions from that thread are: what does it mean for the fate
of a proposal if one subsystem Acks the ABI and another Naks it for a
device that crosses subsystem functionality? Would a cynical hardware
response just lead to plumbing an NVME admin queue, or CXL mailbox to
get device-specific commands past another subsystem's objection?

My default answer would be to trust the maintainers of the relevant
subsystems (or try to convince them when you disagree :-)). Not only
should they know the technical implications best, they should also have
a good view of the whole vertical stack, and the implications of
pass-through for their ecosystem. This may result in a single NAK
overriding ACKs, but we could also try to find technical solutions when
we'll face such issues, to enforce different sets of rules for the
different functions of a device.

Subsystem hopping is something we're recently noticed for camera ISPs,
where a vendor wanted to move from V4L2 to DRM. Technical reasons for
doing so were given, and they were (in my opinion) rather excuses. The
unspoken real (again in my opinion) reason was to avoid documenting the
firmware interface and ship userspace binary blobs with no way for free
software to use all the device's features. That's something we have been
fighting against for years, trying to convince vendors that they can
provide better and more open camera support without the world
collapsing, with increasing success recently. Saying amen to
pass-through in this case would be a huge step back that would hurt
users and the whole ecosystem in the short and long term.

In my view, DRM is a more suitable model for complex ISPs than V4L2:

I know we disagree on this topic :-) I'm sure we'll continue the
conversation, but I think the technical discussion likely belongs to a
different mail thread.

quoted

- Userspace Complexity: ISPs demand a highly complex and evolving API,
similar to Vulkan or OpenGL. Applications typically need a framework
like libcamera to utilize ISPs effectively, much like Mesa for
graphics cards.

- Lack of Standardization: There's no universal standard for ISPs;
each vendor implements unique features and usage patterns. DRM
addresses this through vendor-specific IOCTLs

- Proprietary Architectures: Vendors often don't fully disclose their
hardware architectures. DRM cleverly only necessitates a Mesa
implementation, not comprehensive documentation.

This point isn't technical and is more on-topic for this mail thread.

V4L2 doesn't require hundreds of pages of comprehensive documentation in
text form. An open-source userspace implementation that covers the
feature set exposed by the driver is acceptable in place of
documentation (provided, of course, that the userspace code wouldn't be
deliberately obfuscated). This is similar in spirit to the rule for GPU
DRM drivers.

In DRM vendors typically define a custom IOCTL per driver to pass
command buffers.
Only the command buffer structure, and a mesa implementation using
that command buffer to support the standard features is required.

In V4l2 custom IOCTLs are discouraged. Random command buffers cannot
be passed from userspace, they are typically formed in the driver from
a strictly checked struct.

V4L2 has a mechanism to pass buffers between userspace and kernelspace,
and that mechanism is used in mainline drivers to pass camera ISP
parameters. They're not called "command buffers" but that's just a
difference in terminology. The technical means to pass command buffers
to the driver is thus there, I see no meaningful difference with DRM.
Where things can differ is in the contents of those buffers, and the
requirements for documentation or open userspace implementations, but
that's not a technical question.

There are two things here:

- The political/strategic/philosophical/religious aspect: The industry
definitely prefers the strategic requirements imposed by DRM. In fact
some vendors had some huge legal troubles when they had tried to
follow v4l2 requirements.

That's I'm willing to debate.

- The technical aspect: DRM is more mature when it comes to
sending/receiving buffers to the hardware, and an ISP looks *much*
more similar to an accel device or a GPU than a UVC camera.

But this I don't agree with. I think we should forgo the technical
discussion and stop pretending that DRM is better for this use case from
a technical point of view, and focus on the other aspect of the
discussion. (We can of course reopen the technical discussion if new
concrete arguments emerge.)

quoted

quoted

quoted

quoted

Our current approach of pushing back against vendors, instead of
seeking compromise, has resulted in the vast majority of the market
(99% if not more) relying on out-of-tree drivers. This leaves users
with no options for utilizing their cameras outside of Android.

DRM allows a hybrid model, where:
- Open Source Foundation: Standard use cases are covered by a fully
open-source stack.
- Vendor Differentiation: Vendors retain the freedom to implement
proprietary features (e.g., automatic makeup) as closed source.

V4L2 does as well, you can implement all kind of closed-source ISP
control algorithms in userspace, as long as there's an open-source
implementation that exercises the same hardware features. A good analogy

Is it really mandatory to have an open-source 3A algorithm? I thought
defining the input and output from the algorithm was good enough.

What really matters is documenting the ISP parameters with enough
details to allow for the implementation of open-source userspace code.
Once you have that, 3A is quite simple. You can refine it (especially
AWB) to great length, for instance using NPUs to compute parameters, and
there's absolutely no issue with such userspace implementations being
closed.

In practice, some vendors prefer documenting the parameters by writing
an open-source userspace implementation, partly maybe because developers
are more familiar writing code than formal documentation. I would be
fine either way, as long as there's enough information to make use of
the ISP.

Even with vendor passthrough there is still a need to provide a full
open source implementation (probably based on libcamera).

We have a different interpretation of "full" :-) I want to aim for
"full" to cover all the features exposed by the driver UAPI. There could
be some exceptions that we can discuss if there are compeling arguments,
but that would result in a 99% coverage, not a 20%, 50% or 80% coverage.

So you will have enough information to use all the common
functionality of a camera.

quoted

quoted

AFAIK for some time there was no ipu3 open source algorithm, and the
driver has been upstream.

It sneaked in before we realized we had to enforce rules :-) That's
actually a good example, when we wrote open-source userspace support, we
realized that the level of documentation included in the IPU3 kernel
header was nowhere close to what was needed to make use of the device.

quoted

quoted

for people less familiar with ISPs is shader compilers, GPU vendors are
free to ship closed-source implementations that include more
optimizations, as long as the open-source, less optimized implementation
covers the same GPU ISA, so that open-source developers can also work on
optimizing it.

I believe a more accurate description is that in v4l2 is that we
expect that all the registers, device architecture and behaviour to be
documented and accessed with standard IOCTLs. Anything not documented
cannot be accessed by userspace.

In DRM their concern is that there is a fully open source
implementation that the user can use. Vendors have custom IOCTLs and
they can offer proprietary software for some use cases.

Custom ioctls are not closed secrets in DRM, so comparing custom ioctls
vs. standard ioctls isn't very relevant to this discussion. I really
don't see how this would be about ioctls, it's about making the featured
exposed by the drivers, through whatever means a particular subsystem
allows, usable by open userspace.

quoted

quoted

Thinking that DRM would offer a free pass-through path compared to V4L2
doesn't seem realistic to me. Both subsystems will have similar rules.

DRM does indeed allow vendors to pass random command buffers and they
will be sent to the hardware. We cannot do that in v4l2.

You can pass a command buffer to a V4L2 device and have the driver send
it to the device firmware (ISPs using real command buffers usually run a
firmware). If you want the driver to be merged upstream, you have to
document the command buffer in enough details.

Documenting with "enough details" is not enough. In V4L2, we have to
deeply inspect every single buffer to make sure that it is not sending
an unknown combination of command+arguments, or in other situations we
construct the command buffer in the driver.

quoted

quoted

I might be wrong, but GPU drivers do not deeply inspect the command
buffers to make sure that they do not use any feature not covered by
mesa.

That's correct, but I don't think that's relevant. The GPU market has
GLSL and Vulkan. An open-source compliant implementation will end up
exercising a very very large part of the device ISA, command submission
mechanism and synchronization primitives, if not all of it. There's
little a vendor would keep under the hood and use in closed-source
userspace only. For cameras, there's no standard userspace API that
covers by design a very large part of what is the ISP equivalent of a

Libcamera supports Camera HAL3, gstreamer, v4l2, pipewire...

Those are not comparable to GLSL or Vulkan, they are much higher level.

If we are afraid of vendors providing "toy" implemententations to pass
the openness requirements, we can add more features/tests to
libcamera.

And at the end of the day, there will be humans deciding if what a
vendor has provided is good enough or not.

quoted

GPU ISA. Even "command buffers" are not a proper description of the
parameters the vast majority of ISPs consume.

Modern ISPs are definitely going in the direction of "command buffers"

Examples please, with technical details. I've seen this argument being
used for at least a year, with very little evidence. There are ISPs that
are more programmable than others, but in practice firmwares mostly
expose fixed functions, not ISA-level programmability.

quoted

quoted

quoted

quoted

This approach would allow billions of users to access their hardware
more securely and with in-tree driver support. Our current stubborn
pursuit of an idealistic goal has already negatively impacted both
users and the ecosystem.

The late wins, in my opinion, cannot scale to the consumer market, and
Linux will remain a niche market for ISPs.

If such a hybrid model goes against Linux goals, this is something
that should be agreed upon by the whole community, so we have the same
criteria for all subsystems.

-- 
Regards,

Laurent Pinchart

On Fri, Jul 26, 2024 at 10:04:48AM +0200, Ricardo Ribalda Delgado wrote:

On Thu, Jul 25, 2024 at 10:07 PM Laurent Pinchart wrote:

quoted

On Thu, Jul 25, 2024 at 04:43:14PM -0300, Jason Gunthorpe wrote:

quoted

On Thu, Jul 25, 2024 at 10:31:25PM +0300, Laurent Pinchart wrote:

quoted

I don't think those are necessarily relevant examples, as far as device
pass-through goes. Vendors have many times reverted to proprietary ways,
and they still do, at least in the areas of the kernel I'm most active
in. I've seen first hand a large SoC vendor very close to opening a
significant part of their camera stack and changing their mind at the
last minute when they heard they could possibly merge their code through
a different subsystem with a pass-through blank cheque.

If someone came with a fully open source framework for (say) some
camera,

We have such a framework, it's called libcamera :-) Multiple vendors are
already collaborating.

quoted

with a passthrough kernel driver design, would you reject it
soley because it is passthrough based and you are scared that
something else will use it to do something not open source?

It depends what "passthrough kernel driver design" means. If it means
accessing the PCI registers directly from userspace, yes. That's what X
used to do before KMS, and I'm glad it's now a distant past.

Nobody has suggested giving PCI register access to userspace.

I know you didn't, but as I didn't expect Jason to be familiar with the
camera ISP discussions, I didn't want to make any specific assumption
regarding what he meant by passthrough kernel driver design.

quoted

If it means a kernel driver that takes the majority of its runtime
parameters from a buffer blob assembled by userspace, while controlling
clocks, power domains and performing basic validation in kernelspace,
then I've already acked multiple drivers with such a design, exactly
because they have open-source userspace that doesn't try to keep many
device features proprietary and usable by closed-source userspace only.

If that was an option we would not be having this discussion.

Vendors cannot have vendor access in v4l2.
"""
It is not an option to upstream a driver that has support for
undocumented closed features. Basically maintainers can't put their
name on something that contains unverifiable (for them) and unusable
(by all except the vendor) features.
"""
https://linuxtv.org/news.php?entry=2022-11-14-1.hverkuil

What is not an option exactly in my description above ? We have multiple
V4L2 drivers for ISPs. They receive ISP parameters from userspace
through a data buffer. It's not allowed to be opaque, but it doesn't
prevent vendor closed-source userspace implementations with additional
*camera* features, as long as the *hardware* features are available to
everybody.

quoted

quoted

I wouldn't agree with that position, I think denying users useful open
source solutions out of fear is not what Linux should be doing.

-- 
Regards,

Laurent Pinchart

On Fri, Jul 26, 2024 at 03:49:49PM +0300, Laurent Pinchart wrote:

What is not an option exactly in my description above ? We have multiple
V4L2 drivers for ISPs. They receive ISP parameters from userspace
through a data buffer. It's not allowed to be opaque, but it doesn't
prevent vendor closed-source userspace implementations with additional
*camera* features, as long as the *hardware* features are available to
everybody.

How far do you take opaque?

In mlx5 we pass command buffers from user to kernel to HW and the
kernel does only a little checking.

There is a 12kloc file describing the layout of alot of commands:
include/linux/mlx5/mlx5_ifc.h

There is an open PDF describing in detail some subset of this:
https://network.nvidia.com/files/doc-2020/ethernet-adapters-programming-manual.pdf

There are in-kernel implementations driving most of those commands.

Other commands are only issued by userspace, and we have open source
DPDK, rdma-core and UCX implementations driving them.

ie, this is really quite good as far as a device providing open source
solutions goes.

However, no doubt there is more FW capability and commands than even
this vast amount documents - so lets guess that propritary code is
using this interface with unknown commands too.

From a camera perspective would you be OK with that? Let's guess that
90% of use cases are covered with fully open code. Do we also need to
forcefully close the door to an imagined 10% of proprietary cases just
to be sure open always wins?

Does closing the door have to come at the cost of a technically clean
solution? Doing validation in the kernel to enforce an ideological
position would severely degrade mlx5, and would probably never really
be robust.

Jason

On Fri, Jul 26, 2024 at 10:02:27AM +0200, Ricardo Ribalda Delgado wrote:

On Thu, Jul 25, 2024 at 9:44 PM Laurent Pinchart wrote:

quoted

On Thu, Jul 25, 2024 at 04:20:35PM +0300, Leon Romanovsky wrote:

quoted

On Thu, Jul 25, 2024 at 03:02:13PM +0200, Ricardo Ribalda Delgado wrote:

quoted

On Thu, Jul 25, 2024 at 2:23 PM Leon Romanovsky wrote:

quoted

On Thu, Jul 25, 2024 at 11:26:38AM +0200, Ricardo Ribalda Delgado wrote:

quoted

On Wed, Jul 24, 2024 at 10:02 PM Laurent Pinchart wrote:

<...>

quoted

It would be great to define what are the free software communities
here. Distros and final users are also "free software communities" and
they do not care about niche use cases covered by proprietary
software.

Are you certain about that?

As a user, and as an open source Distro developer I have a small hint.
But you could also ask users what they think about not being able to
use their notebook's cameras. The last time that I could not use some
basic hardware from a notebook with Linux was 20 years ago.

Lucky you, I still have consumer hardware (speaker) that doesn't work
with Linux, and even now, there is basic hardware in my current
laptop (HP docking station) that doesn't work reliably in Linux.

quoted

quoted

quoted

They only care (and should care) about normal workflows.

What is a normal workflow?
Does it mean that if user bought something very expensive he
should not be able to use it with free software, because his
usage is different from yours?

Thanks

It means that we should not block the standard usage for 99% of the
population just because 1% of the users cannot do something fancy with
their device.

Right, the problem is that in some areas the statistics slightly different.
99% population is blocked because 1% of the users don't need it and
don't think that it is "normal" flow.

quoted

Let me give you an example. When I buy a camera I want to be able to
do Video Conferencing and take some static photos of documents. I do
not care about: automatic makeup, AI generated background, unicorn
filters, eyes recentering... But we need to give a way to vendors to
implement those things closely, without the marketing differentiators,
vendors have zero incentive to invest in Linux, and that affects all
the population.

I've seen these kind of examples being repeatedly given in discussions
related to camera ISP support in Linux. They are very misleading. These
are not the kind of features that are relevant for the device
pass-through discussion these day. Those are high-level use cases
implemented in userspace, and vendors can ship any closed-source
binaries they want there. What I care about is the features exposed by
the kernel to userspace API.

The ISPs are gradually becoming programmable devices and they indeed
help during all of those examples.

I'd like to see more technical information to substantiate this claim.
So far what I've sometimes seen is ISPs that include programmable
elements, but hiding those behind a firmware that exposes a fixed
(configurable) pipeline. I've also heard of attempts to expose some of
that programmability to the operating system, which were abandoned in
the end due to lack usefulness.

Userspace needs to send/receive information from the ISP, and that is
exactly what vendors want to keep in the close.

But that's exactly what we need to implement an open userspace ecosystem
:-)

Describing how they implement those algorithms is a patent minefield
and their differentiating factor.

Those are also arguments I've heard many times before. The
differentiating factor for cameras today is mostly in userspace ISP
control algorithms, and nobody is telling vendors they need to open all
that.

When it comes to patents, we all know how software patents is a
minefield, and hardware is also affected. I can't have much sympathy for
this argument though, those patents mostly benefit the largest players
in the market, and those are the ones who currently claim they can't
open anything due to patents.

quoted

quoted

quoted

This challenge seems to be solved for GPUs. I am using my AMD GPU
freely and my nephew can install the amdgpu-pro proprietary user space
driver to play duke nukem (or whatever kids play now) at 2000 fps.

There are other other subsystems that allow vendor passthrough and
their ecosystem has not collapsed.

Yes, I completely agree with you on that.

quoted

Can we have some general guidance of what is acceptable? Can we define
together the "normal workflow" and focus on a *full* open source
implementation of that?

I don't think that is possible to define "normal workflow". Requirement
to have open-source counterpart to everything exposed through UAPI is a
valid one. I'm all for that.

That's my current opinion as well, as least when it comes to the kernel
areas I mostly work with.

-- 
Regards,

Laurent Pinchart

On Fri, Jul 26, 2024 at 10:03:36AM +0200, Ricardo Ribalda Delgado wrote:

On Thu, Jul 25, 2024 at 9:32 PM Laurent Pinchart wrote:

quoted

On Wed, Jul 24, 2024 at 04:37:21PM -0400, James Bottomley wrote:

quoted

On Wed, 2024-07-24 at 23:00 +0300, Laurent Pinchart wrote:
[...]

quoted

What I get from the discussions I've followed or partcipated in over
the years is that the main worry of free software communities is
being forced to use closed-source userspace components, whether that
would be to make the device usable at all, or to achieve decent level
of performance or full feature set. We've been through years of
mostly closed-source GPU support, of printer "windrivers", and quite
a few other horrors. The good news is that we've so far overcome lots
(most) of those challenges. Reverse engineering projects paid off,
and so did working hand-in-hand with industry actors in multiple ways
(both openly and behind the scenes). One could then legitimately ask
why we're still scared.

I don't think I am.  We're mostly fully capable of expounding at length
on the business rationale for being open if the thing they're hiding
isn't much of a differentiator anyway (or they're simply hiding it to
try to retain some illusion of control), so we shouldn't have any fear
of being able to make our case in language business people understand.

I also think this fear is partly a mindset problem on our part.  We
came out of the real fight for openness and we do embrace things like a
licence that forces open code (GPL) and symbols that discourage
proprietary drivers (EXPORT_SYMBOL_GPL), so we've somewhat drunk the
FSF coolaid that if we don't stand over manufacturers every second and
force them they'll slide back to their old proprietary ways.  However,
if you look at the entirely permissive ecosystem that grew up after we
did (openstack, docker, kubernetes, etc.) they don't have any such fear
and yet they still have large amounts of uncompelled openness and give
back.

I don't think those are necessarily relevant examples, as far as device
pass-through goes. Vendors have many times reverted to proprietary ways,
and they still do, at least in the areas of the kernel I'm most active
in. I've seen first hand a large SoC vendor very close to opening a
significant part of their camera stack and changing their mind at the
last minute when they heard they could possibly merge their code through
a different subsystem with a pass-through blank cheque.

Without knowing who that large SoC vendor is, and what they will be
willing to open, it is difficult to know what opportunity has been
lost.

I would also like to discuss this in more details, but I can't do so
publicly :-( That's why I don't want to focus too much on this argument.

I would argue that if they have cancelled their open plans based
on an hypothesis, their willingness to open was not that high.

That's exactly what I previously described in this mail thread as areas
where cooperation with vendors is in its infancy. Convincing large
industry players is long term work, and it's very demotivating when you
think there's finally light at the end of the tunel, and plans gets
cancelled at the last minute. We're not giving up though.

It would be more healthy for the ecosystem, if those discussions were
done more openly, at least all the core maintainers should be
involved.

Can you get all the camera vendors you work with to talk about all this
openly ? :-) We all know it's a difficult and long process.

quoted

I'm willing to believe it can be different in other areas, which may
partly explain why different subsystems and different developers have
different biases and have trouble understand each other's point of view.

It is not different in other areas, it is the same area. At the end of
the day it is the same chip manufacturers, with the same legal teams.
It is our attitude that is different.

I don't agree with that. Not only are there different manufacturers,
there are also different teams, with different approaches. The world is
not uniform.

quoted

quoted

quoted

I can't fully answer that question, but there are two points that I
think are relevant. Note that due to my background and experience,
this will be heavily biased towards consumer and embedded hardware,
not data centre-grade devices. Some technologies from the latter
however have a tendency to migrate to the former over time, so the
distinction isn't necessarily as relevant as one may consider.

The first point is that hardware gets more complicated over time, and
in some markets there's also an increase in the number of vendors and
devices. There's a perceived (whether true or not) danger that we
won't be able to keep up with just reverse engineering and a
development model relying on hobyists. Getting vendors involved is
important if we want to scale.

Yes, but there are lots of not very useful complex devices being
produced every day that fail to capture market share.  Not having
reverse engineered drivers for them is no real loss.  If a device does
gain market share, it gains a huge pool of users some of whom become
interested in reverse engineering, so I think market forces actually
work in our favour: we get reverse engineering mostly where the devices
are actually interesting and capture market share.  It's self scaling.

I can't agree with that, sorry. Not only is the difficulty to
reverse-engineer some classes of devices increasing, but saying that
only devices that make it to the top of the market share chart are worth
considering will leave many users on the side of the road.

Today we have left BILLIONS of users at the other side of the road.

Because we're the only ones to be blamed, or more accuretely, the blame
is solely on me. Vendors are all saints who want the best for the world.
This argument doesn't hold. Please stop with this, I don't want to get
to name calling.

quoted

quoted

quoted

Second, I think there's a fear of regression. For some categories of
devices, we have made slow but real progress to try and convince the
industry to be more open. This sometimes took a decade of work,
patiently building bridges and creating ecosystems brick by brick.
Some of those ecosystems are sturdy, some not so. Giving pass-through
a blank check will likely have very different effects in different
areas. I don't personally believe it will shatter everything, but I'm
convinced it carries risk in areas where cooperation with vendors is
in its infancy or is fragile for any other reason.

I also think we're on the rise in this space.  Since most cloud
workloads are on Linux, there's huge market pressure on most "found in
the cloud" devices (like accelerators and GPUs) to have an easy to
consume Linux story.  Nvidia is a case in point.  When it only cared
about fast games on some other OS, we get shafted with a proprietary
graphics drivers.  Now it's under pressure to be the number one AI
accelerator provider for the cloud it's suddenly wondering about open
source drivers to make adoption easier.

I can't comment on Nvidia and their inference engines in particular. The
server market may be in a better position that the consumer and embedded
market, and if that's the case, I'm happy for the servers. That doesn't
solve the issues in other markets though.

quoted

quoted

Finally, let's not forget that pass-through APIs are not an all or
nothing option. To cite that example only, DRM requires GPU drivers
to have an open-source userspace implementation to merge the kernel
driver, and the same subsystems strongly pushes for API
standardization for display controllers. We can set different rules
for different cases.

I certainly think we can afford to experiment here, yes.

-- 
Regards,

Laurent Pinchart

On Thu, 2024-07-25 at 20:37 +0300, Leon Romanovsky wrote:

I don't want to hijack this thread, but I tried now my two Jabra
speakers (one is used and another is brand new which I got from IT)
and both don't work on my FC40 laptop.

Bring it to Plumbers and we'll try it on a variety of systems ... just
in case it's a fluke that the Jabras we use work (would be embarrassing
for us to buy the wrong ones next time).

Regards,

James

On Fri, Jul 26, 2024 at 10:11:06AM -0300, Jason Gunthorpe wrote:

On Fri, Jul 26, 2024 at 03:49:49PM +0300, Laurent Pinchart wrote:

quoted

What is not an option exactly in my description above ? We have multiple
V4L2 drivers for ISPs. They receive ISP parameters from userspace
through a data buffer. It's not allowed to be opaque, but it doesn't
prevent vendor closed-source userspace implementations with additional
*camera* features, as long as the *hardware* features are available to
everybody.

How far do you take opaque?

In mlx5 we pass command buffers from user to kernel to HW and the
kernel does only a little checking.

I won't comment on the mlx5 case in particular, I don't have enough
knowledge to do so.

When it comes to validation of commands by the kernel, at the very least
the kernel driver needs to be able to guarantee safety. For instance,
that means that any command that can result in DMA operations need to be
validated (e.g. verifying buffer addresses and sizes) and/or executed
partly by the driver (e.g. mapping a buffer through an IOMMU), depending
on hardware constraints.

There is a 12kloc file describing the layout of alot of commands:
include/linux/mlx5/mlx5_ifc.h

There is an open PDF describing in detail some subset of this:
https://network.nvidia.com/files/doc-2020/ethernet-adapters-programming-manual.pdf

There are in-kernel implementations driving most of those commands.

For camera ISPs, most of the "commands" wouldn't be driven by the
kernel. I don't have expectations when it comes to what commands the
kernel exercises directly, I think that highly depends on the device
type.

Other commands are only issued by userspace, and we have open source
DPDK, rdma-core and UCX implementations driving them.

ie, this is really quite good as far as a device providing open source
solutions goes.

However, no doubt there is more FW capability and commands than even
this vast amount documents - so lets guess that propritary code is
using this interface with unknown commands too.

From a camera perspective would you be OK with that? Let's guess that
90% of use cases are covered with fully open code. Do we also need to
forcefully close the door to an imagined 10% of proprietary cases just
to be sure open always wins?

For command buffers interpreted by a firmware, it would be extremely
difficult, if not impossible, to ensure that nothing undocumented is
possible to access from userspace. I think we have two issues here, one
of them is to define what we required to be openly accessible, and the
other to avoid vendors cheating by claiming to be more open than they
actually are.

I believe the latter can't be solved technically. At the end of the day
it's a matter of trust, and the only option I can see is to build that
trust with the vendors, and to make it clear that breaches of trust will
have consequences. A vendor that deliberately lies would end up on my
personal backlist for as long as they don't implement structural
solutions to be a better player in the ecosystem. What is acceptable as
consequences is for the communities to decide. We have previously banned
people or organizations from contributing to the kernel for certain
periods of time (the University of Minnesota case comes to my mind for
instance), and other options can be considered too.

As for the first issue, I think it's a difficult one as it is very
difficult to quantify the features covered by open implementations, as
well as their importance. You could have a 99% open command set where
the 1% would impact open implementations extremely negatively, the same
way you could have a 50% open command set where the other 50% isn't of
any use to anyone but the vendor (for instance used to debug the
firmware implementation).

For cameras, the feature set can I believe be expressed in terms of ISP
processing blocks that are usable by open implementations, as well as in
terms of the type of usage of those features. For instance, is it
acceptable that a vendor documents how to use 2D noise reduction but
makes 3D (temporal) noise reduction available only to close-source
userspace ? My personal answer is no. I want to aim for very close to
100% of the features, and certainly 100% of the "large" features. I can
close my eyes on features that are very very niche, but what is niche
today may not be tomorrow. For instance, if a camera ISP implements a
feature used only for very specific camera sensors targetted at
autonomous driving in a new generation of research prototypes that only
one company has access to, I'll be inclined to ignore that. That
technology may however become much more mainstream 5 years later.

The other aspect is the type of usage of the features. For camera ISPs
again, some parameters will be computed through a tuning process, and
will be fixed for the lifetime of the camera. I want those parameters to
be documented, to enable users to tune cameras for their own use cases.
This is less important in some market segments, such as laptops for
instance, but is crucial for embedded devices. This is an area where
I've previously been called unreasonable, and I don't think that's fair.
The tuned-and-immutable parameters are not plentiful, as most of the
tuning results in data that needs to be combined with runtime
information to compute ISP parameters, so the "this is for tuning only"
argument doesn't hold as much as one may think. For real immutable
parameters, a large number of them are related to image processing steps
that are very common and found in most ISPs, such as lens shading
correction or geometric distorsion correction. I don't see why we should
let vendors keep those closed.

I'm sorry that this discussion is turning into something very
camera-centric, but that's the relevant area I know best. I hope that
discussing problems related to device pass-through in different areas in
the open will help build a wider shared understanding of the problems,
and hopefully help designing solutions by better taking into account the
various aspects of the issues.

Does closing the door have to come at the cost of a technically clean
solution? Doing validation in the kernel to enforce an ideological
position would severely degrade mlx5, and would probably never really
be robust.

-- 
Regards,

Laurent Pinchart

I know this is a topic proposed for the maintainers summit, but given
the number of people who seem to have an opinion and be interested in
dicussing it, would a session at LPC be a better candidate ? I don't
expect the maintainer summit to invite all relevant experts from all
subsystems, that would likely overflow the room.

The downside of an LPC session is that it could easily turn into a
heated stage fight, and there are probably also quite a few arguments
that can't really be made in the open :-S

-- 
Regards,

Laurent Pinchart

On Fri, 26 Jul 2024 17:27:31 +0300
Laurent Pinchart [off-list ref] wrote:

I know this is a topic proposed for the maintainers summit, but given
the number of people who seem to have an opinion and be interested in
dicussing it, would a session at LPC be a better candidate ? I don't
expect the maintainer summit to invite all relevant experts from all
subsystems, that would likely overflow the room.

The downside of an LPC session is that it could easily turn into a
heated stage fight, and there are probably also quite a few arguments
that can't really be made in the open :-S

You can always submit a BoF. We have also had last minute BoFs made if
there's rooms available.

-- Steve

On Fri, Jul 26, 2024 at 12:59 PM Laurent Pinchart
[off-list ref] wrote:

On Fri, Jul 26, 2024 at 10:04:33AM +0200, Ricardo Ribalda Delgado wrote:

quoted

On Thu, Jul 25, 2024 at 10:02 PM Laurent Pinchart wrote:

quoted

On Mon, Jul 22, 2024 at 01:56:11PM +0200, Ricardo Ribalda Delgado wrote:

quoted

On Mon, Jul 22, 2024 at 1:18 PM Laurent Pinchart wrote:

quoted

On Mon, Jul 22, 2024 at 12:42:52PM +0200, Ricardo Ribalda Delgado wrote:

quoted

On Sun, Jul 21, 2024 at 9:25 PM Laurent Pinchart wrote:

quoted

On Tue, Jul 09, 2024 at 03:15:13PM -0700, Dan Williams wrote:

quoted

James Bottomley wrote:

quoted

quoted

The upstream discussion has yielded the full spectrum of positions on
device specific functionality, and it is a topic that needs cross-
kernel consensus as hardware increasingly spans cross-subsystem
concerns. Please consider it for a Maintainers Summit discussion.

I'm with Greg on this ... can you point to some of the contrary
positions?

This thread has that discussion:

http://lore.kernel.org/0-v1-9912f1a11620+2a-fwctl_jgg@nvidia.com

I do not want to speak for others on the saliency of their points, all I
can say is that the contrary positions have so far not moved me to drop
consideration of fwctl for CXL.

Where CXL has a Command Effects Log that is a reasonable protocol for
making decisions about opaque command codes, and that CXL already has a
few years of experience with the commands that *do* need a Linux-command
wrapper.

Some open questions from that thread are: what does it mean for the fate
of a proposal if one subsystem Acks the ABI and another Naks it for a
device that crosses subsystem functionality? Would a cynical hardware
response just lead to plumbing an NVME admin queue, or CXL mailbox to
get device-specific commands past another subsystem's objection?

My default answer would be to trust the maintainers of the relevant
subsystems (or try to convince them when you disagree :-)). Not only
should they know the technical implications best, they should also have
a good view of the whole vertical stack, and the implications of
pass-through for their ecosystem. This may result in a single NAK
overriding ACKs, but we could also try to find technical solutions when
we'll face such issues, to enforce different sets of rules for the
different functions of a device.

Subsystem hopping is something we're recently noticed for camera ISPs,
where a vendor wanted to move from V4L2 to DRM. Technical reasons for
doing so were given, and they were (in my opinion) rather excuses. The
unspoken real (again in my opinion) reason was to avoid documenting the
firmware interface and ship userspace binary blobs with no way for free
software to use all the device's features. That's something we have been
fighting against for years, trying to convince vendors that they can
provide better and more open camera support without the world
collapsing, with increasing success recently. Saying amen to
pass-through in this case would be a huge step back that would hurt
users and the whole ecosystem in the short and long term.

In my view, DRM is a more suitable model for complex ISPs than V4L2:

I know we disagree on this topic :-) I'm sure we'll continue the
conversation, but I think the technical discussion likely belongs to a
different mail thread.

quoted

- Userspace Complexity: ISPs demand a highly complex and evolving API,
similar to Vulkan or OpenGL. Applications typically need a framework
like libcamera to utilize ISPs effectively, much like Mesa for
graphics cards.

- Lack of Standardization: There's no universal standard for ISPs;
each vendor implements unique features and usage patterns. DRM
addresses this through vendor-specific IOCTLs

- Proprietary Architectures: Vendors often don't fully disclose their
hardware architectures. DRM cleverly only necessitates a Mesa
implementation, not comprehensive documentation.

This point isn't technical and is more on-topic for this mail thread.

V4L2 doesn't require hundreds of pages of comprehensive documentation in
text form. An open-source userspace implementation that covers the
feature set exposed by the driver is acceptable in place of
documentation (provided, of course, that the userspace code wouldn't be
deliberately obfuscated). This is similar in spirit to the rule for GPU
DRM drivers.

In DRM vendors typically define a custom IOCTL per driver to pass
command buffers.
Only the command buffer structure, and a mesa implementation using
that command buffer to support the standard features is required.

In V4l2 custom IOCTLs are discouraged. Random command buffers cannot
be passed from userspace, they are typically formed in the driver from
a strictly checked struct.

V4L2 has a mechanism to pass buffers between userspace and kernelspace,
and that mechanism is used in mainline drivers to pass camera ISP
parameters. They're not called "command buffers" but that's just a
difference in terminology. The technical means to pass command buffers
to the driver is thus there, I see no meaningful difference with DRM.
Where things can differ is in the contents of those buffers, and the
requirements for documentation or open userspace implementations, but
that's not a technical question.

There are two things here:

- The political/strategic/philosophical/religious aspect: The industry
definitely prefers the strategic requirements imposed by DRM. In fact
some vendors had some huge legal troubles when they had tried to
follow v4l2 requirements.

That's I'm willing to debate.

quoted

- The technical aspect: DRM is more mature when it comes to
sending/receiving buffers to the hardware, and an ISP looks *much*
more similar to an accel device or a GPU than a UVC camera.

But this I don't agree with. I think we should forgo the technical
discussion and stop pretending that DRM is better for this use case from
a technical point of view, and focus on the other aspect of the
discussion. (We can of course reopen the technical discussion if new
concrete arguments emerge.)

I disagree. ISP devices are EXACTLY the same as accel devices. I
suspect that you want to avoid them handled in DRM because then you
prefer the v4l2 openness requirements. There are no technical benefit
from v4l2 for using ISPs:

Everyone agrees that the current V4L2 API is not very suitable for the
current generation of ISPs: it is too cumbersome.

https://linuxtv.org/news.php?entry=2022-11-14-1.hverkuil

quoted

quoted

quoted

quoted

quoted

Our current approach of pushing back against vendors, instead of
seeking compromise, has resulted in the vast majority of the market
(99% if not more) relying on out-of-tree drivers. This leaves users
with no options for utilizing their cameras outside of Android.

DRM allows a hybrid model, where:
- Open Source Foundation: Standard use cases are covered by a fully
open-source stack.
- Vendor Differentiation: Vendors retain the freedom to implement
proprietary features (e.g., automatic makeup) as closed source.

V4L2 does as well, you can implement all kind of closed-source ISP
control algorithms in userspace, as long as there's an open-source
implementation that exercises the same hardware features. A good analogy

Is it really mandatory to have an open-source 3A algorithm? I thought
defining the input and output from the algorithm was good enough.

What really matters is documenting the ISP parameters with enough
details to allow for the implementation of open-source userspace code.
Once you have that, 3A is quite simple. You can refine it (especially
AWB) to great length, for instance using NPUs to compute parameters, and
there's absolutely no issue with such userspace implementations being
closed.

In practice, some vendors prefer documenting the parameters by writing
an open-source userspace implementation, partly maybe because developers
are more familiar writing code than formal documentation. I would be
fine either way, as long as there's enough information to make use of
the ISP.

Even with vendor passthrough there is still a need to provide a full
open source implementation (probably based on libcamera).

We have a different interpretation of "full" :-) I want to aim for
"full" to cover all the features exposed by the driver UAPI. There could
be some exceptions that we can discuss if there are compeling arguments,
but that would result in a 99% coverage, not a 20%, 50% or 80% coverage.

Full means that I can use my camera using debian main and nothing more
than that.  It is a pretty accepted description of what fully open
means.

quoted

So you will have enough information to use all the common
functionality of a camera.

quoted

quoted

AFAIK for some time there was no ipu3 open source algorithm, and the
driver has been upstream.

It sneaked in before we realized we had to enforce rules :-) That's
actually a good example, when we wrote open-source userspace support, we
realized that the level of documentation included in the IPU3 kernel
header was nowhere close to what was needed to make use of the device.

quoted

quoted

for people less familiar with ISPs is shader compilers, GPU vendors are
free to ship closed-source implementations that include more
optimizations, as long as the open-source, less optimized implementation
covers the same GPU ISA, so that open-source developers can also work on
optimizing it.

I believe a more accurate description is that in v4l2 is that we
expect that all the registers, device architecture and behaviour to be
documented and accessed with standard IOCTLs. Anything not documented
cannot be accessed by userspace.

In DRM their concern is that there is a fully open source
implementation that the user can use. Vendors have custom IOCTLs and
they can offer proprietary software for some use cases.

Custom ioctls are not closed secrets in DRM, so comparing custom ioctls
vs. standard ioctls isn't very relevant to this discussion. I really
don't see how this would be about ioctls, it's about making the featured
exposed by the drivers, through whatever means a particular subsystem
allows, usable by open userspace.

quoted

quoted

Thinking that DRM would offer a free pass-through path compared to V4L2
doesn't seem realistic to me. Both subsystems will have similar rules.

DRM does indeed allow vendors to pass random command buffers and they
will be sent to the hardware. We cannot do that in v4l2.

You can pass a command buffer to a V4L2 device and have the driver send
it to the device firmware (ISPs using real command buffers usually run a
firmware). If you want the driver to be merged upstream, you have to
document the command buffer in enough details.

Documenting with "enough details" is not enough. In V4L2, we have to
deeply inspect every single buffer to make sure that it is not sending
an unknown combination of command+arguments, or in other situations we
construct the command buffer in the driver.

quoted

quoted

I might be wrong, but GPU drivers do not deeply inspect the command
buffers to make sure that they do not use any feature not covered by
mesa.

That's correct, but I don't think that's relevant. The GPU market has
GLSL and Vulkan. An open-source compliant implementation will end up
exercising a very very large part of the device ISA, command submission
mechanism and synchronization primitives, if not all of it. There's
little a vendor would keep under the hood and use in closed-source
userspace only. For cameras, there's no standard userspace API that
covers by design a very large part of what is the ISP equivalent of a

Libcamera supports Camera HAL3, gstreamer, v4l2, pipewire...

Those are not comparable to GLSL or Vulkan, they are much higher level.

quoted

If we are afraid of vendors providing "toy" implemententations to pass
the openness requirements, we can add more features/tests to
libcamera.

And at the end of the day, there will be humans deciding if what a
vendor has provided is good enough or not.

quoted

GPU ISA. Even "command buffers" are not a proper description of the
parameters the vast majority of ISPs consume.

Modern ISPs are definitely going in the direction of "command buffers"

Examples please, with technical details. I've seen this argument being
used for at least a year, with very little evidence. There are ISPs that
are more programmable than others, but in practice firmwares mostly
expose fixed functions, not ISA-level programmability.

You have not seen many programmable ISPs in v4l2 because they cannot
be supported with v4l2.
Instead vendors have to create huge firmwares that limits the
capability of the device.
I do not want to look into those firmwares.

quoted

quoted

quoted

quoted

quoted

This approach would allow billions of users to access their hardware
more securely and with in-tree driver support. Our current stubborn
pursuit of an idealistic goal has already negatively impacted both
users and the ecosystem.

The late wins, in my opinion, cannot scale to the consumer market, and
Linux will remain a niche market for ISPs.

If such a hybrid model goes against Linux goals, this is something
that should be agreed upon by the whole community, so we have the same
criteria for all subsystems.

--
Regards,

Laurent Pinchart

-- 
Ricardo Ribalda

On Fri, Jul 26, 2024 at 3:11 PM Laurent Pinchart
[off-list ref] wrote:

On Fri, Jul 26, 2024 at 10:02:27AM +0200, Ricardo Ribalda Delgado wrote:

quoted

On Thu, Jul 25, 2024 at 9:44 PM Laurent Pinchart wrote:

quoted

On Thu, Jul 25, 2024 at 04:20:35PM +0300, Leon Romanovsky wrote:

quoted

On Thu, Jul 25, 2024 at 03:02:13PM +0200, Ricardo Ribalda Delgado wrote:

quoted

On Thu, Jul 25, 2024 at 2:23 PM Leon Romanovsky wrote:

quoted

On Thu, Jul 25, 2024 at 11:26:38AM +0200, Ricardo Ribalda Delgado wrote:

quoted

On Wed, Jul 24, 2024 at 10:02 PM Laurent Pinchart wrote:

<...>

quoted

It would be great to define what are the free software communities
here. Distros and final users are also "free software communities" and
they do not care about niche use cases covered by proprietary
software.

Are you certain about that?

As a user, and as an open source Distro developer I have a small hint.
But you could also ask users what they think about not being able to
use their notebook's cameras. The last time that I could not use some
basic hardware from a notebook with Linux was 20 years ago.

Lucky you, I still have consumer hardware (speaker) that doesn't work
with Linux, and even now, there is basic hardware in my current
laptop (HP docking station) that doesn't work reliably in Linux.

quoted

quoted

quoted

They only care (and should care) about normal workflows.

What is a normal workflow?
Does it mean that if user bought something very expensive he
should not be able to use it with free software, because his
usage is different from yours?

Thanks

It means that we should not block the standard usage for 99% of the
population just because 1% of the users cannot do something fancy with
their device.

Right, the problem is that in some areas the statistics slightly different.
99% population is blocked because 1% of the users don't need it and
don't think that it is "normal" flow.

quoted

Let me give you an example. When I buy a camera I want to be able to
do Video Conferencing and take some static photos of documents. I do
not care about: automatic makeup, AI generated background, unicorn
filters, eyes recentering... But we need to give a way to vendors to
implement those things closely, without the marketing differentiators,
vendors have zero incentive to invest in Linux, and that affects all
the population.

I've seen these kind of examples being repeatedly given in discussions
related to camera ISP support in Linux. They are very misleading. These
are not the kind of features that are relevant for the device
pass-through discussion these day. Those are high-level use cases
implemented in userspace, and vendors can ship any closed-source
binaries they want there. What I care about is the features exposed by
the kernel to userspace API.

The ISPs are gradually becoming programmable devices and they indeed
help during all of those examples.

I'd like to see more technical information to substantiate this claim.
So far what I've sometimes seen is ISPs that include programmable
elements, but hiding those behind a firmware that exposes a fixed
(configurable) pipeline. I've also heard of attempts to expose some of
that programmability to the operating system, which were abandoned in
the end due to lack usefulness.

quoted

Userspace needs to send/receive information from the ISP, and that is
exactly what vendors want to keep in the close.

But that's exactly what we need to implement an open userspace ecosystem
:-)

quoted

Describing how they implement those algorithms is a patent minefield
and their differentiating factor.

Those are also arguments I've heard many times before. The
differentiating factor for cameras today is mostly in userspace ISP
control algorithms, and nobody is telling vendors they need to open all
that.

I disagree. The differentiating factor is what the ISP is capable of
doing and how they do it. Otherwise we would not see new ISPs in the
market.

If you define the arguments passed to an ISP you are defining the
algorithm, and that is a trade secret and/or a patent violation.

When it comes to patents, we all know how software patents is a
minefield, and hardware is also affected. I can't have much sympathy for
this argument though, those patents mostly benefit the largest players
in the market, and those are the ones who currently claim they can't
open anything due to patents.

Big players do not usually sue each other. The big problem is patent
trolls that "shoot at everything that moves".

I dislike patents, but it is the world we have to live in. No vendor
is going to take our approach if they risk a multi million dollar
lawsuit.

quoted

quoted

quoted

quoted

This challenge seems to be solved for GPUs. I am using my AMD GPU
freely and my nephew can install the amdgpu-pro proprietary user space
driver to play duke nukem (or whatever kids play now) at 2000 fps.

There are other other subsystems that allow vendor passthrough and
their ecosystem has not collapsed.

Yes, I completely agree with you on that.

quoted

Can we have some general guidance of what is acceptable? Can we define
together the "normal workflow" and focus on a *full* open source
implementation of that?

I don't think that is possible to define "normal workflow". Requirement
to have open-source counterpart to everything exposed through UAPI is a
valid one. I'm all for that.

That's my current opinion as well, as least when it comes to the kernel
areas I mostly work with.

--
Regards,

Laurent Pinchart

-- 
Ricardo Ribalda

On Fri, Jul 26, 2024 at 4:22 PM Laurent Pinchart
[off-list ref] wrote:

On Fri, Jul 26, 2024 at 10:11:06AM -0300, Jason Gunthorpe wrote:

quoted

On Fri, Jul 26, 2024 at 03:49:49PM +0300, Laurent Pinchart wrote:

quoted

What is not an option exactly in my description above ? We have multiple
V4L2 drivers for ISPs. They receive ISP parameters from userspace
through a data buffer. It's not allowed to be opaque, but it doesn't
prevent vendor closed-source userspace implementations with additional
*camera* features, as long as the *hardware* features are available to
everybody.

How far do you take opaque?

In mlx5 we pass command buffers from user to kernel to HW and the
kernel does only a little checking.

I won't comment on the mlx5 case in particular, I don't have enough
knowledge to do so.

When it comes to validation of commands by the kernel, at the very least
the kernel driver needs to be able to guarantee safety. For instance,
that means that any command that can result in DMA operations need to be
validated (e.g. verifying buffer addresses and sizes) and/or executed
partly by the driver (e.g. mapping a buffer through an IOMMU), depending
on hardware constraints.

quoted

There is a 12kloc file describing the layout of alot of commands:
include/linux/mlx5/mlx5_ifc.h

There is an open PDF describing in detail some subset of this:
https://network.nvidia.com/files/doc-2020/ethernet-adapters-programming-manual.pdf

There are in-kernel implementations driving most of those commands.

For camera ISPs, most of the "commands" wouldn't be driven by the
kernel. I don't have expectations when it comes to what commands the
kernel exercises directly, I think that highly depends on the device
type.

quoted

Other commands are only issued by userspace, and we have open source
DPDK, rdma-core and UCX implementations driving them.

ie, this is really quite good as far as a device providing open source
solutions goes.

However, no doubt there is more FW capability and commands than even
this vast amount documents - so lets guess that propritary code is
using this interface with unknown commands too.

From a camera perspective would you be OK with that? Let's guess that
90% of use cases are covered with fully open code. Do we also need to
forcefully close the door to an imagined 10% of proprietary cases just
to be sure open always wins?

For command buffers interpreted by a firmware, it would be extremely
difficult, if not impossible, to ensure that nothing undocumented is
possible to access from userspace. I think we have two issues here, one
of them is to define what we required to be openly accessible, and the
other to avoid vendors cheating by claiming to be more open than they
actually are.

I believe the latter can't be solved technically. At the end of the day
it's a matter of trust, and the only option I can see is to build that
trust with the vendors, and to make it clear that breaches of trust will
have consequences. A vendor that deliberately lies would end up on my
personal backlist for as long as they don't implement structural
solutions to be a better player in the ecosystem. What is acceptable as
consequences is for the communities to decide. We have previously banned
people or organizations from contributing to the kernel for certain
periods of time (the University of Minnesota case comes to my mind for
instance), and other options can be considered too.

As for the first issue, I think it's a difficult one as it is very
difficult to quantify the features covered by open implementations, as
well as their importance. You could have a 99% open command set where
the 1% would impact open implementations extremely negatively, the same
way you could have a 50% open command set where the other 50% isn't of
any use to anyone but the vendor (for instance used to debug the
firmware implementation).

It is not that difficult. You just have to define what an acceptable
implementation is. Eg

- Sharpness at specific light.
- Time of convergence for the Auto algorithms
- Noise ratio

We could even use the SoftISP implementation as reference. Is this ISP
working better or worse than SoftISP?

For cameras, the feature set can I believe be expressed in terms of ISP
processing blocks that are usable by open implementations, as well as in
terms of the type of usage of those features. For instance, is it
acceptable that a vendor documents how to use 2D noise reduction but
makes 3D (temporal) noise reduction available only to close-source
userspace ? My personal answer is no. I want to aim for very close to
100% of the features, and certainly 100% of the "large" features. I can
close my eyes on features that are very very niche, but what is niche
today may not be tomorrow. For instance, if a camera ISP implements a
feature used only for very specific camera sensors targetted at
autonomous driving in a new generation of research prototypes that only
one company has access to, I'll be inclined to ignore that. That
technology may however become much more mainstream 5 years later.

We can update our requirements in 5 years. Nothing is written in stone.

Also it is much easier to reverse engineer an open driver, with an
open userspace and a closed userspace than a close driver with a
closed userspace.

The other aspect is the type of usage of the features. For camera ISPs
again, some parameters will be computed through a tuning process, and
will be fixed for the lifetime of the camera. I want those parameters to
be documented, to enable users to tune cameras for their own use cases.
This is less important in some market segments, such as laptops for
instance, but is crucial for embedded devices. This is an area where
I've previously been called unreasonable, and I don't think that's fair.
The tuned-and-immutable parameters are not plentiful, as most of the
tuning results in data that needs to be combined with runtime
information to compute ISP parameters, so the "this is for tuning only"
argument doesn't hold as much as one may think. For real immutable
parameters, a large number of them are related to image processing steps
that are very common and found in most ISPs, such as lens shading
correction or geometric distorsion correction. I don't see why we should
let vendors keep those closed.

We don't have enough leverage for that kind of requirement.

To be fair, we do not ask touchscreen manufacturers to document their
calibration files. Nor any other calibration file in the kernel.

The calibration file for me is like a firmware blob.

I'm sorry that this discussion is turning into something very
camera-centric, but that's the relevant area I know best. I hope that
discussing problems related to device pass-through in different areas in
the open will help build a wider shared understanding of the problems,
and hopefully help designing solutions by better taking into account the
various aspects of the issues.

quoted

Does closing the door have to come at the cost of a technically clean
solution? Doing validation in the kernel to enforce an ideological
position would severely degrade mlx5, and would probably never really
be robust.

--
Regards,

Laurent Pinchart

--
Ricardo Ribalda

On Fri, Jul 26, 2024 at 3:22 PM Laurent Pinchart
[off-list ref] wrote:

On Fri, Jul 26, 2024 at 10:03:36AM +0200, Ricardo Ribalda Delgado wrote:

quoted

On Thu, Jul 25, 2024 at 9:32 PM Laurent Pinchart wrote:

quoted

On Wed, Jul 24, 2024 at 04:37:21PM -0400, James Bottomley wrote:

quoted

On Wed, 2024-07-24 at 23:00 +0300, Laurent Pinchart wrote:
[...]

quoted

What I get from the discussions I've followed or partcipated in over
the years is that the main worry of free software communities is
being forced to use closed-source userspace components, whether that
would be to make the device usable at all, or to achieve decent level
of performance or full feature set. We've been through years of
mostly closed-source GPU support, of printer "windrivers", and quite
a few other horrors. The good news is that we've so far overcome lots
(most) of those challenges. Reverse engineering projects paid off,
and so did working hand-in-hand with industry actors in multiple ways
(both openly and behind the scenes). One could then legitimately ask
why we're still scared.

I don't think I am.  We're mostly fully capable of expounding at length
on the business rationale for being open if the thing they're hiding
isn't much of a differentiator anyway (or they're simply hiding it to
try to retain some illusion of control), so we shouldn't have any fear
of being able to make our case in language business people understand.

I also think this fear is partly a mindset problem on our part.  We
came out of the real fight for openness and we do embrace things like a
licence that forces open code (GPL) and symbols that discourage
proprietary drivers (EXPORT_SYMBOL_GPL), so we've somewhat drunk the
FSF coolaid that if we don't stand over manufacturers every second and
force them they'll slide back to their old proprietary ways.  However,
if you look at the entirely permissive ecosystem that grew up after we
did (openstack, docker, kubernetes, etc.) they don't have any such fear
and yet they still have large amounts of uncompelled openness and give
back.

I don't think those are necessarily relevant examples, as far as device
pass-through goes. Vendors have many times reverted to proprietary ways,
and they still do, at least in the areas of the kernel I'm most active
in. I've seen first hand a large SoC vendor very close to opening a
significant part of their camera stack and changing their mind at the
last minute when they heard they could possibly merge their code through
a different subsystem with a pass-through blank cheque.

Without knowing who that large SoC vendor is, and what they will be
willing to open, it is difficult to know what opportunity has been
lost.

I would also like to discuss this in more details, but I can't do so
publicly :-( That's why I don't want to focus too much on this argument.

quoted

I would argue that if they have cancelled their open plans based
on an hypothesis, their willingness to open was not that high.

That's exactly what I previously described in this mail thread as areas
where cooperation with vendors is in its infancy. Convincing large
industry players is long term work, and it's very demotivating when you
think there's finally light at the end of the tunel, and plans gets
cancelled at the last minute. We're not giving up though.

quoted

It would be more healthy for the ecosystem, if those discussions were
done more openly, at least all the core maintainers should be
involved.

Can you get all the camera vendors you work with to talk about all this
openly ? :-) We all know it's a difficult and long process.

We have pushed lots of vendors to contribute directly upstream. You
can see a lot of code that has landed thanks to this effort.
And I have tried very hard to get them involved in all the discussions upstream.

quoted

quoted

I'm willing to believe it can be different in other areas, which may
partly explain why different subsystems and different developers have
different biases and have trouble understand each other's point of view.

It is not different in other areas, it is the same area. At the end of
the day it is the same chip manufacturers, with the same legal teams.
It is our attitude that is different.

I don't agree with that. Not only are there different manufacturers,
there are also different teams, with different approaches. The world is
not uniform.

quoted

quoted

quoted

quoted

I can't fully answer that question, but there are two points that I
think are relevant. Note that due to my background and experience,
this will be heavily biased towards consumer and embedded hardware,
not data centre-grade devices. Some technologies from the latter
however have a tendency to migrate to the former over time, so the
distinction isn't necessarily as relevant as one may consider.

The first point is that hardware gets more complicated over time, and
in some markets there's also an increase in the number of vendors and
devices. There's a perceived (whether true or not) danger that we
won't be able to keep up with just reverse engineering and a
development model relying on hobyists. Getting vendors involved is
important if we want to scale.

Yes, but there are lots of not very useful complex devices being
produced every day that fail to capture market share.  Not having
reverse engineered drivers for them is no real loss.  If a device does
gain market share, it gains a huge pool of users some of whom become
interested in reverse engineering, so I think market forces actually
work in our favour: we get reverse engineering mostly where the devices
are actually interesting and capture market share.  It's self scaling.

I can't agree with that, sorry. Not only is the difficulty to
reverse-engineer some classes of devices increasing, but saying that
only devices that make it to the top of the market share chart are worth
considering will leave many users on the side of the road.

Today we have left BILLIONS of users at the other side of the road.

Because we're the only ones to be blamed, or more accuretely, the blame
is solely on me. Vendors are all saints who want the best for the world.
This argument doesn't hold. Please stop with this, I don't want to get
to name calling.

Nobody said that it was your fault. You and all the media community
have invested thousands of hours of their personal time in Linux.

It is healthy to make a retrospective of where we are now and if we
are helping our users or not.

The reality is that we are reaching a very small percentage of users,
and vendors prefer to work out of the tree than with us. Even vendors
that traditionally have been very good Linux players have issues
landing their code.

Can we try to figure out why it is happening? Can we try something
new? Maybe inspired in other subsystems?

quoted

quoted

quoted

quoted

Second, I think there's a fear of regression. For some categories of
devices, we have made slow but real progress to try and convince the
industry to be more open. This sometimes took a decade of work,
patiently building bridges and creating ecosystems brick by brick.
Some of those ecosystems are sturdy, some not so. Giving pass-through
a blank check will likely have very different effects in different
areas. I don't personally believe it will shatter everything, but I'm
convinced it carries risk in areas where cooperation with vendors is
in its infancy or is fragile for any other reason.

I also think we're on the rise in this space.  Since most cloud
workloads are on Linux, there's huge market pressure on most "found in
the cloud" devices (like accelerators and GPUs) to have an easy to
consume Linux story.  Nvidia is a case in point.  When it only cared
about fast games on some other OS, we get shafted with a proprietary
graphics drivers.  Now it's under pressure to be the number one AI
accelerator provider for the cloud it's suddenly wondering about open
source drivers to make adoption easier.

I can't comment on Nvidia and their inference engines in particular. The
server market may be in a better position that the consumer and embedded
market, and if that's the case, I'm happy for the servers. That doesn't
solve the issues in other markets though.

quoted

quoted

Finally, let's not forget that pass-through APIs are not an all or
nothing option. To cite that example only, DRM requires GPU drivers
to have an open-source userspace implementation to merge the kernel
driver, and the same subsystems strongly pushes for API
standardization for display controllers. We can set different rules
for different cases.

I certainly think we can afford to experiment here, yes.

--
Regards,

Laurent Pinchart

--
Ricardo Ribalda