On Thu, Feb 05, 2026 at 10:40:12AM +0100, Jacopo Mondi wrote:

On Wed, Feb 04, 2026 at 07:30:18PM +0100, Antoine Bouyer wrote:

quoted

Le 04/02/2026 à 18:12, Jacopo Mondi a écrit :

quoted

On Tue, Feb 03, 2026 at 07:37:34PM +0100, Jacopo Mondi wrote:

quoted

On Thu, Jan 29, 2026 at 12:00:24AM +0100, Michael Riesch wrote:

quoted

On 1/28/26 09:17, Antoine Bouyer wrote:

quoted

On 1/26/26 10:44 AM, Michael Riesch wrote:

quoted

On 1/23/26 09:09, Antoine Bouyer wrote:

quoted

Hi all,

This RFC patch series introduces the NXP Neo Image Signal Processor (ISP)
driver, used in the NXP i.MX95 SoC and future devices in the i.MX9 family.
The series also includes updates to the generic v4l2-isp interface to
support extended statistics required by the Neo ISP.

The Neo ISP processes one or more camera streams, converting RAW formats
into YUV or RGB outputs. Its architecture is largely influenced by the
PISP driver. The hardware supports up to eight contexts, with three sink
pads (main input, HDR input, and parameter buffers) and three source pads
(RGB output, IR output, and statistics metadata).

At this stage, both legacy (fixed-size) and extensible (dynamic-size)
parameter/statistics buffers are supported through the generic v4l2-isp
framework, similar to rkisp1 and Mali-C55. The driver currently supports
M2M operation; direct CSI-to-ISP streaming is not yet implemented.

How do you envisage the direct CSI-to-ISP streaming shall be supported?

At this stage, this streaming mode still needs to be evaluated on
neoisp. We should follow the integration model used by existing ISP
drivers to avoid duplicating solutions.

Fair point, but I have had the impression that there are not many
examples (if any). The rkisp1 driver, for instance, only supports inline
mode although the HW should be able to do both.

But any pointers most welcome, I won't claim I have the full overview.

quoted

Below are my initial thoughts on the specific points you raised:

quoted

   - How shall the final media graph(s) look like?

The media entities would remain mostly identical, except for the absence
of ISI. The topology would be a direct linkg from sensor->csi-

quoted

formatter->neoisp.

If support for inline mode has to be added later, the ISP will need to
be registered in the same media graph of the CSI-2 receiver to be able
to link the two, right ?

yes correct.

quoted

quoted

How do you envision to control the ISP operating mode, because I'm
afraid if you register the ISP in its own media graph, you're locking
yourself there as implementing inline mode would require a different
media topology with all the implications on the rest of the userspace
stack.

This might not be a problem if you know that the inline vs m2m mode is
SoC sythesis time parameter. Some SoCs will integrate neoisp inline, some
other as m2m. In this case you'll likely need two pipeline handlers
in libcamera, but if that's per SoC-line maybe is acceptable. The fact
you suggests in inline mode there won't be an ISI makes me think this
actually depends on the SoC design ?

Actually, this is not really at SoC synthesis time, neoisp HW does support
both modes, that is configurable. But ISP HW can run in a single mode only
once it is configured. Streaming mode is tightly coupled with CSI HW, then
ISP cannot be used in M2M mode with another sensor simultaneously.

Yes, my point is trying to understand "how it is configured" and what
your expectations are.

Will the board .dts (or a camera .dtso) decide how the ISP is operated
by defining its endpoint connections ? Assuming with the same SoC both
inline and m2m modes are possible, without differences in the SoC
design/integration, will users of the same board have to modify the
.dts or load ad-hoc .dtso to decide what mode is in use ?

That's what we do on i.MX8MP, to route the CSI-2 receiver output to
either the ISI or the ISP. And it was a really bad design mistake. Let's
not repeat it: if the hardware allows both inline and offline operation,
then selecting between the two modes should be possible without changing
the device tree, without rebooting and without reloading modules. All
components must be in the same media graph. This includes the CSI-2
receiver, ISI and ISP.

Then, the question of how the media topology will look and which
components registers what has to be clarified.

Let's try to make a taxonomy of the cases we have in mainline (or on
their way to mainline).

In the mali example I mentioned, the operating mode is selected by the
.dtsi as Mali can be integrated either inline or in m2m mode in
different SoCs. RZ/V2H in example, will always be m2m as it doesn't
interface the CSI-2 receiver with the ISP but rather interfaces the
ISP with a companion chip the performs memory access on its behalf
(the IVC). A different design that incorporates Mali inline will
instead have to interface the CSI-2 receiver with the ISP with
internal busses/glue logic and will then have to described this in dts.

This is fine as the ISP integration is different and then having the
description in dts is legit.

Note that the C55 could be integrated in an SoC in a way that would
support both inline and offline modes. In that case selection of the
operation mode must be done by userspace, without changing DT.

The ISP driver unconditionally registers an async notifier and the
downstream component (csi-2 or IVC) will register its async subdev(s)
which will all appear in the ISP media graph. This is possible because
the assumption is that the CSI-2 receiver (or the companion chip)
won't register their own media graph.

The Renesas V4H example I mentioned is instead different. The ISP can
be operated in inline and m2m, on the same SoC without any
modification to hardware and to the dts/dtsi. It's basically a user
choice we defer to runtime.

The V4H already has a component that registers a media graph: the
CSI-2/VIN block which is found in many SoCs of the same (and older)
generations. The ISP is present only in some SoC, but the CSI-2/VIN is
always there. In this case, to support both inline and m2m modes, the
VIN registers the media device and, with the trick I pointed you to in
Niklas' code, the ISP registers a subdev in the VIN media graph. Then
the inline/m2m mode can be selected by media link enablement at
run-time. Now, inline mode is not yet supported on V4H and there might
be dragons there, but at least, both modes should be possible on the same
SoC.

On the other extremes we have the RaspberryPi PiSP BE and RkISP1.

RPi knows the only SoC where the PiPS will be found is their one. The
ISP cannot function inline and will always be m2m. In this case, a
dedicated media graph for the ISP is the simplest and cleanest
solution.

RkISP1 instead will always be inline only. It registers a media device
and an async notifier, the connected CSI-2 receiver will register an
async subdev and will be connected to the device tree endpoint of the
ISP device node.

What model is the closest one to the neoisp integration that you
envision on NXP SoCs ?

quoted

quoted

One small correction after some more research:

we actually already have a pipeline in libcamera that supports inline
and (will soon) support m2m: the mali c55 one. My take on "probably
need two pipeline handlers" was not correct then.

Yes, I saw your patchwork on libcamera about this coming upgrade. Spent some
time analyzing it ':) Seems we are quite aligned as per my understanding:
inline mode (i.e. streaming mode with neoisp) _or_ M2M mode using IVC video
device from Mali. Is that right ?

quoted

As said, Mali-C55 can be integrated inline or in m2m mode and this is
decided based on the device tree endpoint connections.

Good. Do you have an example available ?

It's in mainline, but there's nothing exciting there as the assumption
is that there will always be a connection on the first endpoint and
the driver simply registers a notifier for the connected async subdev. If
it's a CSI-2 receiver then we're inline. If it's a companion chip
we're m2m.

The libcamera pipeline (not upstream yet) inspects the media entity
function of the entity connected to the ISP sink pad#0. If it's a
CSI-2 reciver we're inline. If it's not, we're m2m. Based on that it
operated the pipeline differently.

quoted

quoted

So, if you know neoisp will be integrated either inline or m2m in
different SoC lines, maybe deferring it to device tree is good enough
at the expense of a slightly more complicated pipeline ?

As said, SoC/ISP HW does support both modes. But I think that the selection
can be done in device tree too. So that after bootup, a camera will be used
only in 1 mode.

quoted

I guess this has implications on the bindings definition as well..

Most probably yes. Can this be done as second phase once evaluation is
completed ?

I think you should asses from the very beginning what is the planned
integration model of the ISP in order not to corner yourself in a
place where it will be hard to support inline without re-writing
the driver's media device registration logic.

Looking at the below media graph of CSI/ISI you should ask the question "how
will I register the ISP subdev in the CSI-2 media graph when inline"
and "how will I describe inline vs m2m mode if the underlying hardware
design doesn't change?" as deferring it to the .dts might not be the
most correct way to go in that case ?

The driver doesn't need to support all modes right away, but we need to
design the DT bindings and media graph to ensure that additional modes
can be added later without breaking backward compatibility.

quoted

quoted

quoted

However, if you plan to allow deferring inline/m2m mode selection to
the system integrators or even have it as a run-time parameter, then
you should really consider having the ISP in the same media graph as
the CSI-2 receiver and operate the whole CSI-2/ISI/ISP as a single
media graph, where you could select the operating mode through media link
enablement or dts endpoint connections

Niklas (in cc) has addressed a similar situation, where inline and m2m
mode can be selected by link enablement at runtime here
https://patchwork.linuxtv.org/project/linux-media/patch/20251225171054.1370856-3-niklas.soderlund+renesas@ragnatech.se/
(see risp_cs_internal_ops)

quoted

OK, I thought that ISI was still around...

quoted

quoted

   - How many media devices are registered and which driver registers it
     or them?

That will be part of the evaluation. My initial assumption is that
neoisp would be the appropriate component to register the media device
in this mode, since ISI is not involved, and ISI currently performs the
registration in the M2M configuration.

Isn't the ISP registering its own media graph ?

Yes, 8 copies of ISP media graph, that can be used with the 8 output video
devices of the ISI media graph.

I suggest you do what RPi does. The mainline driver only registers one
instance and they carry a little patch downstream that implements the
for() loop where multiple instances are registered. Duplicating media graphs
is not desirable (at least in mainline) as we can have ISPs with 256
contexts, we don't want 256 media graphs.

A framework level solution with proper priority handling and job
scheduling is what is required and that's what the context work should
end up being.

quoted

quoted

quoted

Can we get a copy of all media graphs on an i.MX95 system including
the ISI and the CSI-2 receiver ?

Here is an example with multiple sensors. Or do you need it in another
format ?

No it's fine, thanks!

quoted

digraph board {
        rankdir=TB
        n00000001 [label="{{<port0> 0 | <port1> 1 | <port2> 2 | <port3> 3 | <port4> 4} | crossbar\n/dev/v4l-subdev8 | {<port5> 5 | <port6> 6 | <port7> 7 | <port8> 8 | <port9> 9 | <port10> 10 | <port11> 11 | <port12> 12}}", shape=Mrecord, style=filled, fillcolor=green]
        n00000001:port5 -> n0000000f:port0 [style=bold]
        n00000001:port6 -> n0000001a:port0 [style=bold]
        n00000001:port7 -> n00000025:port0 [style=bold]
        n00000001:port8 -> n00000030:port0 [style=bold]
        n00000001:port9 -> n0000003b:port0 [style=bold]
        n00000001:port10 -> n00000046:port0 [style=bold]
        n00000001:port11 -> n00000051:port0 [style=bold]
        n00000001:port12 -> n0000005c:port0 [style=bold]
        n0000000f [label="{{<port0> 0} | mxc_isi.0\n/dev/v4l-subdev9 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
        n0000000f:port1 -> n00000012 [style=bold]
        n00000012 [label="mxc_isi.0.capture\n/dev/video8", shape=box, style=filled, fillcolor=yellow]
        n0000001a [label="{{<port0> 0} | mxc_isi.1\n/dev/v4l-subdev10 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
        n0000001a:port1 -> n0000001d [style=bold]
        n0000001d [label="mxc_isi.1.capture\n/dev/video9", shape=box, style=filled, fillcolor=yellow]
        n00000025 [label="{{<port0> 0} | mxc_isi.2\n/dev/v4l-subdev11 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
        n00000025:port1 -> n00000028 [style=bold]
        n00000028 [label="mxc_isi.2.capture\n/dev/video10", shape=box, style=filled, fillcolor=yellow]
        n00000030 [label="{{<port0> 0} | mxc_isi.3\n/dev/v4l-subdev12 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
        n00000030:port1 -> n00000033 [style=bold]
        n00000033 [label="mxc_isi.3.capture\n/dev/video13", shape=box, style=filled, fillcolor=yellow]
        n0000003b [label="{{<port0> 0} | mxc_isi.4\n/dev/v4l-subdev13 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
        n0000003b:port1 -> n0000003e [style=bold]
        n0000003e [label="mxc_isi.4.capture\n/dev/video14", shape=box, style=filled, fillcolor=yellow]
        n00000046 [label="{{<port0> 0} | mxc_isi.5\n/dev/v4l-subdev14 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
        n00000046:port1 -> n00000049 [style=bold]
        n00000049 [label="mxc_isi.5.capture\n/dev/video21", shape=box, style=filled, fillcolor=yellow]
        n00000051 [label="{{<port0> 0} | mxc_isi.6\n/dev/v4l-subdev15 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
        n00000051:port1 -> n00000054 [style=bold]
        n00000054 [label="mxc_isi.6.capture\n/dev/video22", shape=box, style=filled, fillcolor=yellow]
        n0000005c [label="{{<port0> 0} | mxc_isi.7\n/dev/v4l-subdev16 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
        n0000005c:port1 -> n0000005f [style=bold]
        n0000005f [label="mxc_isi.7.capture\n/dev/video23", shape=box, style=filled, fillcolor=yellow]
        n00000067 [label="mxc_isi.output\n", shape=box, style=filled, fillcolor=yellow]
        n00000067 -> n00000001:port4 [style=bold]
        n0000006e [label="{{<port0> 0} | 4ac10000.syscon:formatter@20\n/dev/v4l-subdev17 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
        n0000006e:port1 -> n00000001:port2 [style=bold]
        n00000073 [label="{{<port0> 0} | csidev-4ad30000.csi\n/dev/v4l-subdev18 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
        n00000073:port1 -> n0000006e:port0 [style=bold]
        n00000078 [label="{{<port0> 0 | <port1> 1 | <port2> 2 | <port3> 3} | max96724 2-0027\n/dev/v4l-subdev19 | {<port4> 4 | <port5> 5}}", shape=Mrecord, style=filled, fillcolor=green]
        n00000078:port4 -> n00000073:port0 [style=dashed]
        n00000081 [label="{{} | mx95mbcam 8-0040\n/dev/v4l-subdev20 | {<port0> 0}}", shape=Mrecord, style=filled, fillcolor=green]
        n00000081:port0 -> n00000078:port0 [style=bold]
        n00000085 [label="{{} | mx95mbcam 9-0040\n/dev/v4l-subdev21 | {<port0> 0}}", shape=Mrecord, style=filled, fillcolor=green]
        n00000085:port0 -> n00000078:port1 [style=bold]
        n00000089 [label="{{} | mx95mbcam 10-0040\n/dev/v4l-subdev22 | {<port0> 0}}", shape=Mrecord, style=filled, fillcolor=green]
        n00000089:port0 -> n00000078:port2 [style=bold]
        n0000008d [label="{{} | mx95mbcam 11-0040\n/dev/v4l-subdev23 | {<port0> 0}}", shape=Mrecord, style=filled, fillcolor=green]
        n0000008d:port0 -> n00000078:port3 [style=bold]
}

quoted

quoted

If I'm not mistaken you'll have 8 copies of the ISP media graphs, and
that's exactly what we're working on with the context framework :)

Ok. Then I should have a look to context framework too ...

Please, I hope to be able to resume working on it sooner or later
given the right use case.

quoted

quoted

quoted

quoted

... since it is not, your assumption seems very reasonable.

quoted

quoted

   - How can the user decide whether direct (csi2isp) or indirect
     (mem2mem) streaming shall be used?

That will also be part of the evaluation. From dts would be my first
option, but may prevent using both modes on same platform then.

Of course this depends what the hardware is able to do, but in case the
HW is reconfigurable easily, I doubt that device tree is a good choice
to solve that.

quoted

quoted

While it is certainly OK to introduce this support only at a later
stage, it makes sense to consider this right from the start to avoid
some nasty changes e.g. in how this hardware is exposed to user space.

Also, we are facing a similiar challenge with recent Rockchip ISP
hardware (RK3588, RK3576, ...) and it would be great to hear your
thoughts about that.

Is there an existing discussion thread available on this topic? I would
be very interested in following it.

Not yet, I am afraid. But there should be one or two soon (TM) :-)

It's probably time to have one :)

Good. Please loop me in ;)

You are in, this is the conversation ;)

It might be a good discussion point for the media summit in Nice
co-located with Embedded Recipes if people with interest in the topic
will going the be there.

Possibly even for a whole day workshop on Monday before the media
summit.

I'm also adding Anthony from ARM as I know he's going through the same
inline/m2m duality you're now facing.

quoted

quoted

quoted

quoted

quoted

quoted

quoted

This series is posted as RFC because extending the v4l2-isp interface may
overlap with ongoing work. If similar development already exists, I am
happy to rebase or adapt the series accordingly. If preferred, the series
can also be split into two parts: the v4l2-isp rework and the Neo ISP
driver introduction.

A few checkpatch warnings in v4l2-ioctl.c remain intentionally to stay
consistent with the existing style in that file.

Testing was performed on the i.MX95 EVK using the media/next kernel in
standalone M2M mode. End-to-end camera-to-ISP capture has been validated
using the downstream NXP kernel, as some hardware dependencies are not
yet upstreamed.

[snip]

-- 
Regards,

Laurent Pinchart