[RFC v1 04/11] media: Documentation: Add NXP neoisp driver documentation
From: Antoine Bouyer <hidden>
Date: 2026-01-23 08:06:27
Also in:
linux-devicetree, linux-media, lkml
Subsystem:
media input infrastructure (v4l/dvb), the rest · Maintainers:
Mauro Carvalho Chehab, Linus Torvalds
Document the NXP neoisp driver both in the admin-guide for neoisp IP description, and in the userpace-api for neoisp interface description. Signed-off-by: Antoine Bouyer <redacted> --- .../admin-guide/media/nxp-neoisp-diagram.dot | 22 ++ .../admin-guide/media/nxp-neoisp.dot | 16 ++ .../admin-guide/media/nxp-neoisp.rst | 189 ++++++++++++++++++ .../admin-guide/media/v4l-drivers.rst | 1 + .../userspace-api/media/v4l/meta-formats.rst | 1 + .../media/v4l/metafmt-nxp-neoisp.rst | 114 +++++++++++ 6 files changed, 343 insertions(+) create mode 100644 Documentation/admin-guide/media/nxp-neoisp-diagram.dot create mode 100644 Documentation/admin-guide/media/nxp-neoisp.dot create mode 100644 Documentation/admin-guide/media/nxp-neoisp.rst create mode 100644 Documentation/userspace-api/media/v4l/metafmt-nxp-neoisp.rst
diff --git a/Documentation/admin-guide/media/nxp-neoisp-diagram.dot b/Documentation/admin-guide/media/nxp-neoisp-diagram.dot
new file mode 100644
index 000000000000..ac0d950bc1dc
--- /dev/null
+++ b/Documentation/admin-guide/media/nxp-neoisp-diagram.dot@@ -0,0 +1,22 @@ +digraph G { + rankdir = "LR"; + node [shape=rect]; + splines = ortho; + label = "Neoisp pipeline diagram"; + {In0, In1 } -> HC -> {HDR_decomp0, HDR_decomp1}; + HDR_decomp0 -> OBWB0 -> HDR_merge; + HDR_decomp1 -> OBWB1 -> HDR_merge; + HDR_merge -> RGBIR -> IR_compression; + RGBIR -> Statistics; + RGBIR -> OBWB2 ->BNR -> Vignetting -> ColorTemp; + Vignetting -> Demosaic -> RGB2YUV -> DRC -> AF; + DRC-> NR -> EE -> DF -> Gamma -> Packetizer; + DRC -> DMAP -> DF[weight=2]; + DRC -> CCONV -> CAS -> Gamma; + {rank = "same"; RGBIR, DRC} + {rank = "same"; AF, NR, DMAP} + {rank = "same"; IR_compression, Vignetting} + IR_compression -> AXIOutDMA; + Packetizer -> AXIOutDMA [weight=3]; + AXIOutDMA -> {"Out0", "Out1"} +}
diff --git a/Documentation/admin-guide/media/nxp-neoisp.dot b/Documentation/admin-guide/media/nxp-neoisp.dot
new file mode 100644
index 000000000000..abcc2dc9bb55
--- /dev/null
+++ b/Documentation/admin-guide/media/nxp-neoisp.dot@@ -0,0 +1,16 @@ +digraph board { + rankdir=TB + n00000001 [label="{{<port0> 0 | <port1> 1 | <port2> 2} | neoisp\n | {<port3> 3 | <port4> 4 | <port5> 5}}", shape=Mrecord, style=filled, fillcolor=green] + n00000001:port3 -> n00000020 [style=dashed] + n00000001:port4 -> n00000022 [style=dashed] + n00000001:port5 -> n00000024 [style=dashed] + n0000000a [label="neoisp-input0\n/dev/video0", shape=box, style=filled, fillcolor=yellow] + n0000000a -> n00000001:port0 [style=bold] + n00000010 [label="neoisp-input1\n/dev/video1", shape=box, style=filled, fillcolor=yellow] + n00000010 -> n00000001:port1 [style=dashed] + n00000016 [label="neoisp-params\n/dev/video2", shape=box, style=filled, fillcolor=yellow] + n00000016 -> n00000001:port2 [style=dashed] + n00000020 [label="neoisp-frame\n/dev/video3", shape=box, style=filled, fillcolor=yellow] + n00000022 [label="neoisp-ir\n/dev/video4", shape=box, style=filled, fillcolor=yellow] + n00000024 [label="neoisp-stats\n/dev/video5", shape=box, style=filled, fillcolor=yellow] +}
diff --git a/Documentation/admin-guide/media/nxp-neoisp.rst b/Documentation/admin-guide/media/nxp-neoisp.rst
new file mode 100644
index 000000000000..0632de6bc51d
--- /dev/null
+++ b/Documentation/admin-guide/media/nxp-neoisp.rst@@ -0,0 +1,189 @@ +.. SPDX-License-Identifier: GPL-2.0 + +======================================= +NXP Neo Image Signal Processor (neoisp) +======================================= + +Introduction +============ + +Neoisp performs a set of image processing tasks on the RAW camera stream, where +the input camera stream and Neoisp processed output image are stored in DDR or +any system memory fast enough to keep up with Neoisp processing. +The overall Neoisp operation is frame based, that is 1 complete image frame is +read and output pixel by pixel, line by line wise. + +Revisions +========= + +NXP Neoisp hw revisions are listed in UAPI in enum :c:type:`neoisp_version_e`. +Version is stored in the Media device information, hw revision field, +accessible from the ioctl MEDIA_IOC_DEVICE_INFO. + +There are 2 versions: + +- NEOISP_HW_V1: used at least in i.MX95 A0/A1 SoC, not targeted for production. +- NEOISP_HW_V2: used at least in i.MX95 B0 SoC. + +Neoisp hardware +=============== + +The Neoisp registers and pipeline processing are documented in the NXP Image +Signal Processor Specification document (under NDA). + +The NEO pipeline block diagram is shown below: + +.. kernel-figure:: nxp-neoisp-diagram.dot + :alt: Diagram of neoisp pipeline + :align: center + +It is composed by the following HW blocks: + +- a Head Color (HC) selection block, +- two HDR decompression blocks, one for each input, +- three Optical Black correction and White Balance (OBWB) blocks at different + stages in the pipeline, +- a HDR merge block for HDR image capture from the 2 input lines, +- a RGB-IR to RGGB converter, +- a Bayer Noise Reduction (BNR) block, +- a Vignetting block, aka Lens Shading Correction (LSC), +- a Demosaic block for RAW image conversion to RGB, +- a RGB Color Correction Matrix (CCM) and Color Space Converter aka CSC or + RGB2YUV, +- a Dynamic Range Compression (DRC) block, +- the Denoising pipeline (composed by multiple blocks for Noise Reduction, Edge + Enhancement, Gamma Compensation, etc), +- a Packetizer used for UV sub-sampling or RGB packing. + +All these blocks are controlled by SW through registers. Some of these +registers are accessible by the uAPI, so that usespace application and Image +Processing Algorithms (IPA) can configure them through the parameters buffers. + +Neoisp driver +============= + +Neoisp driver is located under drivers/media/platform/nxp/neoisp. +It uses the `V4L2 API` and the `V4L2 subdev API` to register capture and output +video devices in addition to a subdevice for neoisp that connects the video +devices in a single media graph realized using the `Media Controller (MC) API`. + +The media topology registered by Neoisp driver is represented below: + +.. kernel-figure:: nxp-neoisp.dot + :alt: Diagram of neoisp media device topology + :align: center + + +The media graph registers the following video device nodes: + +- neoisp-input0: output device for RAW frames to be submitted to the ISP for processing. +- neoisp-input1: output device for RAW frames short capture in HDR merge mode. +- neoisp-params: output meta device for parameters provided by user space 3A algorithms. +- neoisp-frame: capture device for RGB/YUV pixels of the processed images. +- neoisp-ir: capture device for the infra-red pixels of the processed images. +- neoisp-stats: capture meta device for generated image statistics for user space 3A algorithms. + +neoisp-input0, neoisp-input1 +---------------------------- + +Images to be processed by Neoisp are queued to the neoisp-input0 (and +neoisp-input1 when in HDR mode) output device nodes. Supported image formats +as input to the ISP are: + +- Raw bayer formats: + + - 8 bits raw (V4L2_PIX_FMT_SRGGB8, V4L2_PIX_FMT_SBGGR8, V4L2_PIX_FMT_SGBRG8, + V4L2_PIX_FMT_SGRBG8) + - 10 bits raw (V4L2_PIX_FMT_SRGGB10, V4L2_PIX_FMT_SBGGR10, + V4L2_PIX_FMT_SGBRG10, V4L2_PIX_FMT_SGRBG10) + - 12 bits raw (V4L2_PIX_FMT_SRGGB12, V4L2_PIX_FMT_SBGGR12, + V4L2_PIX_FMT_SGBRG12, V4L2_PIX_FMT_SGRBG12) + - 14 bits raw (V4L2_PIX_FMT_SRGGB14, V4L2_PIX_FMT_SBGGR14, + V4L2_PIX_FMT_SGBRG14, V4L2_PIX_FMT_SGRBG14) + - 16 bits raw (V4L2_PIX_FMT_SRGGB16, V4L2_PIX_FMT_SBGGR16, + V4L2_PIX_FMT_SGBRG16, V4L2_PIX_FMT_SGRBG16) + +- Monochrome formats: + + - 8 bits Monochrome (V4L2_PIX_FMT_GREY) + - 10 bits Monochrome (V4L2_PIX_FMT_Y10) + - 12 bits Monochrome (V4L2_PIX_FMT_Y12) + - 14 bits Monochrome (V4L2_PIX_FMT_Y14) + - 16 bits Monochrome (V4L2_PIX_FMT_Y16) + +.. note:: + RGBIr camera sensors are supported as well, and can be used through user + space activation of the IR block. + +.. note:: + neoisp-input1 link is mutable and should be enabled in case a short capture + image buffer is provided to the ISP for HDR merge. + +.. _neoisp_params: + +neoisp-params +------------- + +The neoisp-params output meta device receives configuration data to be written +to Neoisp registers and internal memory for desired input image processing. +This v4l2 device accepts the `legacy parameters` format, or the `extensible +parameters` format. + +When using the `legacy parameters` format, the parameters buffer is defined by +structure :c:type:`neoisp_meta_params_s`, and userspace should set +:ref:`V4L2_META_FMT_NEO_ISP_PARAMS <v4l2-meta-fmt-neo-isp-params>` as dataformat. + +When using the `extensible parameters` format, the parameters buffer is defined +by structure :c:type:`neoisp_ext_params_s`, and userspace should set +:ref:`V4L2_META_FMT_NEO_ISP_EXT_PARAMS <v4l2-meta-fmt-neo-isp-ext-params>` as +dataformat. + +When the related media link is disabled, the image decoding will be done based +on the default parameters of the ISP. + +neoisp-frame +------------ + +The capture device writes to memory the RGB or YUV pixels of the image processed +by Neoisp when the media link is enabled. If the related media link is disabled, +the processed image will be written to dummy buffer and not delivered to the +neoisp-frame video device node. + +neoisp-ir +--------- + +The capture device writes to memory the RGBIr pixels of the image processed by +Neoisp when the media link is enabled. If the related media link is disabled, +the processed image will not be delivered to the neoisp-ir video device node. + +.. _neoisp_stats: + +neoisp-stats +------------ + +The neoisp-stats capture meta device provides statistics data generated by +Neoisp hardware while processing the input image. +This v4l2 device accepts the `legacy statistics` format, or the `extensible +statistics` format. + +When using the `legacy statistics` format, the statistics buffer is defined by +structure :c:type:`neoisp_meta_stats_s`, and userspace should set +:ref:`V4L2_META_FMT_NEO_ISP_STATS <v4l2-meta-fmt-neo-isp-stats>` as dataformat. + +When using the `extensible statistics` format, the statistics buffer is defined +by structure :c:type:`neoisp_ext_stats_s`, and userspace should set +:ref:`V4L2_META_FMT_NEO_ISP_EXT_STATS <v4l2-meta-fmt-neo-isp-ext-stats>` as +dataformat. + +When the related media link is disabled, the decoding statistics will not be +delivered to the neoisp-stats meta device node. + +Control +======= + +To support additional neoisp hardware revisions, the read-only bitmask control +`V4L2_CID_NEOISP_SUPPORTED_PARAMS_BLOCKS` can be used to query the list of +supported blocks. Each bit represents the availability of the corresponding +entry from the :c:type:`neoisp_param_block_type_e` enum. In current driver +version, default and max values represent the blocks supported by the i.MX95 +SoC.
diff --git a/Documentation/admin-guide/media/v4l-drivers.rst b/Documentation/admin-guide/media/v4l-drivers.rst
index 393f83e8dc4d..5b6c71ae369d 100644
--- a/Documentation/admin-guide/media/v4l-drivers.rst
+++ b/Documentation/admin-guide/media/v4l-drivers.rst@@ -21,6 +21,7 @@ Video4Linux (V4L) driver-specific documentation ivtv mali-c55 mgb4 + nxp-neoisp omap3isp philips qcom_camss
diff --git a/Documentation/userspace-api/media/v4l/meta-formats.rst b/Documentation/userspace-api/media/v4l/meta-formats.rst
index 3e0cab153f0a..46268d955d3a 100644
--- a/Documentation/userspace-api/media/v4l/meta-formats.rst
+++ b/Documentation/userspace-api/media/v4l/meta-formats.rst@@ -18,6 +18,7 @@ These formats are used for the :ref:`metadata` interface only. metafmt-d4xx metafmt-generic metafmt-intel-ipu3 + metafmt-nxp-neoisp metafmt-pisp-be metafmt-pisp-fe metafmt-rkisp1
diff --git a/Documentation/userspace-api/media/v4l/metafmt-nxp-neoisp.rst b/Documentation/userspace-api/media/v4l/metafmt-nxp-neoisp.rst
new file mode 100644
index 000000000000..bcde8edcb441
--- /dev/null
+++ b/Documentation/userspace-api/media/v4l/metafmt-nxp-neoisp.rst@@ -0,0 +1,114 @@ +.. SPDX-License-Identifier: GPL-2.0 + +******************************************************************************** +V4L2_META_FMT_NEO_ISP_PARAMS ('nnip'), V4L2_META_FMT_NEO_ISP_EXT_PARAMS ('nnep') +******************************************************************************** + +The Neoisp image signal processor is configured by userspace through a buffer +of parameters to :ref:`neoisp-params <neoisp_params>` output device node using +the :c:type:`v4l2_meta_format` interface. + +There are two methods that allow to configure the Neoisp: the `legacy +parameters` configuration format and the `extensible parameters` configuration +format. + +.. _v4l2-meta-fmt-neo-isp-params: + +Legacy parameters configuration format +====================================== + +When using the `legacy parameters` configuration format, parameters are passed +to the :ref:`neoisp-params <neoisp_params>` metadata output video node using +the `V4L2_META_FMT_NEO_ISP_PARAMS` meta format. + +The buffer contains a single instance of the C structure +:c:type:`neoisp_meta_params_s` defined in ``nxp_neoisp.h``. So the structure +can be obtained from the buffer by: + +.. code-block:: c + + struct neoisp_meta_params_s *params = (struct neoisp_meta_params_s *) buffer; + +This method supports the Neoisp features currently available, it won't be +maintained for future Neoisp versions. New applications should use the +`extensible parameters` method then. + +.. _v4l2-meta-fmt-neo-isp-ext-params: + +Extensible parameters configuration format +========================================== + +When using the `extensible parameters` configuration format, parameters are +passed to the :ref:`neoisp-params <neoisp_params>` metadata output video node +using the `V4L2_META_FMT_NEO_ISP_EXT_PARAMS` meta format. + +The buffer contains a single instance of the C structure +:c:type:`v4l2_isp_params_buffer` defined in ``v4l2-isp.h``. The +:c:type:`v4l2_isp_params_buffer` structure is designed to allow userspace to +populate the data buffer with only the configuration data for the Neoisp blocks +it intends to configure. The extensible parameters format design allows +developers to define new block types to support new configuration parameters, +and defines a versioning scheme so that it can be extended and versioned +without breaking compatibility with existing applications. + +****************************************************************************** +V4L2_META_FMT_NEO_ISP_STATS ('nnis'), V4L2_META_FMT_NEO_ISP_EXT_STATS ('nnes') +****************************************************************************** + +The Neoisp image signal processor generates statistics data while processing an +input image. These statistics are captured in a buffer and provided to +userspace through the :ref:`neoisp-stats <neoisp_stats>` capture video node +using the :c:type:`v4l2_meta_format` interface. The statistics data are +processed by userspace application to produce the next Neoisp parameters. + +There are two methods to capture the statistics: the `legacy statistics` format +and the `extensible statistics` format. + +.. _v4l2-meta-fmt-neo-isp-stats: + +Legacy statistics format +======================== + +When using the `legacy statistics` format, Neoisp statistics are passed from +the :ref:`neoisp-stats <neoisp_stats>` metadata capture video node using the +`V4L2_META_FMT_NEO_ISP_STATS` meta format. + +The buffer contains a single instance of the C structure +:c:type:`neoisp_meta_stats_s` defined in ``nxp_neoisp.h``. So the structure can +be obtained from the buffer by: + +.. code-block:: c + + struct neoisp_meta_stats_s *params = (struct neoisp_meta_stats_s *) buffer; + +This method supports the Neoisp statistics currently available, it won't be +maintained for future Neoisp versions. New applications should use the +`extensible statistics` method then. + +.. _v4l2-meta-fmt-neo-isp-ext-stats: + +Extensible statistics format +============================ + +When using the `extensible statistics` format, the statistics buffer is passed +from the :ref:`neoisp-stats <neoisp_stats>` metadata capture video node using +the `V4L2_META_FMT_NEO_ISP_EXT_STATS` meta format. + +The buffer contains a single instance of the C structure +:c:type:`v4l2_isp_stats_buffer` defined in ``v4l2-isp.h``. The +:c:type:`v4l2_isp_stats_buffer` structure is designed to allow future Neoisp +driver versions to populate the statistics buffer with future blocks +statistics, and defines a versioning scheme so that it can be extended and +versioned without breaking compatibility with existing applications. + +********************** +Neoisp uAPI data types +********************** + +This chapter describes the data types exposed to userspace by Neoisp driver. + +Some structure members are in a fixed-point format, in this case the related +description will be ended by a fixed-point definition between parenthesis. + +.. kernel-doc:: include/uapi/linux/media/nxp/nxp_neoisp.h +
--
2.52.0