[PATCH v2 00/10] media: Introduce video device state management
From: Jai Luthra <hidden>
Date: 2025-09-19 09:57:06
Also in:
imx, linux-arm-msm, linux-aspeed, linux-media, linux-mediatek, linux-renesas-soc, linux-rockchip, linux-samsung-soc, linux-staging, linux-sunxi, linux-tegra, linux-usb, lkml, openbmc
Hi, This patch series introduces state management for video devices. Currently, V4L2 subdevices have a well-established state management system [1] that allows the framework to store the subdevice state at a central location. Video devices however lack this, leading to inconsistent state handling across drivers and making it difficult to implement features like hardware multiplexing in complex media pipelines [2]. The series is composed of three parts: - Core Infrastructure (PATCH 1-4) Introduces the basic state structure, try state support, initialization callbacks, and driver helpers - Framework Integration (PATCH 5-7) Integrates state passing through the ioctl wrappers and driver implementations - Driver Examples (PATCH 8-10) Use the state to store formats in TI J721E CSI2RX and Rockchip RKISP1 drivers This should also provide a foundation for drivers to extend or subclass the state structure with device-specific variables in future. I have tested capture and format negotiation with the converted drivers on SK AM62A (j721e-csi2rx) and Debix Model A (rkisp1). [1]: https://lore.kernel.org/linux-media/20210610145606.3468235-1-tomi.valkeinen@ideasonboard.com/ (local) [2]: https://lore.kernel.org/linux-media/20250724-multicontext-mainline-2025-v2-0-c9b316773486@ideasonboard.com/ (local) Signed-off-by: Jai Luthra <redacted> --- Changes in v2: - Drop RFC tag - Rebase on top of latest media tree, that includes multiple changes by Jacopo and Laurent, which removes all usage of passing v4l2_fh through the opaque pointer in ioctl implementations, replacing it with a NULL pointer - Allocate the state dynamically, and free it when done with the device or the filehandle - Introduce an init_state callback so that drivers can initialize the state after allocation - Pass the video device state (as an explicitly typed pointer) to all ioctl implemenation across all drivers - Split the single patch introducing video device state into separate patches each making a single logical change before the state is actually passed to the drivers in PATCH 7 - In drivers, use a helper to access the current format from the state - Move the queue busy check back to the drivers, as some codecs and VBI devices may support changing formats, so the framework shouldn't return -EBUSY - Drop pixel format info and configuration storage from rkisp1 internal strucutre, and calculate them on the fly - Fix other review comments from Sakari, Laurent and Jacopo, except those around dropping the enum and using try_fmt hook to update only the state, and s_fmt hook to update hardware registers - Link to v1: https://lore.kernel.org/r/20250703-vdev-state-v1-0-d647a5e4986d@ideasonboard.com (local) --- Jai Luthra (10): media: v4l2-core: Introduce state management for video devices media: v4l2-dev: Add support for try state media: v4l2-dev: Add callback for initializing video device state media: v4l2-dev: Add helpers to get current format from the state media: v4l2-ioctl: Add video_device_state argument to v4l2 ioctl wrappers media: Replace void * with video_device_state * in all driver ioctl implementations media: v4l2-ioctl: Pass device state for G/S/TRY_FMT ioctls media: ti: j721e-csi2rx: Use video_device_state media: rkisp1: Use video_device_state media: rkisp1: Calculate format information on demand drivers/input/rmi4/rmi_f54.c | 22 +- drivers/input/touchscreen/atmel_mxt_ts.c | 23 +- drivers/input/touchscreen/sur40.c | 35 +- .../extron-da-hd-4k-plus/extron-da-hd-4k-plus.c | 34 +- drivers/media/common/saa7146/saa7146_video.c | 37 +- drivers/media/common/videobuf2/videobuf2-v4l2.c | 30 +- drivers/media/dvb-frontends/rtl2832_sdr.c | 38 +- drivers/media/i2c/video-i2c.c | 39 ++- drivers/media/pci/bt8xx/bttv-driver.c | 84 +++-- drivers/media/pci/bt8xx/bttv-vbi.c | 11 +- drivers/media/pci/bt8xx/bttvp.h | 9 +- drivers/media/pci/cobalt/cobalt-v4l2.c | 98 +++--- drivers/media/pci/cx18/cx18-driver.c | 9 +- drivers/media/pci/cx18/cx18-ioctl.c | 114 +++--- drivers/media/pci/cx18/cx18-ioctl.h | 9 +- drivers/media/pci/cx23885/cx23885-417.c | 53 +-- drivers/media/pci/cx23885/cx23885-ioctl.c | 8 +- drivers/media/pci/cx23885/cx23885-ioctl.h | 9 +- drivers/media/pci/cx23885/cx23885-vbi.c | 5 +- drivers/media/pci/cx23885/cx23885-video.c | 82 +++-- drivers/media/pci/cx23885/cx23885.h | 4 +- drivers/media/pci/cx25821/cx25821-video.c | 57 ++- drivers/media/pci/cx88/cx88-blackbird.c | 47 ++- drivers/media/pci/cx88/cx88-vbi.c | 2 +- drivers/media/pci/cx88/cx88-video.c | 56 ++- drivers/media/pci/cx88/cx88.h | 2 +- drivers/media/pci/dt3155/dt3155.c | 25 +- drivers/media/pci/intel/ipu3/ipu3-cio2.c | 29 +- drivers/media/pci/intel/ipu6/ipu6-isys-video.c | 41 ++- drivers/media/pci/ivtv/ivtv-driver.c | 8 +- drivers/media/pci/ivtv/ivtv-ioctl.c | 225 ++++++++---- drivers/media/pci/ivtv/ivtv-ioctl.h | 6 +- drivers/media/pci/mgb4/mgb4_vin.c | 55 ++- drivers/media/pci/mgb4/mgb4_vout.c | 49 ++- drivers/media/pci/saa7134/saa7134-empress.c | 12 +- drivers/media/pci/saa7134/saa7134-video.c | 96 ++++-- drivers/media/pci/saa7134/saa7134.h | 38 +- drivers/media/pci/saa7146/hexium_gemini.c | 10 +- drivers/media/pci/saa7146/hexium_orion.c | 10 +- drivers/media/pci/saa7146/mxb.c | 46 ++- drivers/media/pci/saa7164/saa7164-encoder.c | 39 ++- drivers/media/pci/saa7164/saa7164-vbi.c | 27 +- drivers/media/pci/saa7164/saa7164.h | 9 +- drivers/media/pci/solo6x10/solo6x10-v4l2-enc.c | 46 ++- drivers/media/pci/solo6x10/solo6x10-v4l2.c | 32 +- drivers/media/pci/ttpci/budget-av.c | 10 +- drivers/media/pci/tw5864/tw5864-video.c | 49 ++- drivers/media/pci/tw68/tw68-video.c | 43 ++- drivers/media/pci/tw686x/tw686x-video.c | 53 ++- drivers/media/pci/zoran/zoran_driver.c | 44 ++- drivers/media/platform/allegro-dvt/allegro-core.c | 49 ++- .../media/platform/amlogic/c3/isp/c3-isp-capture.c | 18 +- .../media/platform/amlogic/c3/isp/c3-isp-params.c | 9 +- .../media/platform/amlogic/c3/isp/c3-isp-stats.c | 9 +- drivers/media/platform/amlogic/meson-ge2d/ge2d.c | 36 +- drivers/media/platform/amphion/vdec.c | 30 +- drivers/media/platform/amphion/venc.c | 49 ++- drivers/media/platform/aspeed/aspeed-video.c | 50 ++- drivers/media/platform/atmel/atmel-isi.c | 36 +- drivers/media/platform/broadcom/bcm2835-unicam.c | 33 +- .../media/platform/chips-media/coda/coda-common.c | 69 ++-- .../platform/chips-media/wave5/wave5-helper.c | 2 +- .../platform/chips-media/wave5/wave5-helper.h | 3 +- .../platform/chips-media/wave5/wave5-vpu-dec.c | 54 ++- .../platform/chips-media/wave5/wave5-vpu-enc.c | 62 +++- .../media/platform/imagination/e5010-jpeg-enc.c | 34 +- drivers/media/platform/intel/pxa_camera.c | 34 +- drivers/media/platform/m2m-deinterlace.c | 34 +- drivers/media/platform/marvell/mcam-core.c | 52 +-- .../media/platform/mediatek/jpeg/mtk_jpeg_core.c | 44 ++- drivers/media/platform/mediatek/mdp/mtk_mdp_m2m.c | 34 +- .../media/platform/mediatek/mdp3/mtk-mdp3-m2m.c | 25 +- .../mediatek/vcodec/decoder/mtk_vcodec_dec.c | 52 ++- .../mediatek/vcodec/encoder/mtk_vcodec_enc.c | 50 ++- .../media/platform/microchip/microchip-isc-base.c | 33 +- drivers/media/platform/nuvoton/npcm-video.c | 43 ++- drivers/media/platform/nvidia/tegra-vde/v4l2.c | 38 +- drivers/media/platform/nxp/dw100/dw100.c | 35 +- drivers/media/platform/nxp/imx-jpeg/mxc-jpeg.c | 52 ++- drivers/media/platform/nxp/imx-pxp.c | 34 +- drivers/media/platform/nxp/imx7-media-csi.c | 21 +- drivers/media/platform/nxp/imx8-isi/imx8-isi-m2m.c | 15 +- .../media/platform/nxp/imx8-isi/imx8-isi-video.c | 18 +- drivers/media/platform/nxp/mx2_emmaprp.c | 31 +- drivers/media/platform/qcom/camss/camss-video.c | 27 +- drivers/media/platform/qcom/iris/iris_vidc.c | 48 ++- drivers/media/platform/qcom/venus/vdec.c | 29 +- drivers/media/platform/qcom/venus/venc.c | 38 +- .../media/platform/raspberrypi/pisp_be/pisp_be.c | 42 ++- drivers/media/platform/raspberrypi/rp1-cfe/cfe.c | 40 ++- .../media/platform/renesas/rcar-vin/rcar-v4l2.c | 21 +- drivers/media/platform/renesas/rcar_drif.c | 30 +- drivers/media/platform/renesas/rcar_fdp1.c | 18 +- drivers/media/platform/renesas/rcar_jpu.c | 21 +- drivers/media/platform/renesas/renesas-ceu.c | 36 +- .../media/platform/renesas/rzg2l-cru/rzg2l-video.c | 18 +- drivers/media/platform/renesas/sh_vou.c | 41 ++- drivers/media/platform/renesas/vsp1/vsp1_histo.c | 9 +- drivers/media/platform/renesas/vsp1/vsp1_video.c | 18 +- drivers/media/platform/rockchip/rga/rga.c | 24 +- .../platform/rockchip/rkisp1/rkisp1-capture.c | 289 ++++++++-------- .../media/platform/rockchip/rkisp1/rkisp1-common.h | 10 - .../media/platform/rockchip/rkisp1/rkisp1-params.c | 15 +- .../media/platform/rockchip/rkisp1/rkisp1-stats.c | 9 +- drivers/media/platform/rockchip/rkvdec/rkvdec.c | 34 +- .../media/platform/samsung/exynos-gsc/gsc-m2m.c | 44 ++- .../platform/samsung/exynos4-is/fimc-capture.c | 45 ++- .../platform/samsung/exynos4-is/fimc-isp-video.c | 32 +- .../media/platform/samsung/exynos4-is/fimc-lite.c | 36 +- .../media/platform/samsung/exynos4-is/fimc-m2m.c | 21 +- .../platform/samsung/s3c-camif/camif-capture.c | 54 ++- drivers/media/platform/samsung/s5p-g2d/g2d.c | 26 +- .../media/platform/samsung/s5p-jpeg/jpeg-core.c | 34 +- .../media/platform/samsung/s5p-mfc/s5p_mfc_dec.c | 48 ++- .../media/platform/samsung/s5p-mfc/s5p_mfc_enc.c | 51 ++- drivers/media/platform/st/sti/bdisp/bdisp-v4l2.c | 26 +- drivers/media/platform/st/sti/delta/delta-v4l2.c | 44 ++- drivers/media/platform/st/sti/hva/hva-v4l2.c | 46 ++- drivers/media/platform/st/stm32/dma2d/dma2d.c | 18 +- drivers/media/platform/st/stm32/stm32-dcmi.c | 42 ++- .../st/stm32/stm32-dcmipp/dcmipp-bytecap.c | 20 +- .../media/platform/sunxi/sun4i-csi/sun4i_v4l2.c | 25 +- .../platform/sunxi/sun6i-csi/sun6i_csi_capture.c | 24 +- drivers/media/platform/sunxi/sun8i-di/sun8i-di.c | 31 +- .../platform/sunxi/sun8i-rotate/sun8i_rotate.c | 34 +- .../media/platform/synopsys/hdmirx/snps_hdmirx.c | 47 ++- drivers/media/platform/ti/am437x/am437x-vpfe.c | 52 ++- drivers/media/platform/ti/cal/cal-video.c | 54 ++- drivers/media/platform/ti/davinci/vpif_capture.c | 94 ++--- drivers/media/platform/ti/davinci/vpif_display.c | 58 ++-- .../media/platform/ti/j721e-csi2rx/j721e-csi2rx.c | 127 +++---- drivers/media/platform/ti/omap/omap_vout.c | 49 ++- drivers/media/platform/ti/omap3isp/ispvideo.c | 53 ++- drivers/media/platform/ti/vpe/vpe.c | 25 +- drivers/media/platform/verisilicon/hantro_v4l2.c | 47 ++- drivers/media/platform/via/via-camera.c | 42 ++- drivers/media/platform/xilinx/xilinx-dma.c | 15 +- drivers/media/radio/dsbr100.c | 15 +- drivers/media/radio/radio-cadet.c | 18 +- drivers/media/radio/radio-isa.c | 18 +- drivers/media/radio/radio-keene.c | 15 +- drivers/media/radio/radio-ma901.c | 15 +- drivers/media/radio/radio-miropcm20.c | 15 +- drivers/media/radio/radio-mr800.c | 18 +- drivers/media/radio/radio-raremono.c | 19 +- drivers/media/radio/radio-sf16fmi.c | 15 +- drivers/media/radio/radio-si476x.c | 28 +- drivers/media/radio/radio-tea5764.c | 15 +- drivers/media/radio/radio-tea5777.c | 21 +- drivers/media/radio/radio-timb.c | 26 +- drivers/media/radio/si470x/radio-si470x-common.c | 25 +- drivers/media/radio/si4713/radio-platform-si4713.c | 18 +- drivers/media/radio/si4713/radio-usb-si4713.c | 15 +- drivers/media/radio/tea575x.c | 21 +- drivers/media/test-drivers/vicodec/vicodec-core.c | 54 +-- drivers/media/test-drivers/vim2m.c | 56 +-- drivers/media/test-drivers/vimc/vimc-capture.c | 20 +- drivers/media/test-drivers/visl/visl-video.c | 34 +- drivers/media/test-drivers/vivid/vivid-core.c | 203 ++++++----- drivers/media/test-drivers/vivid/vivid-meta-cap.c | 6 +- drivers/media/test-drivers/vivid/vivid-meta-cap.h | 6 +- drivers/media/test-drivers/vivid/vivid-meta-out.c | 6 +- drivers/media/test-drivers/vivid/vivid-meta-out.h | 9 +- drivers/media/test-drivers/vivid/vivid-radio-rx.c | 16 +- drivers/media/test-drivers/vivid/vivid-radio-rx.h | 16 +- drivers/media/test-drivers/vivid/vivid-radio-tx.c | 6 +- drivers/media/test-drivers/vivid/vivid-radio-tx.h | 6 +- drivers/media/test-drivers/vivid/vivid-sdr-cap.c | 30 +- drivers/media/test-drivers/vivid/vivid-sdr-cap.h | 30 +- drivers/media/test-drivers/vivid/vivid-touch-cap.c | 22 +- drivers/media/test-drivers/vivid/vivid-touch-cap.h | 22 +- drivers/media/test-drivers/vivid/vivid-vbi-cap.c | 26 +- drivers/media/test-drivers/vivid/vivid-vbi-cap.h | 22 +- drivers/media/test-drivers/vivid/vivid-vbi-out.c | 22 +- drivers/media/test-drivers/vivid/vivid-vbi-out.h | 18 +- drivers/media/test-drivers/vivid/vivid-vid-cap.c | 114 +++--- drivers/media/test-drivers/vivid/vivid-vid-cap.h | 117 +++++-- .../media/test-drivers/vivid/vivid-vid-common.c | 24 +- .../media/test-drivers/vivid/vivid-vid-common.h | 34 +- drivers/media/test-drivers/vivid/vivid-vid-out.c | 122 ++++--- drivers/media/test-drivers/vivid/vivid-vid-out.h | 95 +++-- drivers/media/usb/airspy/airspy.c | 37 +- drivers/media/usb/au0828/au0828-video.c | 74 ++-- drivers/media/usb/cx231xx/cx231xx-417.c | 28 +- drivers/media/usb/cx231xx/cx231xx-video.c | 79 +++-- drivers/media/usb/cx231xx/cx231xx.h | 39 ++- drivers/media/usb/dvb-usb/cxusb-analog.c | 48 ++- drivers/media/usb/em28xx/em28xx-video.c | 87 +++-- drivers/media/usb/go7007/go7007-v4l2.c | 75 ++-- drivers/media/usb/gspca/gspca.c | 51 ++- drivers/media/usb/hackrf/hackrf.c | 39 ++- drivers/media/usb/hdpvr/hdpvr-video.c | 60 ++-- drivers/media/usb/msi2500/msi2500.c | 31 +- drivers/media/usb/pvrusb2/pvrusb2-v4l2.c | 98 ++++-- drivers/media/usb/pwc/pwc-v4l.c | 44 ++- drivers/media/usb/s2255/s2255drv.c | 55 ++- drivers/media/usb/stk1160/stk1160-v4l.c | 40 ++- drivers/media/usb/usbtv/usbtv-video.c | 24 +- drivers/media/usb/uvc/uvc_metadata.c | 17 +- drivers/media/usb/uvc/uvc_v4l2.c | 59 +++- drivers/media/v4l2-core/v4l2-ctrls-api.c | 2 +- drivers/media/v4l2-core/v4l2-dev.c | 66 ++++ drivers/media/v4l2-core/v4l2-fh.c | 6 + drivers/media/v4l2-core/v4l2-ioctl.c | 381 +++++++++++---------- drivers/media/v4l2-core/v4l2-mem2mem.c | 56 +-- drivers/staging/media/atomisp/pci/atomisp_ioctl.c | 72 ++-- drivers/staging/media/av7110/av7110_v4l.c | 60 +++- .../media/deprecated/atmel/atmel-isc-base.c | 33 +- drivers/staging/media/imx/imx-media-capture.c | 53 ++- drivers/staging/media/imx/imx-media-csc-scaler.c | 23 +- drivers/staging/media/ipu3/ipu3-v4l2.c | 29 +- drivers/staging/media/ipu7/ipu7-isys-video.c | 30 +- drivers/staging/media/meson/vdec/vdec.c | 26 +- drivers/staging/media/starfive/camss/stf-video.c | 18 +- drivers/staging/media/sunxi/cedrus/cedrus_video.c | 29 +- .../media/sunxi/sun6i-isp/sun6i_isp_capture.c | 24 +- .../media/sunxi/sun6i-isp/sun6i_isp_params.c | 9 +- drivers/staging/media/tegra-video/vi.c | 68 ++-- drivers/staging/most/video/video.c | 39 ++- .../vc04_services/bcm2835-camera/bcm2835-camera.c | 58 ++-- drivers/usb/gadget/function/uvc_v4l2.c | 50 ++- include/media/v4l2-ctrls.h | 5 +- include/media/v4l2-dev.h | 84 +++++ include/media/v4l2-fh.h | 2 + include/media/v4l2-ioctl.h | 238 ++++++------- include/media/v4l2-mem2mem.h | 48 ++- include/media/videobuf2-v4l2.h | 33 +- 227 files changed, 5946 insertions(+), 3322 deletions(-) --- base-commit: 680fa38ea3a19131871a90bac05a610e6247d928 change-id: 20250703-vdev-state-0743baa0ad4b Best regards, -- Jai Luthra [off-list ref]