[FFmpeg-devel] [EXT] [PATCH v5 7/7] avcodec/v4l2_m2m_dec: setup capture queue before enqueue the first frame

[Andriy Gelman]

Hi Ming,

Sorry for the late reply.

On Tue, 29. Mar 06:51, Ming Qian wrote:
> > From: Andriy Gelman [mailto:andriy.gelman at gmail.com]
> > Sent: Saturday, March 26, 2022 11:43 PM
> > To: FFmpeg development discussions and patches <ffmpeg-devel at ffmpeg.org>
> > Cc: Ming Qian <ming.qian at nxp.com>
> > Subject: Re: [FFmpeg-devel] [EXT] [PATCH v5 7/7] avcodec/v4l2_m2m_dec:
> > setup capture queue before enqueue the first frame
> > 
> > Caution: EXT Email
> > 
> > Hi Ming,
> > 
> > On Mon, 21. Mar 07:27, Ming Qian wrote:
> > > Hi Andriy,
> > >
> > >     What do you think of this patch?
> > >
> > >     The Initialization flow defined in
> > linux/Documentation/userspace-api/media/v4l/dev-decoder.rst
> > > 1. Set the coded format on OUTPUT via VIDIOC_S_FMT().
> > > 2. Allocate source (bytestream) buffers via VIDIOC_REQBUFS() on OUTPUT.
> > > 3. Start streaming on the OUTPUT queue via VIDIOC_STREAMON().
> > > 4. This step only applies to coded formats that contain resolution
> > information in the stream. Continue queuing/dequeuing bytestream buffers
> > to/from the OUTPUT queue via VIDIOC_QBUF() and VIDIOC_DQBUF(). The
> > buffers will be processed and returned to the client in order, until required
> > metadata to configure the CAPTURE queue are found. This is indicated by the
> > decoder sending a V4L2_EVENT_SOURCE_CHANGE event with changes set to
> > V4L2_EVENT_SRC_CH_RESOLUTION.
> > >       Note: A client capable of acquiring stream parameters from the
> > bytestream on its own may attempt to set the width and height of the
> > OUTPUT format to non-zero values matching the coded size of the stream, skip
> > this step and continue with the Capture Setup sequence.
> > >
> > > 5. Continue with the Capture Setup sequence.
> > >
> > >     In ffmpeg's implementation, ffmpeg will set non-zero width and height
> > on output queue, so the step 4 should be skipped, and setup the capture
> > queue directly.
> > >     So the flow should be:
> > > 1. Set the coded format and valid resolution on OUTPUT via VIDIOC_S_FMT().
> > > 2. Allocate source (bytestream) buffers via VIDIOC_REQBUFS() on OUTPUT.
> > > 3. Start streaming on the OUTPUT queue via VIDIOC_STREAMON().
> > > 5. Continue with the Capture Setup sequence.
> > >
> > > And this patch is just following the above flow.
> > >
> > > Ming
> > 
> > The v4 version didn't work for me on odroid xu4
> > https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fffmpeg.o
> > rg%2Fpipermail%2Fffmpeg-devel%2F2022-January%2F290679.html&dat
> > a=04%7C01%7Cming.qian%40nxp.com%7Cca988d592253473cbac408da0f3f4
> > 801%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637839061704
> > 093567%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2l
> > uMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=I80bpE70S6JI
> > 8A%2BaXQGq4JyxL0eBbtLN58p5SpqlA6s%3D&reserved=0
> > 
> > I haven't had time retest but don't think anything has changed.
> > 
> > --
> > Andriy
> 
> Hi Andriy,
>     I check the S5P MFC Video Codec driver, and indeed it requires the client enqueue the output buffer which contains the sequence header. And in vidioc_g_fmt(), driver will block and wait until the header is parsed.
>     And indeed my patch will not work on it.
>     But it seems don't meet the specification.
>     For S5P MFC, the step 4 can't be skipped, and the client should wait the V4L2_EVENT_SOURCE_CHANGE event before the step 5 setting up capture queue. 
>     The ffmpeg v4l2 decoder don't skip step 4, but also doesn't wait the V4L2_EVENT_SOURCE_CHANGE event.

> 
> So in current, we should make it work on more devices instead of more spec?

Personally I'd prefer not to break users. Odroid xu4 is a popular board afaik.
But I'm also not the maintainer of v4l2m2m.

IMO it would be better the fix the problem in the driver first.

> 
>     The Initialization flow defined in linux/Documentation/userspace-api/media/v4l/dev-decoder.rst
> 	1. Set the coded format on OUTPUT via VIDIOC_S_FMT().
> 	2. Allocate source (bytestream) buffers via VIDIOC_REQBUFS() on OUTPUT.
> 	3. Start streaming on the OUTPUT queue via VIDIOC_STREAMON().
> 	4. This step only applies to coded formats that contain resolution information in the stream. Continue queuing/dequeuing bytestream buffers to/from the OUTPUT queue via VIDIOC_QBUF() and VIDIOC_DQBUF(). The buffers will be processed and returned to the client in order, until required metadata to configure the CAPTURE queue are found. This is indicated by the decoder sending a V4L2_EVENT_SOURCE_CHANGE event with changes set to V4L2_EVENT_SRC_CH_RESOLUTION.
> 	Note: A client capable of acquiring stream parameters from the bytestream on its own may attempt to set the width and height of the OUTPUT format to non-zero values matching the coded size of the stream, skip this step and continue with the Capture Setup sequence.
> 	5. Continue with the Capture Setup sequence.


-- 
Andriy