[FFmpeg-devel] [EXT] [PATCH v5 7/7] avcodec/v4l2_m2m_dec: setup capture queue before enqueue the first frame

[Ming Qian]

Hi Andriy,

    What do you think of this patch?

    The Initialization flow defined in linux/Documentation/userspace-api/media/v4l/dev-decoder.rst
1. Set the coded format on OUTPUT via VIDIOC_S_FMT().
2. Allocate source (bytestream) buffers via VIDIOC_REQBUFS() on OUTPUT.
3. Start streaming on the OUTPUT queue via VIDIOC_STREAMON().
4. This step only applies to coded formats that contain resolution information in the stream. Continue queuing/dequeuing bytestream buffers to/from the OUTPUT queue via VIDIOC_QBUF() and VIDIOC_DQBUF(). The buffers will be processed and returned to the client in order, until required metadata to configure the CAPTURE queue are found. This is indicated by the decoder sending a V4L2_EVENT_SOURCE_CHANGE event with changes set to V4L2_EVENT_SRC_CH_RESOLUTION.
	Note: A client capable of acquiring stream parameters from the bytestream on its own may attempt to set the width and height of the OUTPUT format to non-zero values matching the coded size of the stream, skip this step and continue with the Capture Setup sequence.

5. Continue with the Capture Setup sequence.

    In ffmpeg's implementation, ffmpeg will set non-zero width and height on output queue, so the step 4 should be skipped, and setup the capture queue directly.
    So the flow should be:
1. Set the coded format and valid resolution on OUTPUT via VIDIOC_S_FMT().
2. Allocate source (bytestream) buffers via VIDIOC_REQBUFS() on OUTPUT.
3. Start streaming on the OUTPUT queue via VIDIOC_STREAMON().
5. Continue with the Capture Setup sequence.

And this patch is just following the above flow.

Ming


> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces at ffmpeg.org] On Behalf Of
> Ming Qian
> Sent: Tuesday, January 4, 2022 5:09 PM
> To: ffmpeg-devel at ffmpeg.org
> Subject: [EXT] [FFmpeg-devel] [PATCH v5 7/7] avcodec/v4l2_m2m_dec: setup
> capture queue before enqueue the first frame
> 
> Caution: EXT Email
> 
> there are two proper ways to setup capture queue.
> 1. client wait the source change event,
>    then setup the capture queue and streamon 2. client setup the capture
> queue in advance,
>    but to avoid time issues, client should start
>    the capture queue before it enqueue the sequence
>    header to decoder driver through output queue.
>    and the sequence header is always in the first
>    frame, so client should start capture before
>    enqueue the first frame.
> 
> ffmpeg use the method 2 to setup capture queue, but currently ffmpeg
> enqueue the first frame before starting the capture queue.
> so in driver side, there are time issues.
> when driver has parsed the resolution from sequence header, but the client
> may not finished setup the capture.
> so driver can't decide whether to notify a source change event to client. and
> the following flow may be chaotic.
> 
> And it's OK that client setup capture queue first, then enqueue the first frame.
> 
> Signed-off-by: Ming Qian <ming.qian at nxp.com>
> ---
>  libavcodec/v4l2_m2m_dec.c | 18 ++++++++----------
>  1 file changed, 8 insertions(+), 10 deletions(-)
> 
> diff --git a/libavcodec/v4l2_m2m_dec.c b/libavcodec/v4l2_m2m_dec.c index
> b0c3d30ac8ae..e67758531ace 100644
> --- a/libavcodec/v4l2_m2m_dec.c
> +++ b/libavcodec/v4l2_m2m_dec.c
> @@ -153,6 +153,14 @@ static int v4l2_receive_frame(AVCodecContext *avctx,
> AVFrame *frame)
>      if (s->draining)
>          goto dequeue;
> 
> +    ret = v4l2_try_start(avctx);
> +    if (ret) {
> +        /* can't recover */
> +        if (ret != AVERROR(ENOMEM))
> +            ret = 0;
> +        goto fail;
> +    }
> +
>      ret = ff_v4l2_context_enqueue_packet(output, &s->buf_pkt);
>      if (ret < 0 && ret != AVERROR(EAGAIN))
>          goto fail;
> @@ -161,16 +169,6 @@ static int v4l2_receive_frame(AVCodecContext *avctx,
> AVFrame *frame)
>      if (ret != AVERROR(EAGAIN))
>          av_packet_unref(&s->buf_pkt);
> 
> -    if (!s->draining) {
> -        ret = v4l2_try_start(avctx);
> -        if (ret) {
> -            /* cant recover */
> -            if (ret != AVERROR(ENOMEM))
> -                ret = 0;
> -            goto fail;
> -        }
> -    }
> -
>  dequeue:
>      return ff_v4l2_context_dequeue_frame(capture, frame, -1);
>  fail:
> --
> 2.33.0
> 
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fffmpeg.
> org%2Fmailman%2Flistinfo%2Fffmpeg-devel&data=04%7C01%7Cming.qi
> an%40nxp.com%7C6b9621a0804c45f9da9108d9cf61fb87%7C686ea1d3bc2b
> 4c6fa92cd99c5c301635%7C0%7C1%7C637768841997715387%7CUnknown%
> 7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwi
> LCJXVCI6Mn0%3D%7C3000&sdata=E7V4kwlJtxbh6jnisMV8FWAwwDzksS
> qLp82kapSW4FA%3D&reserved=0
> 
> To unsubscribe, visit link above, or email ffmpeg-devel-request at ffmpeg.org
> with subject "unsubscribe".