[FFmpeg-devel] [PATCH] avcodec/v4l2_m2m_dec: dequeue frame if input isn't ready

[Andriy Gelman]

On Mon, 27. Dec 03:18, Cameron Gutman wrote:
> On 12/27/21 00:17, Andriy Gelman wrote:
> > On Tue, 14. Dec 02:12, Cameron Gutman wrote:
> >> The V4L2M2M API operates asynchronously, so multiple packets can
> >> be enqueued before getting a batch of frames back. Since it was
> >> only possible to receive a frame by submitting another packet,
> >> there wasn't a way to drain those excess output frames from when
> >> avcodec_receive_frame() returned AVERROR(EAGAIN).
> >>
> > 
> >> In my testing, this change reduced decode latency of a real-time
> >> 60 FPS H.264 stream by approximately 10x (200ms -> 20ms) on a
> >> Raspberry Pi 4.
> > 
> > I was doing some more tests today, but didn't have any luck dequeuing a frame
> > if ff_decode_get_packet() returned EAGAIN.
> 
> Hmm, maybe there is something different about your test harness?
> I'm receiving 720p 60 FPS H.264 ES in real-time from the network.

I used ffmpeg, i.e.
./ffmpeg -codec:v h264_v4l2m2m -i 720p60.h264 -f null -

Anyway, I applied your patch but just removed the comment on latency
because I couldn't reproduce it.

> 
> For each H.264 encoded frame I receive off the network, my basic
> approach is like this (simplified for brevity):
> 
> avcodec_send_packet(&pkt);
> do {
>     frame = av_frame_alloc();
>     if ((err = avcodec_receive_frame(frame)) == 0) {
>         render_frame(frame);
>     }
> } while (err == 0);
> 
> I'll usually get EAGAIN immediately for the first few frames I submit
> (so no output frame yet), but then I'll get a batch of output frames
> back after the first completed decode. That drains the excess latency
> from the pipeline to avoid always being behind.
> 
> For cases where we want to prioritize latency over throughput, I've
> had success with this approach too:
> 
> avcodec_send_packet(&pkt);
> while (avcodec_receive_frame(frame) == AVERROR(EAGAIN)) {
>     msleep(1);
> }
> render_frame(frame);
> 
> In this case, we can retry avcodec_receive_frame() until we get the
> frame back that we just submitted for decoding.

> 
> The patch here enables both of these use-cases by allowing V4L2M2M
> to retry getting a decoded frame without new input data. Both of
> these also work with MMAL after the recent decoupled dataflow patch.
> 
> > Could you share the dataset?
> > 
> 

> It is 720p60.h264 from here:
> https://onedrive.live.com/?authkey=%21ALoKfcPfFeKyhzs&id=C15BF9770619F56%21165617&cid=0C15BF9770619F56

I could not decode this dataset on rpi4 (before or after the patch). Tried to upgrade kernel
to 5.16, but still same problem. Odroid xu4 seems to work fine.

I found a repo in github https://github.com/jc-kynesim/rpi-ffmpeg (I guess Rpi4
team), where the decoding is working. Doing a bisect points to this commit that fixes:
https://github.com/jc-kynesim/rpi-ffmpeg/commit/3e60c08552110d9da1b3b5d3637c6bd8b53eec69

There are lots of changes in the commit, so I'm not sure what exactly fixed the
problem or if it was by chance. It would be good to figure out.

-- 
Andriy