[FFmpeg-devel] [PATCH] avcodec/v4l2_m2m_dec: dequeue frame if input isn't ready

[Andriy Gelman]

On Tue, 28. Dec 18:41, Andriy Gelman wrote:
> On Mon, 27. Dec 03:18, Cameron Gutman wrote:
> > On 12/27/21 00:17, Andriy Gelman wrote:
> > > On Tue, 14. Dec 02:12, Cameron Gutman wrote:
> > >> The V4L2M2M API operates asynchronously, so multiple packets can
> > >> be enqueued before getting a batch of frames back. Since it was
> > >> only possible to receive a frame by submitting another packet,
> > >> there wasn't a way to drain those excess output frames from when
> > >> avcodec_receive_frame() returned AVERROR(EAGAIN).
> > >>
> > > 
> > >> In my testing, this change reduced decode latency of a real-time
> > >> 60 FPS H.264 stream by approximately 10x (200ms -> 20ms) on a
> > >> Raspberry Pi 4.
> > > 
> > > I was doing some more tests today, but didn't have any luck dequeuing a frame
> > > if ff_decode_get_packet() returned EAGAIN.
> > 
> > Hmm, maybe there is something different about your test harness?
> > I'm receiving 720p 60 FPS H.264 ES in real-time from the network.
> 
> I used ffmpeg, i.e.
> ./ffmpeg -codec:v h264_v4l2m2m -i 720p60.h264 -f null -
> 
> Anyway, I applied your patch but just removed the comment on latency
> because I couldn't reproduce it.
> 
> > 
> > For each H.264 encoded frame I receive off the network, my basic
> > approach is like this (simplified for brevity):
> > 
> > avcodec_send_packet(&pkt);
> > do {
> >     frame = av_frame_alloc();
> >     if ((err = avcodec_receive_frame(frame)) == 0) {
> >         render_frame(frame);
> >     }
> > } while (err == 0);
> > 
> > I'll usually get EAGAIN immediately for the first few frames I submit
> > (so no output frame yet), but then I'll get a batch of output frames
> > back after the first completed decode. That drains the excess latency
> > from the pipeline to avoid always being behind.
> > 
> > For cases where we want to prioritize latency over throughput, I've
> > had success with this approach too:
> > 
> > avcodec_send_packet(&pkt);
> > while (avcodec_receive_frame(frame) == AVERROR(EAGAIN)) {
> >     msleep(1);
> > }
> > render_frame(frame);
> > 
> > In this case, we can retry avcodec_receive_frame() until we get the
> > frame back that we just submitted for decoding.
> 
> > 
> > The patch here enables both of these use-cases by allowing V4L2M2M
> > to retry getting a decoded frame without new input data. Both of
> > these also work with MMAL after the recent decoupled dataflow patch.
> > 
> > > Could you share the dataset?
> > > 
> > 
> 
> > It is 720p60.h264 from here:
> > https://onedrive.live.com/?authkey=%21ALoKfcPfFeKyhzs&id=C15BF9770619F56%21165617&cid=0C15BF9770619F56

> 
> I could not decode this dataset on rpi4 (before or after the patch). Tried to upgrade kernel
> to 5.16, but still same problem. Odroid xu4 seems to work fine.

This patch fixes decoding on the RPi4 for me:
https://patchwork.ffmpeg.org/project/ffmpeg/patch/20210819085533.1174-2-ming.qian@nxp.com/

It would be nice to get the fix into the upcoming release.

-- 
Andriy