[FFmpeg-devel] [PATCH v3 0/5] Switch mmaldec to receive_frame API

[Cameron Gutman]

On 12/9/2021 7:01 AM, Andreas Rheinhardt wrote:
> This is an updated and extended version of [1] by Ho Ming Shun
> together with some additions by me. Notice that I am unable
> to test these patches (not even compilation), so I hope that
> others (Ho Ming Shun) do so.
> 
> [1]: https://ffmpeg.org/pipermail/ffmpeg-devel/2021-September/285992.html
> 
> Andreas Rheinhardt (2):
>    avcodec/mmaldec: Avoid creating unnecessary reference, simplify code
>    avcodec/mmaldec: Deduplicate AVClasses
> 
> Ho Ming Shun (3):
>    avcodec/mmaldec: use decoupled dataflow
>    avcodec/mmaldec: re-use AVPacket for extra_data
>    avcodec/mmaldec: fix pointer type warning
> 
>   libavcodec/mmaldec.c | 81 +++++++++++++++++++++++---------------------
>   1 file changed, 42 insertions(+), 39 deletions(-)
> 

This is fantastic work! I'm seeing significantly improved decode latency
here. I've been carrying an out-of-tree hack[0] for years to address the
extremely poor performance of mmaldec due to this async decoding behavior.

The refactoring in this patch set _almost_ matches my dirty hack.

With a realtime 60 FPS stream (so 1 packet per 16 ms) via Moonlight-Qt[1]:
- Current master: ~250 ms
- This patch set: ~18 ms
- My hack: ~14 ms

With this patchset, I usually end up receiving frame N-1 when submitting
frame N. My hack manages to avoid asynchronous decoding altogether, so
I'm always receiving frame N after sending frame N for decoding.

The absolute latency numbers are very close, but going over a frame of
latency is significant. The effective latency difference when it comes to
drawing the resulting image is 16 ms on a 60 Hz monitor due to V-sync.

I assume my hack is unsuitable for inclusion in FFmpeg, but perhaps we
can figure out a way to avoid having to patch FFmpeg to get that behavior.
Maybe an AVOption to set MAX_DELAYED_FRAMES rather than hardcoding it?

In any case, this is a _major_ improvement for my use so feel free to add
my Tested-by to this patch series:

Tested-by: Cameron Gutman <aicommander at gmail.com>



Regards,
Cam


[0]: https://github.com/cgutman/FFmpeg/commit/193a0320afd5f316da407208eb2fdceea888ff64
[1]: https://github.com/moonlight-stream/moonlight-qt