[MPlayer-dev-eng] Serious memory exhaustion when playing H264 by ffh264vda decoder

[Reimar Döffinger]

On 18.11.2013, at 02:04, Xidorn Quan <quanxunzhen at gmail.com> wrote:
> On Mon, Nov 18, 2013 at 8:23 AM, Zongyao Qu <zongyao.qu at gmail.com> wrote:
>> I found the following patch could fix this issue.
> 
> This patch is very helpful to locate the problem, thanks.
> 
> The problem is that, h264_vda in ffmpeg is a special decoder which
> must manage the buffer itself since it is in fact a wrapper decoder of
> h264. But the code which is removed in the patch provided by Zongyao
> explicitly overrides the buffer management functions set during
> initialization of h264_vda.
> 
> @reimar, I reviewed your commit r36422, the one mentioned by Zongyao.
> I found that such code (overrides buffer functions when do_dr1 is
> false) didn't exist in the old code at all, so I think there might be
> some consideration. What did you think then? Besides, I think we have
> to avoid any change in buffer functions (and any parameters might have
> been used by avcodec) after avcodec_open2. I don't have an idea what
> to do now since the code in vd_ffmpeg seems to be fragile. Any
> suggestion?

I think mostly that h264_vda should not expose its internal functions in the avctx.
Next I think a good workaround would be not to touch these fields if CODEC_CAP_DR1 isn't set.
Lastly I don't think the code is particularly fragile. If the code used the values set before open that should still be fine, it just will cause a few additional warnings.
The problem is that h264_vda seems to override the values, that is IMO an API violation, the documentation says that the user sets them, and nothing about libavcodec setting them.
This would break things even if some user just _calls_ avctx.get_buffer themselves.