[FFmpeg-devel] [PATCH v2 0/8] ddagrab source filter, lavfi.c wrapped_avframe and dependent changes

[Timo Rothenpieler]

On 09.07.2022 02:01, Soft Works wrote:
> I can submit the missing bit of differences as a patch. I thought
> it was no longer needed. The requirements have also changed over
> MSDK versions. The 8bit file mapping that I've shown recently exists
> in fact because an earlier MSDK version was requesting such texture.
> 
> Anyway, when something is causing trouble, then it should be fixed
> in hwcontext_qsv.

Settings those flags correctly isn't at all a bad idea though, and not 
fixing any issue I actively experienced. Just an oversight I noticed 
while reading the code.

Fixing the heap overflow is the major issue and can really only be done 
in the d3d11 hwcontext.
Look at the texture_flags array.
It only ever gets initialized to the size of the initial pool size.
With a non-fixed-size pool, that grows over time, that obviously is an 
issue, and will trash whatever comes after the hwcontext on the heap 
whenever more frames are requested than initially allocated.
With a initial size of 0, that is... immediately.


An entire other issue this does not address at all, but which also does 
not cause any memory corruption at least:

QSV can't properly deal with a non-fixed-size pool.
What happens if more fresh frames are allocated after the qsv hwctx has 
been derived?
 From the looks of it, it iterates and maps all the textures only once 
at init.