[FFmpeg-devel] [PATCH v2 0/8] ddagrab source filter, lavfi.c wrapped_avframe and dependent changes

[Timo Rothenpieler]

On 09.07.2022 02:28, Soft Works wrote:
> 
> 
>> -----Original Message-----
>> From: ffmpeg-devel <ffmpeg-devel-bounces at ffmpeg.org> On Behalf Of
>> Timo Rothenpieler
>> Sent: Saturday, July 9, 2022 2:15 AM
>> To: ffmpeg-devel at ffmpeg.org
>> Subject: Re: [FFmpeg-devel] [PATCH v2 0/8] ddagrab source filter,
>> lavfi.c wrapped_avframe and dependent changes
>>
>> On 09.07.2022 02:01, Soft Works wrote:
>>> I can submit the missing bit of differences as a patch. I thought
>>> it was no longer needed. The requirements have also changed over
>>> MSDK versions. The 8bit file mapping that I've shown recently
>> exists
>>> in fact because an earlier MSDK version was requesting such
>> texture.
>>>
>>> Anyway, when something is causing trouble, then it should be fixed
>>> in hwcontext_qsv.
>>
>> Settings those flags correctly isn't at all a bad idea though, and
>> not
>> fixing any issue I actively experienced. Just an oversight I noticed
>> while reading the code.
>>
>> Fixing the heap overflow is the major issue and can really only be
>> done
>> in the d3d11 hwcontext.
>> Look at the texture_flags array.
>> It only ever gets initialized to the size of the initial pool size.
>> With a non-fixed-size pool, that grows over time, that obviously is
>> an
>> issue, and will trash whatever comes after the hwcontext on the heap
>> whenever more frames are requested than initially allocated.
>> With a initial size of 0, that is... immediately.
>>
>>
>> An entire other issue this does not address at all, but which also
>> does
>> not cause any memory corruption at least:
>>
>> QSV can't properly deal with a non-fixed-size pool.
>> What happens if more fresh frames are allocated after the qsv hwctx
>> has
>> been derived?
>>   From the looks of it, it iterates and maps all the textures only
>> once
>> at init.
> 
> There's an allocator pattern which filters are using, so when using
> non-array textures it might be possible to have bigger pools than
> textures allocated. But that's just a "might".
> Please give me one or two days to respond with a better and more
> comprehensive answer. Right now, I'm deeply drowned into something
> else ;-)

That issue is entirely unrelated to the ddagrab filter.
All it needs is the heap overflow fixed, since it very much does run 
into that.

The other stuff are just follow ups I found while skimming through that 
code.
So if you want more time to look into those, that's no big deal at all 
from my end.
And the fix for the heap overflow is as straight forward as it gets.