[FFmpeg-devel] [PATCH v2 0/8] ddagrab source filter, lavfi.c wrapped_avframe and dependent changes

[Soft Works]

> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces at ffmpeg.org> On Behalf Of
> Timo Rothenpieler
> Sent: Saturday, July 9, 2022 2:15 AM
> To: ffmpeg-devel at ffmpeg.org
> Subject: Re: [FFmpeg-devel] [PATCH v2 0/8] ddagrab source filter,
> lavfi.c wrapped_avframe and dependent changes
> 
> On 09.07.2022 02:01, Soft Works wrote:
> > I can submit the missing bit of differences as a patch. I thought
> > it was no longer needed. The requirements have also changed over
> > MSDK versions. The 8bit file mapping that I've shown recently
> exists
> > in fact because an earlier MSDK version was requesting such
> texture.
> >
> > Anyway, when something is causing trouble, then it should be fixed
> > in hwcontext_qsv.
> 
> Settings those flags correctly isn't at all a bad idea though, and
> not
> fixing any issue I actively experienced. Just an oversight I noticed
> while reading the code.
> 
> Fixing the heap overflow is the major issue and can really only be
> done
> in the d3d11 hwcontext.
> Look at the texture_flags array.
> It only ever gets initialized to the size of the initial pool size.
> With a non-fixed-size pool, that grows over time, that obviously is
> an
> issue, and will trash whatever comes after the hwcontext on the heap
> whenever more frames are requested than initially allocated.
> With a initial size of 0, that is... immediately.
> 
> 
> An entire other issue this does not address at all, but which also
> does
> not cause any memory corruption at least:
> 
> QSV can't properly deal with a non-fixed-size pool.
> What happens if more fresh frames are allocated after the qsv hwctx
> has
> been derived?
>  From the looks of it, it iterates and maps all the textures only
> once
> at init.

There's an allocator pattern which filters are using, so when using
non-array textures it might be possible to have bigger pools than
textures allocated. But that's just a "might".
Please give me one or two days to respond with a better and more 
comprehensive answer. Right now, I'm deeply drowned into something
else ;-)

Thanks,
sw