[FFmpeg-devel] avpicture_layout() writes passed the buffer length for pseudo-paletted formats (bug)

[Stefano Sabatini]

On date Tuesday 2011-10-18 16:24:15 +0200, Michael Niedermayer encoded:
> On Mon, Oct 17, 2011 at 11:27:21PM -0400, Matthew Einhorn wrote:
> > On Thu, Aug 25, 2011 at 10:21 AM, Stefano Sabatini <
> > stefano.sabatini-lala at poste.it> wrote:
> > 
> > > On date Sunday 2011-08-21 14:09:12 -0400, Matthew Einhorn encoded:
> > > > Hi,
> > > >
> > > > The problem is that in the docs avpicture_layout() says it writes a
> > > > picture with size avpicture_get_size(). So one would make the buffer
> > > > input to avpicture_layout() of that size. However, for pseudo-paletted
> > > > formats (as listed in avpicture_get_size()) the amount written to
> > > > buffer is avpicture_get_size()+1024 (which is also the
> > > > avpicture_fill() size).
> > > >
> > > > If the buffer size is less than avpicture_get_size() (as indicated by
> > > > dest_size in avpicture_layout()) an error is returned by
> > > > avpicture_layout(). However, for these pseudo-paletted formats no
> > > > error is returned if the buffer is less than avpicture_get_size()+1024
> > > > and the function simply writes passed the actual buffer length for a
> > > > total of avpicture_get_size()+1024.
> > > >
> > > > The reason for this it seems is that avpicture_layout() wasn't
> > > > supposed to write the palette for these pseudo-paletted formats, but
> > > > the function still writes it resulting in the problem.
> > > >
> > > >
> > > > The solution is to:
> > > > *Either remove the exception for pseudo-paletted formats so that
> > > > avpicture_get_size() now completely relies on avpicture_fill() (will
> > > > probably break backwards compatibility) and both return the same size.
> > > > *Make an exception in avpicture_layout() as well for the
> > > > pseudo-paletted formats so that the palette is not written for them.
> > > > *If this "is" intended behavior update avpicture_layout() docs to
> > > > indicate that for these formats the buffer needs to be larger than
> > > > avpicture_get_size() by 1024.
> > >
> > > > *Or maybe there's some other deeper issue here (other then to prevent
> > > > copying of a standard palette) as to why these pseudo-paletted formats
> > > > are treated specially?
> > >
> > > Best solution is to get rid of these pseudo-palette hacks altogheter,
> > > please wait since I have some cached patches for that.
> > > --
> > >
> > 
> > Bump.
> > Just reminding about these patches.
> 
> btw, fate passed with your patch

Feel free to apply, I can fix the hack later.