[FFmpeg-devel] avpicture_layout() writes passed the buffer length for pseudo-paletted formats (bug)

[Matthew Einhorn]

On Thu, Aug 25, 2011 at 10:21 AM, Stefano Sabatini <
stefano.sabatini-lala at poste.it> wrote:

> On date Sunday 2011-08-21 14:09:12 -0400, Matthew Einhorn encoded:
> > Hi,
> >
> > The problem is that in the docs avpicture_layout() says it writes a
> > picture with size avpicture_get_size(). So one would make the buffer
> > input to avpicture_layout() of that size. However, for pseudo-paletted
> > formats (as listed in avpicture_get_size()) the amount written to
> > buffer is avpicture_get_size()+1024 (which is also the
> > avpicture_fill() size).
> >
> > If the buffer size is less than avpicture_get_size() (as indicated by
> > dest_size in avpicture_layout()) an error is returned by
> > avpicture_layout(). However, for these pseudo-paletted formats no
> > error is returned if the buffer is less than avpicture_get_size()+1024
> > and the function simply writes passed the actual buffer length for a
> > total of avpicture_get_size()+1024.
> >
> > The reason for this it seems is that avpicture_layout() wasn't
> > supposed to write the palette for these pseudo-paletted formats, but
> > the function still writes it resulting in the problem.
> >
> >
> > The solution is to:
> > *Either remove the exception for pseudo-paletted formats so that
> > avpicture_get_size() now completely relies on avpicture_fill() (will
> > probably break backwards compatibility) and both return the same size.
> > *Make an exception in avpicture_layout() as well for the
> > pseudo-paletted formats so that the palette is not written for them.
> > *If this "is" intended behavior update avpicture_layout() docs to
> > indicate that for these formats the buffer needs to be larger than
> > avpicture_get_size() by 1024.
>
> > *Or maybe there's some other deeper issue here (other then to prevent
> > copying of a standard palette) as to why these pseudo-paletted formats
> > are treated specially?
>
> Best solution is to get rid of these pseudo-palette hacks altogheter,
> please wait since I have some cached patches for that.
> --
>

Bump.
Just reminding about these patches.

Thanks,
Matt