[FFmpeg-devel] avpicture_layout() writes passed the buffer length for pseudo-paletted formats (bug)

[Michael Niedermayer]

On Mon, Oct 17, 2011 at 11:27:21PM -0400, Matthew Einhorn wrote:
> On Thu, Aug 25, 2011 at 10:21 AM, Stefano Sabatini <
> stefano.sabatini-lala at poste.it> wrote:
> 
> > On date Sunday 2011-08-21 14:09:12 -0400, Matthew Einhorn encoded:
> > > Hi,
> > >
> > > The problem is that in the docs avpicture_layout() says it writes a
> > > picture with size avpicture_get_size(). So one would make the buffer
> > > input to avpicture_layout() of that size. However, for pseudo-paletted
> > > formats (as listed in avpicture_get_size()) the amount written to
> > > buffer is avpicture_get_size()+1024 (which is also the
> > > avpicture_fill() size).
> > >
> > > If the buffer size is less than avpicture_get_size() (as indicated by
> > > dest_size in avpicture_layout()) an error is returned by
> > > avpicture_layout(). However, for these pseudo-paletted formats no
> > > error is returned if the buffer is less than avpicture_get_size()+1024
> > > and the function simply writes passed the actual buffer length for a
> > > total of avpicture_get_size()+1024.
> > >
> > > The reason for this it seems is that avpicture_layout() wasn't
> > > supposed to write the palette for these pseudo-paletted formats, but
> > > the function still writes it resulting in the problem.
> > >
> > >
> > > The solution is to:
> > > *Either remove the exception for pseudo-paletted formats so that
> > > avpicture_get_size() now completely relies on avpicture_fill() (will
> > > probably break backwards compatibility) and both return the same size.
> > > *Make an exception in avpicture_layout() as well for the
> > > pseudo-paletted formats so that the palette is not written for them.
> > > *If this "is" intended behavior update avpicture_layout() docs to
> > > indicate that for these formats the buffer needs to be larger than
> > > avpicture_get_size() by 1024.
> >
> > > *Or maybe there's some other deeper issue here (other then to prevent
> > > copying of a standard palette) as to why these pseudo-paletted formats
> > > are treated specially?
> >
> > Best solution is to get rid of these pseudo-palette hacks altogheter,
> > please wait since I have some cached patches for that.
> > --
> >
> 
> Bump.
> Just reminding about these patches.

btw, fate passed with your patch

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

The misfortune of the wise is better than the prosperity of the fool.
-- Epicurus
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20111018/7f309c01/attachment.asc>