[FFmpeg-cvslog] Fixes avpicture_layout to not write past buffer end.

[Matthew Einhorn]

ffmpeg | branch: master | Matthew Einhorn <moiein2000 at gmail.com> | Wed Aug 24 20:14:03 2011 -0400| [e662b263d9c500270a8f1dc7e1b81b51d5bdfd4e] | committer: Michael Niedermayer

Fixes avpicture_layout to not write past buffer end.

avpicture_get_size() returns the size of buffer required for avpicture_layout.
For pseudo-paletted formats (gray8...) this size does not include the palette.
However, avpicture_layout doesn't know this and still writes the palette. Consequently,
avpicture_layout writes passed the length of the buffer. This fixes it
by fixing avpicture_layout so that it doesn't write the palette for these formats.

Signed-off-by: Matthew Einhorn <moiein2000 at gmail.com>
Signed-off-by: Michael Niedermayer <michaelni at gmx.at>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=e662b263d9c500270a8f1dc7e1b81b51d5bdfd4e
---

 libavcodec/imgconvert.c |   10 ++++++++++
 1 files changed, 10 insertions(+), 0 deletions(-)

diff --git a/libavcodec/imgconvert.c b/libavcodec/imgconvert.c
index 5f5a50e..64b0838 100644
--- a/libavcodec/imgconvert.c
+++ b/libavcodec/imgconvert.c
@@ -337,6 +337,16 @@ int avpicture_layout(const AVPicture* src, enum PixelFormat pix_fmt, int width,
         }
     }
 
+    switch (pix_fmt) {
+    case PIX_FMT_RGB8:
+    case PIX_FMT_BGR8:
+    case PIX_FMT_RGB4_BYTE:
+    case PIX_FMT_BGR4_BYTE:
+    case PIX_FMT_GRAY8:
+        // do not include palette for these pseudo-paletted formats
+        return size;
+    }
+
     if (desc->flags & PIX_FMT_PAL)
         memcpy((unsigned char *)(((size_t)dest + 3) & ~3), src->data[1], 256 * 4);