[FFmpeg-devel] [PATCH] get rid of nonsense palette pointer for RGB8, GRAY8 etc.

[Reimar Döffinger]

On Fri, Mar 20, 2009 at 07:42:47PM +0100, Michael Niedermayer wrote:
> removing the palette for gray8 will require a special case in all paletted
> encoders. I think this is worse than fixing whatever failed to set the
> palette

Haven't found that code yet, but I suspect it may already be correct and
the palette was only not copied by rawenc.c, so I won't bother with it
further.

> the code that dumps it into a raw file

See attached. Haven't found documentation for avpicture_layout though.
It also misses the regression test updates, because currently the
regressions tests for me create the file b-libav-gray.yuv with size 0,
but if I run the command manually it has the correct size of 304128...

For no particular reason I am also a bit suspicious if that
avpicture_layout function works correctly for the alpha formats...

About fixing the swscale planarCopy, are you fine with this ugly hack (I
assume there is no need to "reset" the GRAY8 palette data in dst):
Index: libswscale/swscale.c
===================================================================
--- libswscale/swscale.c        (revision 29018)
+++ libswscale/swscale.c        (working copy)
@@ -2111,6 +2111,9 @@
                       int srcSliceH, uint8_t* dst[], int dstStride[])
 {
     int plane;
+    // ignore palette for GRAY8
+    if (!dst[2]) dst[1] = NULL;
+    if (!src[2]) src[1] = NULL;
     for (plane=0; plane<4; plane++)
     {
         int length= (plane==0 || plane==3) ? c->srcW  : -((-c->srcW)>>c->chrDstHSubSample);
-------------- next part --------------
Index: libavcodec/imgconvert.c
===================================================================
--- libavcodec/imgconvert.c	(revision 18076)
+++ libavcodec/imgconvert.c	(working copy)
@@ -727,6 +727,7 @@
 int avpicture_layout(const AVPicture* src, int pix_fmt, int width, int height,
                      unsigned char *dest, int dest_size)
 {
+    unsigned char *dest_start = dest;
     const PixFmtInfo* pf = &pix_fmt_info[pix_fmt];
     int i, j, w, ow, h, oh, data_planes;
     const unsigned char* s;
@@ -777,10 +778,13 @@
          }
     }
 
-    if (pf->pixel_type == FF_PIXEL_PALETTE)
-        memcpy((unsigned char *)(((size_t)dest + 3) & ~3), src->data[1], 256 * 4);
+    if (pf->pixel_type == FF_PIXEL_PALETTE) {
+        dest += (-(long)dest) & 3;
+        memcpy(dest, src->data[1], 256 * 4);
+        dest += 256 * 4;
+    }
 
-    return size;
+    return dest - dest_start;
 }
 
 int avpicture_get_size(int pix_fmt, int width, int height)