[Ffmpeg-devel] swscale and palette ... what am I missing?

[Karl H. Beckers]

Am Mittwoch, den 28.02.2007, 01:00 +0100 schrieb
ffmpeg-devel-request at mplayerhq.hu:
> [...] 
> > Am I missing anything obvious?
> 
> hmm dunno, very quickly looking at your mail it looks all ok, i
> suggest you
> place a few random av_log() in the code to see where the non zero pal
> becomes
> NULL
> 
> and of course a patch to fix this is very welcome if its a bug :)
> 
> [...]

Well then,

looking at swscale_template.c seems to suggest the following (though I
wonder how this can ever have worked in any test, but be that as it
may):

static int RENAME(swScale)(SwsContext *c, uint8_t* src[], int srcStride[], int srcSliceY,
              int srcSliceH, uint8_t* dst[], int dstStride[]){

does not get the pal parameter, but sets it for consecutive functions.
It does it here:

if(isPacked(c->srcFormat)){
	pal= src[1];
	src[0]=
	src[1]=
	src[2]= src[0];
	srcStride[0]=
	srcStride[1]=
	srcStride[2]= srcStride[0];
}

since pal ist set to NULL before that, pal will never be anything else
if we don't enter here. This raises two issues:

1) isPacked did non return true for PIX_FMT_PAL8

khb at ubuntu:~/Quellen/xvidcap/trunk/ffmpeg/libswscale$ diff -Naurw swscale.c.orig swscale.c
--- swscale.c.orig      2007-02-28 09:21:36.000000000 +0100
+++ swscale.c   2007-02-28 09:22:22.000000000 +0100
@@ -116,7 +116,7 @@
                        || (x)==PIX_FMT_NV12 || (x)==PIX_FMT_NV21\
                        || (x)==PIX_FMT_GRAY16BE || (x)==PIX_FMT_GRAY16LE\
                        || (x)==PIX_FMT_GRAY8 || (x)==PIX_FMT_YUV410P)
-#define isPacked(x)    ((x)==PIX_FMT_YUYV422 || (x)==PIX_FMT_UYVY422 ||isRGB(x) || isBGR(x))
+#define isPacked(x)    ((x)==PIX_FMT_PAL8 || (x)==PIX_FMT_YUYV422 || (x)==PIX_FMT_UYVY422 ||isRGB(x) || isBGR(x))
 
 #define RGB2YUV_SHIFT 16
 #define BY ((int)( 0.098*(1<<RGB2YUV_SHIFT)+0.5))


2) the picture passed as input to sws_scale is changed.

That is smth. you at least need to know. I was keeping the AVFrame
structure and only loading new data into data[0] and setting data[1]
only on the first frame, since the palette will stay the same. Because
the above part of swScale changes data[1] on the source AVFrame to equal
data[0] in the first call, the palette will be rubbish on the second
call. But pal will still be valid as long as isPacked returns true. 

Is this essential, or couldn't we just leave the source AVFrame alone?



Fixing (1) and setting data[1] for on the frame before any call to
sws_scale (for 2) almost gets me where I want to go. Now it is just that
the result seems to indicate some error with the color planes.

You can take a look at:
http://www.jarre-de-the.net/computing/test-0000.mpeg
http://www.jarre-de-the.net/computing/tester.png
(for some reason the png will only display correctly with ImageMagick's
"display")
For the MPEG the destination pix_fmt was, of course, yuv420p, for the
png (because libswscale does not support pal8 as an output format) the
output pix_fmt was rgba32. Since both show the same effect (looking like
the color planes are off), it seems that the issue is in the conversion
FROM pal8 rather than in the conversion TO smth, or does every
conversion go through a common intermediate pix format?

Will try to see if I can find some more, but the end of my wits will be
the inlined asm code (which I haven't done since my Amiga 500 days ;S)

Regards,
Karl.