[Ffmpeg-devel] swscale and palette ... what am I missing?

[Karl H. Beckers]

Hi all,

I'm trying to convert a pal8 frame to yuv420p using libswscale rather
than converting it to rbg myself. From what I'm reading in the archives
I kinda think this should be possible, but I'm not getting past a
segfault in swScale_MMX with input_pixfmt = PIX_FMT_PAL8 where RGBx work
nicely.

I'm setting up my frame like the following, contiguous char data in
data[0], linesize in bytes in linesize[0] and pointer to palette as arbg
(with alpha always 0) in data[1]:
            avpicture_fill ((AVPicture *) p_inpic, scratchbuf8bit,
                            input_pixfmt, image->width, image->height);
            p_inpic->data[0] = scratchbuf8bit;
            p_inpic->linesize[0] = image->width;
            p_inpic->data[1] = job->color_table;

then I construct my SwsContext:
            img_resample_ctx = sws_getContext (image->width,
                                               image->height,
                                               input_pixfmt,
                                               out_st->codec->width,
                                               out_st->codec->height,
                                               out_st->codec->pix_fmt,1,
                                               NULL, NULL, NULL);

and later call the sws_scale function:
    if (sws_scale (img_resample_ctx, p_inpic->data, p_inpic->linesize,
                   0, image->height, p_outpic->data, p_outpic->linesize)
 		< 0) {
        fprintf (stderr, .......


this always yields a segfault immediately:

Program terminated with signal 11, Segmentation fault.
#0  0x0836dd0f in swScale_MMX (c=0x87f2ca0, src=0x87c5490,
srcStride=0xb639228c, srcSliceY=0, srcSliceH=144, dst=0x87c5570,
dstStride=0xb6392298)
    at swscale_template.c:2288
2288                    int b= pal[d]     &0xFF;


Trying to debug this:

Starting program: /home/kb87850/xvidcap/bin/xvidcap 
[Thread debugging using libthread_db enabled]
[New Thread -1221232448 (LWP 7570)]
[New Thread -1238369376 (LWP 7571)]
[mpeg4 @ 0x8444f60]removing common factors from framerate
[New Thread -1248855136 (LWP 7572)]
[Switching to Thread -1238369376 (LWP 7571)]

Breakpoint 1, swScale_MMX (c=0xb5950fc0, src=0xb5935910,
srcStride=0xb62ff28c, srcSliceY=0, srcSliceH=144, dst=0xb593c9e0,
dstStride=0xb62ff298)
    at swscale_template.c:2285
2285            for(i=0; i<width; i++)
(gdb) print i
$7 = <value optimized out>
(gdb) print width
No symbol "width" in current context.
(gdb) step
2288                    int b= pal[d]     &0xFF;
(gdb) print d
No symbol "d" in current context.
(gdb) print src[0]
$8 = (uint8_t *) 0xb59359d8 ""
(gdb) print /d src[0]
$9 = 3046332888
(gdb) print pal
$10 = (uint8_t *) 0x0
(gdb) print *pal
Cannot access memory at address 0x0
(gdb) print pal[3046332888]
$11 = 0 '\0'


comparing with the relevant code in swscale_template.c:

static inline void RENAME(palToY)(uint8_t *dst, uint8_t *src, int width,
uint32_t *pal)
{
        int i;
        for(i=0; i<width; i++) 
        {    
                int d= src[i];
                int b= pal[d]     &0xFF;
                int g=(pal[d]>>8 )&0xFF;
                int r= pal[d]>>16;
av_log(NULL, AV_LOG_DEBUG, "i: %i\n", i);

                dst[i]= ((RY*r + GY*g + BY*b)>>RGB2YUV_SHIFT) + 16;
        }    
}

in gdb I see neither width nor stepping through the program see the line
"int d= src[i];" at all. pal seems to be a NULL pointer, but I don't
know why ... the palette needs to go in AVFrame.data[1] after all,
doesn't it?

Am I missing anything obvious?

TIA,

Karl.