[Libav-user] avcodec_decode_video2 doesn't get pal8 palette and how to get "closest" pix_fmt

[Stefano Sabatini]

On date Sunday 2011-07-24 04:51:49 -0400, Matthew Einhorn encoded:
> Hi,
> 
> I'm using the ffmpeg API to decode videos and I ran into two issues
> while using it.
> 
> 1. I'm decoding by calling in a loop av_read_frame followed by
> avcodec_decode_video2 and if frameFinished I then use sws_scale to
> convert from my source format to dest format. This work fine for all
> videos except pal8 videos. For pal8, sws_scale results in a completely
> black picture. I suspected that the pal8 palette must be massed up and
> indeed the palette returned by avcodec_decode_video2 is all zero.

If the problem is the source image, you should check that the palette
in data[1] provided to libswscale is correct (e.g. you can check that
it is different from 0). Also check if the ff* tools can deal with the
file correctly. If not a sample program / sample test showing the
problem may be useful.

> In particular, from what I seemed to have read and seen of ffmpeg, for
> pal8 AVFrame data[0] is the data, while data[1] is the palette. When
> calling avcodec_decode_video2 on a pal8 video, data[0] is indeed data
> (bunch of different values), while data[1] is an array with all
> elements zero. Indeed, when I edited data[1] to some random values the
> sws_scale output image was not black anymore and you could see the
> remnants of my picture.
> 

> So I'm wondering, is the video file broken and that's why the palette
> doesn't show up? Or did I miss a flag when initializing codec/format
> context etc. so that the palette isn't read?

AFAIK you don't need any special hacks for working with palette
formats.
 
> 2. I'm looking for a function similar to avcodec_find_best_pix_fmt.
> What I want is to pass in a list of formats and the function would
> return what's the closest format. For example, say the source format
> is pal8 and I pass in as possible destination formats: RGB24 and
> GRAY8. Then the function should return GRAY8.
> avcodec_find_best_pix_fmt would return in that case RGB24 which "is"
> the best format, but in this case would waste 2 extra bytes since pal8
> is only 8 bytes depth and gray to start with.
> 
> Does a function like this exist? Would it be easy for me to write such
> a function using the ffmpeg API? And if so can I get some pointers?

Should be easy to hack the logic of avcodec_find_best_pix_fmt() for
implementing an avcodec_find_closest_pix_fmt() or such.

> Thanks in advance for any help,