[Libav-user] avcodec_decode_video2 doesn't get pal8 palette and how to get "closest" pix_fmt

[Matthew Einhorn]

Hi,

I'm using the ffmpeg API to decode videos and I ran into two issues
while using it.

1. I'm decoding by calling in a loop av_read_frame followed by
avcodec_decode_video2 and if frameFinished I then use sws_scale to
convert from my source format to dest format. This work fine for all
videos except pal8 videos. For pal8, sws_scale results in a completely
black picture. I suspected that the pal8 palette must be massed up and
indeed the palette returned by avcodec_decode_video2 is all zero.

In particular, from what I seemed to have read and seen of ffmpeg, for
pal8 AVFrame data[0] is the data, while data[1] is the palette. When
calling avcodec_decode_video2 on a pal8 video, data[0] is indeed data
(bunch of different values), while data[1] is an array with all
elements zero. Indeed, when I edited data[1] to some random values the
sws_scale output image was not black anymore and you could see the
remnants of my picture.

So I'm wondering, is the video file broken and that's why the palette
doesn't show up? Or did I miss a flag when initializing codec/format
context etc. so that the palette isn't read?


2. I'm looking for a function similar to avcodec_find_best_pix_fmt.
What I want is to pass in a list of formats and the function would
return what's the closest format. For example, say the source format
is pal8 and I pass in as possible destination formats: RGB24 and
GRAY8. Then the function should return GRAY8.
avcodec_find_best_pix_fmt would return in that case RGB24 which "is"
the best format, but in this case would waste 2 extra bytes since pal8
is only 8 bytes depth and gray to start with.

Does a function like this exist? Would it be easy for me to write such
a function using the ffmpeg API? And if so can I get some pointers?

Thanks in advance for any help,
Matt