[Libav-user] Mapping / converting QTCapture's to FFmpeg's pixel format

[Alex Cohn]

On Tue, Feb 5, 2013 at 7:51 AM, Brad O'Hearne <brado at bighillsoftware.com> wrote:
> I have in my mind that the first one is probably the right pixel format (also
> the one I need to use), but there's some state of the resulting frame /
> packet that isn't being set properly I'm guessing, and it is causing to
> dereference a null pointer somewhere inside libavcodec. Does anyone have any
> idea what might be causing this? Is there some value I need to set before
> encoding the video?

Unfortunately, most video codecs support only few (usually, one) pixel
formats - both as input for encoding and output for decoding. When you
use ffmpeg command line and specify an incompatible pixel format,
ffmpeg will perform format conversion for you. But when you use the
libavcodec library, you must take care of this conversion yourself.

It works (well, fails miserably) as expected when you specify
AV_PIX_FMT_NV12 (which, as far as I can tell, is the ffmpeg equivalent
for kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange) for a codec that
does not support this format. You can use swscale to convert the
images to AV_PIX_FMT_YUV420P, which is supported by vast majority of
ffmpeg video encoders.

It does puzzle me, though, that you experience crash when you
configure your capture with kCVPixelFormatType_420YpCbCr8Planar. How
do you pass it to avpicture_fill()? If I understand correctly,
kCVPixelFormatType_420YpCbCr8Planar exposes three fields:
ComponentInfoY, ComponentInfoCb, and ComponentInfoCr. But ffmpeg
expects a contiguous byte array of Y pixels (w*h), followed by Cb
(w/2*h/2).

Therefore, even for conversion from
kCVPixelFormatType_420YpCbCr8Planar to AV_PIX_FMT_YUV420P you need
some extra step: create a byte array of size w*h*3/2 and copy the
three planes to this array. If the source has line padding, you should
take care of this, too.

Regards,
Alex