[Libav-user] Decoding h264 (NV12) stream from NVENC

[Hendrik Leppkes]

On Mon, Jul 31, 2017 at 10:00 AM, Michael IV <explomaster at gmail.com> wrote:
> Hi! I am using NVENC encoder to create h264 stream.The YUV format is NV12.
> Now, I need also to decode that stream back to YUV.I wrote a module for
> that,based on
> this example:
>
> https://gist.github.com/roxlu/9329339
>
> But, because my YUV format is not 420P, but NV12 I am trying to force it on
> AVCodecContext when initializing it with this line:
>
>         avcodec_register_all();
>         mCodec = avcodec_find_decoder(AV_CODEC_ID_H264);
>         mCodecContext = avcodec_alloc_context3(mCodec);
>         mCodecContext->pix_fmt = AVPixelFormat::AV_PIX_FMT_NV12;
>
>
> But when I am receiving the decoded AVFrame, I see that the AVFrame->format
> property is
> set to zero, which is enum for 420P. Also inspecting the planes and other
> props of this struct I see that the decoder tried to decode it as 420P. (3
> planes , 3 line sizes etc.) .And of course,when trying to play the output of
> such a YUV frame , there is just a green screen.
> I am trying to understand where is my error.Is there some data that decoder
> misses from the h264 stream and it doesn't allow it to guess the pixel
> format?Or maybe I need to set pixel format for the codec context in a some
> different way?
>

You can't control the output pixel format of the H264 decoder. A video
is not encoded as "NV12", its encoded as 4:2:0 YUV, NV12 is just one
representation of this format.
As such, our decoder will always decode 4:2:0 YUV content to yuv420p,
not nv12. If you require NV12 in your application, you can convert
from yuv420p to nv12 using swscale (a lossless conversion).

- Hendrik