[Libav-user] av_hwframe_transfer_data does not succeed from NV12 to YUV420P - How to get YUV420P from cuvid?

[Mark Thompson]

On 05/06/18 11:15, Sergio Basurco wrote:
> I'm decoding h264 with ffmpeg. I want to use the hwaccel decoders. I'm using the cuvid decoder via API. In the fftools code there's a function "hwaccel_retrieve_data" that is supposed to convert the decoded frame (NV12) into any other format, I'm trying YUV420P.
> 
> The conversion does not return any error, however the resulting data is not correct. Here's the original NV12 frame:
> 
> https://imgur.com/a/jAb8h12
> 
> And here's the conversion to YUV420P (only avframe->data[0] and avframe->data[1] have any data, data[2] is expected to have the V data but it is missing).
> 
> https://imgur.com/a/ihQeJ0M
> 
> 
> I think I'm on the right track, based on the code in hwcontext_cuda.c aparently YUV420P is a supported format, I cannot get my head around how I can tell the decoder to convert from NV12 to YUV420P though.
> 
> Any tips will be appreciated, I'll update if I find anything.

See the documentation for av_hwframe_transfer_data() (<http://ffmpeg.org/doxygen/trunk/hwcontext_8h.html#abf1b1664b8239d953ae2cac8b643815a>), in particular:

"If src has an AVHWFramesContext attached, then the format of dst (if set) must use one of the formats returned by av_hwframe_transfer_get_formats(src, AV_HWFRAME_TRANSFER_DIRECTION_FROM). If dst has an AVHWFramesContext attached, then the format of src must use one of the formats returned by av_hwframe_transfer_get_formats(dst, AV_HWFRAME_TRANSFER_DIRECTION_TO)"

For CUDA, no conversion during transfer is supported so the only usable output format returned by av_hwframe_transfer_get_formats() is the same format as the GPU-side frame itself (NV12 in your case): <http://git.videolan.org/?p=ffmpeg.git;a=blob;f=libavutil/hwcontext_cuda.c#l176>.

- Mark