[FFmpeg-devel] How to Decode Packet Encoded in Hardware (nvenc)?

[Ming Ying]

Hi!

TL;DR: I am encoding AVFrames using h264_nvenc on my GPU-enabled computer,
and now I wish to send  these packets to a computer without a GPU/hardware
acceleration and decode them there. How can I decode these packets and
convert them to YUV420 pixel format?

--------------------------------------

After reading these examples (1
<https://github.com/FFmpeg/FFmpeg/blob/master/doc/examples/vaapi_encode.c>)
(2
<https://github.com/FFmpeg/FFmpeg/blob/a0ac49e38ee1d1011c394d7be67d0f08b2281526/fftools/ffmpeg_cuvid.c?fbclid=IwAR2wB0I78EFznVUupbjxIa0l7AIunxpCSHOfiVz9kMdzJ0w__MHBJH7UK3w>)
I was able to encode frames using h264_nvenc using the following procedure:

1. Create h264_nvenc codec and AVCodecContext
2. Create hw_device_ctx
3. Create hw_frames_ctx, pass it into the AVCodecContext
4. Create a "sw_frame" and "hw_frame"
5. Read raw buffer data into the sw_frame, and use hw_transfer_data to
transfer the sw_frame to the hw_frame
6. Encode the hw_frame

The issue I'm having is that I wish to decode the hw_frame (which has
AV_PIX_FMT_CUDA) into an AVFrame, and convert the pixel format to
AV_PIX_FMT_YUV420P. I have found examples
<https://github.com/FFmpeg/FFmpeg/blob/master/doc/examples/hw_decode.c> on
how to do this if you are decoding on a GPU-enabled device using
h264_cuvid. But, I can't figure out how to decode and convert to YUV a
hardware-encoded packet using just software decoding (or, alternatively,
another type of hardware decoding like QSV).

What I've attempted so far:

I've tried following the suggestions in this post
<https://stackoverflow.com/questions/47049312/how-can-i-convert-an-ffmpeg-avframe-with-pixel-format-av-pix-fmt-cuda-to-a-new-a>.
It seems that setting the decoder pixel format to AV_PIX_FMT_CUDA leads to
a successful decode, but I can't figure out how to convert the pixel format
to YUV (calling sws_scale results in a "slice parameters are invalid"
error). I also see that YUV420P is a valid software pixel format for the
CUDA hardware pixel format, but not sure what to do with that information.

Thank you for your help! Much appreciated :)