[FFmpeg-user] H264 VAAPI Encoder

[Mark Thompson]

On 17/06/16 23:33, Andy Furniss wrote:
> 
> AMD are working on vaapi encode for mesa, only a few patches about so far and they got rejected - though not for functionality.
> 
> They do work with gstreamer, but trying above with ffmpeg fails as below (render node or X) seems the surface is seen as rgb - but the h/w takes nv12.
> 
> Maybe mesa/the driver is giving false info?
> 
> vainfo
> libva info: VA-API version 0.38.1
> libva info: va_getDriverName() returns 0
> libva info: User requested driver 'radeonsi'
> libva info: Trying to open /usr/lib/dri/radeonsi_drv_video.so
> libva info: Found init function __vaDriverInit_0_38
> libva info: va_openDriver() returns 0
> vainfo: VA-API version: 0.38 (libva 1.6.3.pre1)
> vainfo: Driver version: mesa gallium vaapi
> vainfo: Supported profile and entrypoints
>       VAProfileMPEG2Simple            : VAEntrypointVLD
>       VAProfileMPEG2Main              : VAEntrypointVLD
>       VAProfileVC1Simple              : VAEntrypointVLD
>       VAProfileVC1Main                : VAEntrypointVLD
>       VAProfileVC1Advanced            : VAEntrypointVLD
>       VAProfileH264Baseline           : VAEntrypointVLD
>       VAProfileH264Baseline           : VAEntrypointEncSlice
>       VAProfileH264Baseline           : VAEntrypointEncPicture
>       VAProfileH264Main               : VAEntrypointVLD
>       VAProfileH264High               : VAEntrypointVLD
>       VAProfileNone                   : VAEntrypointVideoProc
> 
> ffmpeg doesn't seem to care about VAProfileH264Baseline
> 
> I tried avconv and it does care = bail, but I can use -profile 66 and then get the same error as ffmpeg (though it converts to rgba rather than rgb0)

Default for H264 is High, so yeah you need -profile 66 to get it to work.

> [AVHWDeviceContext @ 0x3ca4c40] Format 0x3231564e -> unknown.
> [AVHWDeviceContext @ 0x3ca4c40] Format 0x32315659 -> unknown.
> [AVHWDeviceContext @ 0x3ca4c40] Format 0x56595559 -> unknown.
> [AVHWDeviceContext @ 0x3ca4c40] Format 0x59565955 -> unknown.
> [AVHWDeviceContext @ 0x3ca4c40] Format 0x41524742 -> bgra.
> [AVHWDeviceContext @ 0x3ca4c40] Format 0x41424752 -> rgba.
> [AVHWDeviceContext @ 0x3ca4c40] Format 0x58524742 -> bgr0.
> [AVHWDeviceContext @ 0x3ca4c40] Format 0x58424752 -> rgb0.

The detection queries the capabilities of a video processor instance (VAProfileNone / VAEntrypointVideoProc) to try to determine which formats are interesting, then ignores the ones which aren't.  It should probably have said "not supported" here rather than "unknown" for nv12 (and yv12 immediately below it).

I've sent a patch to change the behaviour slightly which will improve this case, see <https://lists.libav.org/pipermail/libav-devel/2016-June/077669.html>.

> [auto-inserted scaler 0 @ 0x3cf2a60] w:720 h:576 fmt:yuv420p sar:1/1 -> w:720 h:576 fmt:nv12 sar:1/1 flags:0x4
> [auto-inserted scaler 1 @ 0x3cf4ec0] w:720 h:576 fmt:nv12 sar:1/1 -> w:720 h:576 fmt:rgb0 sar:1/1 flags:0x4
> [hwupload @ 0x3cd61c0] Surface format is rgb0.

hwupload reports its input capabilities based on what is returned above, so it will be saying that it can only take the RGB formats as input.  Hence ffmpeg auto-inserts a swscale instance to change it back from nv12 to rgba.  (The filter chain becomes 'format=nv12,format=rgba,hwupload'.)

> [AVHWFramesContext @ 0x3cf1700] Created surface 0x1.
> [AVHWFramesContext @ 0x3cf1700] Direct mapping possible.
> [h264_vaapi @ 0x3ce7dc0] Using fixed QP = 20 / 20 / 24 for IDR / P / B frames.
> [h264_vaapi @ 0x3ce7dc0] Using (null) as format of reconstructed frames.
> [AVHWFramesContext @ 0x3d37fa0] Unsupported format: (null).

The input to the encode is now in VAAPI rgba surfaces, but that doesn't match any of the capabilities so it doesn't know what to make the reconstructed frames.  It should have fallen back to the one supported format (i.e. nv12), but a typo stoped it from doing that: fixed in <https://lists.libav.org/pipermail/libav-devel/2016-June/077668.html>.

> [h264_vaapi @ 0x3ce7dc0] Failed to initialise reconstructed frame context: -22.
> Output #0, matroska, to 'out.mkv':
>     Stream #0:0, 0, 0/0: Unknown: none, SAR 1:1 DAR 0:0
>     Metadata:
>       encoder         : Lavc57.46.100 h264_vaapi
> Stream mapping:
>   Stream #0:0 -> #0:0 (mpeg2video (native) -> h264 (h264_vaapi))
> Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
> [AVIOContext @ 0x3ce8760] Statistics: 0 seeks, 0 writeouts
> [AVIOContext @ 0x3bfa540] Statistics: 2019472 bytes read, 2 seeks

So, I also went and built mesa with the encode patches and had a go.  With the changes above, it does get to actually trying to encode, but that nuked my GPU to the point of requiring reboot.

With the linked patch you should be able to have a go too: if it works for you then I'm interested in what your setup is.  If it nukes your GPU then, well, you had some warning at least.  I think I'll wait until it's a bit more stable before pursuing this further.

Thanks,

- Mark