[FFmpeg-user] AMD GPU + Vaapi encoding problem

[Dennis Mungai]

On Wed, 8 Apr 2020 at 19:25, Ostap Batug <ostik0bleat at gmail.com> wrote:

> First of all sorry if this is a duplicate question, but during my search
> online i haven't found a solution.
> Hello, i have a problem, my CPU is pretty slow so i wanted to try use VAAPI
> encoding for rtmp streaming. (On the side note, using vaapi is kind of
> counter intuitive, but that can be ju me being a newbie) Regardless of what
> i try i get an error. Searching online i haven't found any information, but
> here's the same error even without streaming, but direct copy paste from "
> https://trac.ffmpeg.org/wiki/Hardware/VAAPI" of the middle screen capture
> command
> $ ffmpeg -vaapi_device /dev/dri/renderD128 -f x11grab -video_size 1920x1080
> -i :0 -vf 'format=nv12,hwupload' -c:v h264_vaapi -qp 24 output.mp4
> And this was the complete output:
> ffmpeg version n4.2.2 Copyright (c) 2000-2019 the FFmpeg developers
>   built with gcc 9.3.0 (Artix Linux 9.3.0-1)
>   configuration: --prefix=/usr --disable-debug --disable-static
> --disable-stripping --enable-fontconfig --enable-gmp --enable-gnutls
> --enable-gpl --enable-ladspa --enable-libaom --enable-libass
> --enable-libbluray --enable-libdav1d --enable-libdrm --enable-libfreetype
> --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libjack
> --enable-libmfx --enable-libmodplug --enable-libmp3lame
> --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg
> --enable-libopus --enable-libpulse --enable-libsoxr --enable-libspeex
> --enable-libssh --enable-libtheora --enable-libv4l2 --enable-libvidstab
> --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264
> --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid
> --enable-nvdec --enable-nvenc --enable-omx --enable-shared
> --enable-version3
>   libavutil      56. 31.100 / 56. 31.100
>   libavcodec     58. 54.100 / 58. 54.100
>   libavformat    58. 29.100 / 58. 29.100
>   libavdevice    58.  8.100 / 58.  8.100
>   libavfilter     7. 57.100 /  7. 57.100
>   libswscale      5.  5.100 /  5.  5.100
>   libswresample   3.  5.100 /  3.  5.100
>   libpostproc    55.  5.100 / 55.  5.100
> [x11grab @ 0x555f24fbc000] Stream #0: not enough frames to estimate rate;
> consider increasing probesize
> Input #0, x11grab, from ':0':
>   Duration: N/A, start: 1586362599.799368, bitrate: N/A
>     Stream #0:0: Video: rawvideo (BGR[0] / 0x524742), bgr0, 1920x1080,
> 29.97 fps, 1000k tbr, 1000k tbn, 1000k tbc
> Stream mapping:
>   Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (h264_vaapi))
> Press [q] to stop, [?] for help
> [h264_vaapi @ 0x555f24fc9000] Driver does not support some wanted packed
> headers (wanted 0xd, found 0).
> [h264_vaapi @ 0x555f24fc9000] Driver does not support packed sequence
> headers, but a global header is requested.
> [h264_vaapi @ 0x555f24fc9000] No global header will be written: this may
> result in a stream which is not usable for some purposes (e.g. not muxable
> to some containers).
> Output #0, mp4, to 'output.mp4':
>   Metadata:
>     encoder         : Lavf58.29.100
>     Stream #0:0: Video: h264 (h264_vaapi) (High) (avc1 / 0x31637661),
> vaapi_vld, 1920x1080, q=-1--1, 29.97 fps, 30k tbn, 29.97 tbc
>     Metadata:
>       encoder         : Lavc58.54.100 h264_vaapi
> [h264_vaapi @ 0x555f24fc9000] Failed to end picture encode issue: 5
> (invalid VAContextID).
> [h264_vaapi @ 0x555f24fc9000] Encode failed: -5.
> Video encoding failed
> Conversion failed!
>
> Please help,
> My GPU is:
> 00:01.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI]
> Trinity [Radeon HD 7660G]
> And glxinfo:
> OpenGL vendor string: X.Org
> OpenGL renderer string: AMD ARUBA (DRM 2.50.0 / 5.5.10-artix1-1, LLVM
> 9.0.1)
> OpenGL core profile version string: 4.3 (Core Profile) Mesa 20.0.4
> OpenGL core profile shading language version string: 4.30
> OpenGL core profile context flags: (none)
> OpenGL core profile profile mask: core profile
> OpenGL core profile extensions:
> OpenGL version string: 3.1 Mesa 20.0.4
> OpenGL shading language version string: 1.40
> OpenGL context flags: (none)
> OpenGL extensions:
> OpenGL ES profile version string: OpenGL ES 3.1 Mesa 20.0.4
> OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.10
> OpenGL ES profile extensions:
> And vainfo:
> vainfo: VA-API version: 1.7 (libva 2.6.0)
> vainfo: Driver version: Mesa Gallium driver 20.0.4 for AMD ARUBA (DRM
> 2.50.0 / 5.5.10-artix1-1, LLVM 9.0.1)
> vainfo: Supported profile and entrypoints
>       VAProfileMPEG2Simple            : VAEntrypointVLD
>       VAProfileMPEG2Main              : VAEntrypointVLD
>       VAProfileVC1Simple              : VAEntrypointVLD
>       VAProfileVC1Main                : VAEntrypointVLD
>       VAProfileVC1Advanced            : VAEntrypointVLD
>       VAProfileH264ConstrainedBaseline: VAEntrypointVLD
>       VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
>       VAProfileH264Main               : VAEntrypointVLD
>       VAProfileH264Main               : VAEntrypointEncSlice
>       VAProfileH264High               : VAEntrypointVLD
>       VAProfileH264High               : VAEntrypointEncSlice
>       VAProfileNone                   : VAEntrypointVideoProc
>
> I would be really thankful
>
>
Use the baseline profile, set a fixed bitrate (via -b:v) and explicitly
disable B-frames:

ffmpeg -vaapi_device /dev/dri/renderD128 -f x11grab -video_size 1920x1080
-i :0 -vf 'format=nv12,hwupload' -c:v h264_vaapi -b:v 7M -profile 578 -bf 0
output.mp4

The Mesa driver's VAAPI bits on AMD GPUs can be quite limited, and you may
have better results with their Vulkan-based AMF encoder(s).
Try the snippet above and report back.

You may also get profiles such as main and high to work as vainfo claims
support for them, but set them explicitly via -profile:v and retest.