[FFmpeg-devel] [PATCH 00/16] NVidia Tegra hardware decoding backend
averne
averne381 at gmail.com
Thu May 30 22:43:02 EEST 2024
Hi all,
This patch series implements a hardware decoding backend for nvidia
Tegra devices, notably the Nintendo Switch. It was primarily written
for HorizonOS (Nintendo Switch OS), but also supports nvidia's
Linux4Tegra distro. As for hardware, all Tegras later than the X1
(T210) should be supported, although the patch does not implement
features that were added to subsequent revisions of multimedia
engines (eg. 12-bit HEVC). However, since I only own T210 devices
(Switch and jetson nano), I was not able to verify this.
The backend is essentially a userspace NVDEC driver, as due to the
OS design of the Switch, we cannot link to nvidia's system libraries.
It notably uses (sparse) hardware documentation released by nvidia
here: https://github.com/NVIDIA/open-gpu-doc/tree/master/classes/video.
It supports all codecs available in hardware (MPEG1/2/4, VC1, H264,
HEVC, VP8, VP9 and JPEG), with dynamic frequency scaling, and
hardware-accelerated frame transfer.
At the moment I'm submitting the series with some nvidia headers
pulled from various sources, but I do think they should rather be
put in nv-codec-headers, let me know.
The code was tested for memory bugs and leaks with valgrind and
asan on L4T. Some quick performance testing (decoding with -f null -)
showed results in line with official software, tested against the
nvv4l2 backend that was posted here a while ago:
https://lists.ffmpeg.org/pipermail/ffmpeg-devel/2020-June/263759.html.
Note that the numbers are skewed because frame transfer cannot be
disabled in nvidia's backend.
- HEVC Main 10 @ 4k (~80Mbps): nvtegra 79fps, nvv4l2 66fps
- HEVC Main 10 @ 1080p (~5Mbps): nvtegra 402fps, nvv4l2 229fps
- H264 @ 1080p (~3Mbps): nvtegra 286fps, nvv4l2 260fps
Several homebrew applications have been using this backend for some
time, with no bugs reported. As far as I'm aware, this is the
complete list of them:
- NXMP, a media player based on mpv: https://github.com/proconsule/nxmp
- WiliWili, a bilibili client: https://github.com/xfangfang/wiliwili
- Switchfin, a Jellyfin client: https://github.com/dragonflylee/switchfin
- Moonlight-Switch, a Moonlight client: https://github.com/XITRIX/Moonlight-Switch
- chiaki: https://git.sr.ht/~kkwong/chiaki/
- My own media player, unreleased at this time
Nintendo Switch support assumes a working devkitA64 homebrew
environment, instructions regarding setup can be found here:
https://devkitpro.org/wiki/devkitPro_pacman. The hwaccel can then be
configured by eg.:
```
source /opt/devkitpro/switchvars.sh && ./configure
--cross-prefix=aarch64-none-elf- --enable-cross-compile --arch=aarch64
--cpu=cortex-a57 --target-os=horizon --enable-pic --enable-gpl
--enable-nvtegra
```
It should probably be noted that NVDEC usage on discrete gpus is
very similar. As far as I know, the main difference is that the
interfacing is done through the GPFIFO block (same block that
manages the 3D engine), instead of host1x.
Thank you for your consideration.
averne (16):
avutil/buffer: add helper to allocate aligned memory
configure,avutil: add support for HorizonOS
avutil: add ioctl definitions for tegra devices
avutil: add hardware definitions for NVDEC, NVJPG and VIC
avutil: add common code for nvtegra
avutil: add nvtegra hwcontext
hwcontext_nvtegra: add dynamic frequency scaling routines
nvtegra: add common hardware decoding code
nvtegra: add mpeg1/2 hardware decoding
nvtegra: add mpeg4 hardware decoding
nvtegra: add vc1 hardware decoding
nvtegra: add h264 hardware decoding
nvtegra: add hevc hardware decoding
nvtegra: add vp8 hardware decoding
nvtegra: add vp9 hardware decoding
nvtegra: add mjpeg hardware decoding
configure | 30 +
libavcodec/Makefile | 11 +
libavcodec/h263dec.c | 6 +
libavcodec/h264_slice.c | 6 +-
libavcodec/h264dec.c | 3 +
libavcodec/hevcdec.c | 17 +-
libavcodec/hevcdec.h | 2 +
libavcodec/hwaccels.h | 10 +
libavcodec/hwconfig.h | 2 +
libavcodec/mjpegdec.c | 6 +
libavcodec/mpeg12dec.c | 12 +
libavcodec/mpeg4videodec.c | 3 +
libavcodec/nvtegra_decode.c | 517 +++++++++
libavcodec/nvtegra_decode.h | 94 ++
libavcodec/nvtegra_h264.c | 506 +++++++++
libavcodec/nvtegra_hevc.c | 633 +++++++++++
libavcodec/nvtegra_mjpeg.c | 336 ++++++
libavcodec/nvtegra_mpeg12.c | 319 ++++++
libavcodec/nvtegra_mpeg4.c | 344 ++++++
libavcodec/nvtegra_vc1.c | 455 ++++++++
libavcodec/nvtegra_vp8.c | 334 ++++++
libavcodec/nvtegra_vp9.c | 665 ++++++++++++
libavcodec/vc1dec.c | 9 +
libavcodec/vp8.c | 6 +
libavcodec/vp9.c | 10 +-
libavutil/Makefile | 9 +
libavutil/buffer.c | 31 +
libavutil/buffer.h | 7 +
libavutil/clb0b6.h | 303 ++++++
libavutil/clc5b0.h | 436 ++++++++
libavutil/cle7d0.h | 129 +++
libavutil/cpu.c | 7 +
libavutil/hwcontext.c | 4 +
libavutil/hwcontext.h | 1 +
libavutil/hwcontext_internal.h | 1 +
libavutil/hwcontext_nvtegra.c | 1045 ++++++++++++++++++
libavutil/hwcontext_nvtegra.h | 92 ++
libavutil/nvdec_drv.h | 1858 ++++++++++++++++++++++++++++++++
libavutil/nvhost_ioctl.h | 511 +++++++++
libavutil/nvjpg_drv.h | 189 ++++
libavutil/nvmap_ioctl.h | 451 ++++++++
libavutil/nvtegra.c | 1035 ++++++++++++++++++
libavutil/nvtegra.h | 258 +++++
libavutil/nvtegra_host1x.h | 94 ++
libavutil/pixdesc.c | 4 +
libavutil/pixfmt.h | 8 +
libavutil/vic_drv.h | 279 +++++
47 files changed, 11085 insertions(+), 3 deletions(-)
create mode 100644 libavcodec/nvtegra_decode.c
create mode 100644 libavcodec/nvtegra_decode.h
create mode 100644 libavcodec/nvtegra_h264.c
create mode 100644 libavcodec/nvtegra_hevc.c
create mode 100644 libavcodec/nvtegra_mjpeg.c
create mode 100644 libavcodec/nvtegra_mpeg12.c
create mode 100644 libavcodec/nvtegra_mpeg4.c
create mode 100644 libavcodec/nvtegra_vc1.c
create mode 100644 libavcodec/nvtegra_vp8.c
create mode 100644 libavcodec/nvtegra_vp9.c
create mode 100644 libavutil/clb0b6.h
create mode 100644 libavutil/clc5b0.h
create mode 100644 libavutil/cle7d0.h
create mode 100644 libavutil/hwcontext_nvtegra.c
create mode 100644 libavutil/hwcontext_nvtegra.h
create mode 100644 libavutil/nvdec_drv.h
create mode 100644 libavutil/nvhost_ioctl.h
create mode 100644 libavutil/nvjpg_drv.h
create mode 100644 libavutil/nvmap_ioctl.h
create mode 100644 libavutil/nvtegra.c
create mode 100644 libavutil/nvtegra.h
create mode 100644 libavutil/nvtegra_host1x.h
create mode 100644 libavutil/vic_drv.h
--
2.45.1
More information about the ffmpeg-devel
mailing list