[FFmpeg-user] Poor quality HEVC from nvenc

[Elliott Balsley]

I am experimenting with hevc_nvenc and I am having trouble getting a high quality image with VBR mode. I tried giving it a super high bitrate of 100Mbps but this only encodes 29Mbps, which makes me think I am missing some important setting. How can I make it use more bits, and are there any other parameters I should try for better quality? I’m using Tesla T4. I tried B frame refs instead of weighted prediction but that didn’t make any difference and I believe you can’t use both at the same time.
I’ve tried several different combinations of nvenc parameters, but I am not sure if I need to also adjust any ffmpeg options too. Here’s an example command. I can post before/after images if it helps. The main problem is blocky artifacts in the darker areas.

    $ ffmpeg -y -i ArriHD_short.mov -pix_fmt yuv444p16le -an -c:v hevc_nvenc -preset p7 -tune hq -profile:v main10 -rc vbr -rc-lookahead 24 -weighted_pred 1 -spatial-aq 1 -b:v 100M aq-weigh.mkv
    ffmpeg version N-100390-ge0e9c94c2b Copyright (c) 2000-2020 the FFmpeg developers
      built with gcc 7 (GCC)
      configuration: --enable-gpl --enable-nonfree --enable-nvenc --enable-libx264 --enable-libx265 --enable-libfdk_aac --enable-libopus --enable-libnpp --enable-cuda-nvcc --extra-cflags=-I/usr/local/cuda-10.2/targets/x86_64-linux/include --extra-ldflags=-L/usr/local/cuda-10.2/targets/x86_64-linux/lib
      libavutil      56. 62.100 / 56. 62.100
      libavcodec     58.115.102 / 58.115.102
      libavformat    58. 65.100 / 58. 65.100
      libavdevice    58. 11.103 / 58. 11.103
      libavfilter     7. 92.100 /  7. 92.100
      libswscale      5.  8.100 /  5.  8.100
      libswresample   3.  8.100 /  3.  8.100
      libpostproc    55.  8.100 / 55.  8.100
    Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'ArriHD_short.mov':
      Metadata:
        major_brand     : qt
        minor_version   : 537199360
        compatible_brands: qt
        creation_time   : 2020-12-10T22:41:31.000000Z
      Duration: 00:00:20.25, start: 0.000000, bitrate: 175506 kb/s
        Stream #0:0(eng): Video: prores (HQ) (apch / 0x68637061), yuv422p10le(tv, bt709, progressive), 1920x1080, 173968 kb/s, SAR 1:1 DAR 16:9, 24 fps, 24 tbr, 24 tbn, 24 tbc (default)
        Metadata:
          creation_time   : 2020-12-10T22:41:31.000000Z
          handler_name    : Apple Video Media Handler
          vendor_id       : appl
          encoder         : Apple ProRes 422 HQ
          timecode        : 00:00:12:21
        Stream #0:1(eng): Audio: pcm_s16le (sowt / 0x74776F73), 48000 Hz, stereo, s16, 1536 kb/s (default)
        Metadata:
          creation_time   : 2020-12-10T22:41:31.000000Z
          handler_name    : Apple Sound Media Handler
          vendor_id       : [0][0][0][0]
        Stream #0:2(eng): Data: none (tmcd / 0x64636D74), 0 kb/s (default)
        Metadata:
          creation_time   : 2020-12-10T22:41:31.000000Z
          handler_name    : Time Code Media Handler
          reel_name       : 01_NAB2017_ARRI_Camera_Showreel_Part01_UHD_WL_HDR10_24fps_stereo24bit
          timecode        : 00:00:12:21
    Stream mapping:
      Stream #0:0 -> #0:0 (prores (native) -> hevc (hevc_nvenc))
    Press [q] to stop, [?] for help
    Output #0, matroska, to 'aq-weigh.mkv':
      Metadata:
        major_brand     : qt
        minor_version   : 537199360
        compatible_brands: qt
        encoder         : Lavf58.65.100
        Stream #0:0(eng): Video: hevc (Rext), yuv444p16le(tv, bt709, progressive), 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 100000 kb/s, 24 fps, 1k tbn (default)
        Metadata:
          creation_time   : 2020-12-10T22:41:31.000000Z
          handler_name    : Apple Video Media Handler
          vendor_id       : appl
          timecode        : 00:00:12:21
          encoder         : Lavc58.115.102 hevc_nvenc
        Side data:
          cpb: bitrate max/min/avg: 0/0/100000000 buffer size: 200000000 vbv_delay: N/A
    frame=  486 fps= 88 q=14.0 Lsize=   72130kB time=00:00:20.20 bitrate=29238.8kbits/s speed=3.64x
    video:72125kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.007138%