[FFmpeg-devel] [PATCH 2/2] avcodec/videotoolboxenc: Check and set hevc profile

[Zhao Zhili]

From: Zhao Zhili <zhilizhao at tencent.com>

1. If user don't specify the profile, set it to main10 when pixel
   format is 10 bits. Before the patch, videotoolbox output main
   profile bitstream with 10 bit input, which can be confusing.
2. Warning when user set profile to main explicitly with 10 bit
   input. It works, but not the best choice.
3. Reject main 10 profile with 8 bit input, it doesn't work.
---
 libavcodec/videotoolboxenc.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/libavcodec/videotoolboxenc.c b/libavcodec/videotoolboxenc.c
index b685bf4cc4..b0e827d14a 100644
--- a/libavcodec/videotoolboxenc.c
+++ b/libavcodec/videotoolboxenc.c
@@ -894,17 +894,35 @@ static bool get_vt_hevc_profile_level(AVCodecContext *avctx,
 {
     VTEncContext *vtctx = avctx->priv_data;
     int profile = vtctx->profile;
+    const AVPixFmtDescriptor *desc = av_pix_fmt_desc_get(
+            avctx->pix_fmt == AV_PIX_FMT_VIDEOTOOLBOX ? avctx->sw_pix_fmt
+                                                      : avctx->pix_fmt);
+    int bit_depth = desc ? desc->comp[0].depth : 0;
 
     *profile_level_val = NULL;
 
     switch (profile) {
         case AV_PROFILE_UNKNOWN:
+            // Set profile automatically if user don't specify
+            if (bit_depth == 10) {
+                *profile_level_val =
+                        compat_keys.kVTProfileLevel_HEVC_Main10_AutoLevel;
+                break;
+            }
             return true;
         case AV_PROFILE_HEVC_MAIN:
+            if (bit_depth > 0 && bit_depth != 8)
+                av_log(avctx, AV_LOG_WARNING,
+                       "main profile with %d bit input\n", bit_depth);
             *profile_level_val =
                 compat_keys.kVTProfileLevel_HEVC_Main_AutoLevel;
             break;
         case AV_PROFILE_HEVC_MAIN_10:
+            if (bit_depth > 0 && bit_depth != 10) {
+                av_log(avctx, AV_LOG_ERROR,
+                       "Invalid main10 profile with %d bit input\n", bit_depth);
+                return false;
+            }
             *profile_level_val =
                 compat_keys.kVTProfileLevel_HEVC_Main10_AutoLevel;
             break;
-- 
2.42.0