[FFmpeg-cvslog] avcodec/videotoolboxenc: fix vtctx reset condition

[Hao Guan]

ffmpeg | branch: master | Hao Guan <hguandl at gmail.com> | Fri Jun 21 14:08:20 2024 +0800| [cd2f8a22e94700c68b1de7968df11e8bebfd315b] | committer: Zhao Zhili

avcodec/videotoolboxenc: fix vtctx reset condition

In vtenc_populate_extradata, the cleanup function vtenc_reset should not
be used when no error occurs, otherwise some color information is lost
(#11036).

This patch checks the status code and conducts the correct cleanup.

Signed-off-by: Hao Guan <hguandl at gmail.com>
Signed-off-by: Zhao Zhili <zhilizhao at tencent.com>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=cd2f8a22e94700c68b1de7968df11e8bebfd315b
---

 libavcodec/videotoolboxenc.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/libavcodec/videotoolboxenc.c b/libavcodec/videotoolboxenc.c
index 42065cbcb3..bc505abee4 100644
--- a/libavcodec/videotoolboxenc.c
+++ b/libavcodec/videotoolboxenc.c
@@ -2767,7 +2767,14 @@ static int vtenc_populate_extradata(AVCodecContext   *avctx,
 
 pe_cleanup:
     CVPixelBufferRelease(pix_buf);
-    vtenc_reset(vtctx);
+
+    if (status) {
+        vtenc_reset(vtctx);
+    } else if (vtctx->session) {
+        CFRelease(vtctx->session);
+        vtctx->session = NULL;
+    }
+
     vtctx->frame_ct_out = 0;
 
     av_assert0(status != 0 || (avctx->extradata && avctx->extradata_size > 0));