[FFmpeg-devel] [PATCH] avcodec/nvenc: fix NULL pointer dereference

[Timo Rothenpieler]

On 28.02.2023 19:43, Zhao Zhili wrote:
> From: Zhao Zhili <zhilizhao at tencent.com>
> 
> This can happen if encoder init failed before setup the queue, then
> reorder_queue_flush() is called by close().
> 
> Fix ticket #10221
> 
> Signed-off-by: Zhao Zhili <zhilizhao at tencent.com>
> ---
>   libavcodec/nvenc.c | 2 ++
>   1 file changed, 2 insertions(+)
> 
> diff --git a/libavcodec/nvenc.c b/libavcodec/nvenc.c
> index 8a28454042..dda2dc3ba1 100644
> --- a/libavcodec/nvenc.c
> +++ b/libavcodec/nvenc.c
> @@ -178,6 +178,8 @@ static void reorder_queue_flush(AVFifo *queue)
>   {
>       FrameData fd;
>   
> +    if (!queue)
> +        return;
>       while (av_fifo_read(queue, &fd, 1) >= 0)
>           av_buffer_unref(&fd.frame_opaque_ref);
>   }

This should probably be an assert0, and the function should not be 
called to begin with if the queue is NULL.