[FFmpeg-cvslog] avcodec/pnmenc: Check av_image_get_buffer_size()

[Andreas Rheinhardt]

ffmpeg | branch: master | Andreas Rheinhardt <andreas.rheinhardt at outlook.com> | Sat Nov 19 02:03:05 2022 +0100| [715bf3509a96ee679449c0ba479b500fa646c217] | committer: Andreas Rheinhardt

avcodec/pnmenc: Check av_image_get_buffer_size()

Fixes the crash in ticket #10050.
Also ensure that we don't overflow before ff_get_encode_buffer().

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt at outlook.com>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=715bf3509a96ee679449c0ba479b500fa646c217
---

 libavcodec/pnmenc.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/libavcodec/pnmenc.c b/libavcodec/pnmenc.c
index 9eb663306d..c998dd410c 100644
--- a/libavcodec/pnmenc.c
+++ b/libavcodec/pnmenc.c
@@ -42,7 +42,10 @@ static int pnm_encode_frame(AVCodecContext *avctx, AVPacket *pkt,
     int size = av_image_get_buffer_size(avctx->pix_fmt,
                                         avctx->width, avctx->height, 1);
 
-    if ((ret = ff_get_encode_buffer(avctx, pkt, size + 200, 0)) < 0)
+    if (size < 0)
+        return size;
+
+    if ((ret = ff_get_encode_buffer(avctx, pkt, size + 200U, 0)) < 0)
         return ret;
 
     bytestream_start =