[FFmpeg-cvslog] avcodec/smcenc: make sure ny/nx are >= 0

[Paul B Mahol]

ffmpeg | branch: master | Paul B Mahol <onemda at gmail.com> | Tue Jun 18 09:41:37 2024 +0200| [c22488f718f2d51a24b432927399b596fb1eca91] | committer: Michael Niedermayer

avcodec/smcenc: make sure ny/nx are >= 0

(cherry picked from commit 5ad38785e7ad4067a288e9d5e8ce2c4ed2bf584a)

Fixes: out of array read
Fixes: 68939/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_SMC_fuzzer-587804104884224

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=c22488f718f2d51a24b432927399b596fb1eca91
---

 libavcodec/smcenc.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/libavcodec/smcenc.c b/libavcodec/smcenc.c
index 789aef4f77..f8a3322bb1 100644
--- a/libavcodec/smcenc.c
+++ b/libavcodec/smcenc.c
@@ -184,8 +184,8 @@ static void smc_encode_stream(SMCContext *s, const AVFrame *frame,
             const ptrdiff_t offset = xpixel_ptr - src_pixels;
             const int sy = offset / stride;
             const int sx = offset % stride;
-            const int ny = sx < 4 ? sy - 4 : sy;
-            const int nx = sx < 4 ? width - 4 + (width & 3) : sx - 4;
+            const int ny = sx < 4 ? FFMAX(sy - 4, 0) : sy;
+            const int nx = sx < 4 ? FFMAX(width - 4 + (width & 3), 0) : sx - 4;
             const uint8_t *old_pixel_ptr = src_pixels + nx + ny * stride;
             int compare = 0;