[FFmpeg-cvslog] avcodec/osq: avoid using too large numbers for shifts and integers in update_residue_parameter()

[Michael Niedermayer]

ffmpeg | branch: master | Michael Niedermayer <michael at niedermayer.cc> | Fri Sep 15 00:49:41 2023 +0200| [56c334d732dbbce43b0c8fc0809ec545b7946832] | committer: Michael Niedermayer

avcodec/osq: avoid using too large numbers for shifts and integers in update_residue_parameter()

Fixes: 2.96539e+09 is outside the range of representable values of type 'int'
Fixes: Assertion n>=0 && n<=32 failed at libavcodec/get_bits.h:423
Fixes: 62241/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_OSQ_fuzzer-4525761925873664
Fixes: 70406/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_OSQ_fuzzer-6545326804434944

Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=56c334d732dbbce43b0c8fc0809ec545b7946832
---

 libavcodec/osq.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/libavcodec/osq.c b/libavcodec/osq.c
index 1bd4485f07..42a46b25ce 100644
--- a/libavcodec/osq.c
+++ b/libavcodec/osq.c
@@ -161,11 +161,15 @@ static int update_residue_parameter(OSQChannel *cb)
 
     sum = cb->sum;
     x = sum / cb->count;
-    rice_k = av_ceil_log2(x);
+    rice_k = ceil(log2(x));
     if (rice_k >= 30) {
-        rice_k = floor(sum / 1.4426952 + 0.5);
-        if (rice_k < 1)
+        double f = floor(sum / 1.4426952 + 0.5);
+        if (f <= 1) {
             rice_k = 1;
+        } else if (f >= 31) {
+            rice_k = 31;
+        } else
+            rice_k = f;
     }
 
     return rice_k;