[FFmpeg-devel] stsz overflow

[Frank Barchard]

On Mon, Aug 24, 2009 at 4:08 PM, Alex Converse <alex.converse at gmail.com>wrote:
>
> The intermediate product here is the part that overflows. A final
> num_bytes calculated with appropriate intermediate precision should
> fit in in an unsigned 32-bit integer. Why not just fix that rather
> than reduce the number of entries supported?


Alex,
Sorry, thats not going true overflows, where the final num_bytes is >
MAX_INT
Also this expression will overflow.
init_get_bits(&gb, buf, 8*num_bytes);

This patch uses uint64_t to avoid math overflow, but checks the size before
attempting the av_malloc()

Index: libavformat/mov.c
===================================================================
--- libavformat/mov.c   (revision 19695)
+++ libavformat/mov.c   (working copy)
@@ -1224,6 +1224,7 @@
     unsigned int i, entries, sample_size, field_size, num_bytes;
     GetBitContext gb;
     unsigned char* buf;
+    uint64_t size;

     if (c->fc->nb_streams < 1)
         return 0;
@@ -1258,12 +1259,15 @@

     if(entries >= UINT_MAX / sizeof(int))
         return -1;
+    size= ((uint64_t)entries*field_size+4)>>3;
+    if(size*8 > INT_MAX)
+        return -1;
+    num_bytes = (unsigned int)size;
+
     sc->sample_sizes = av_malloc(entries * sizeof(int));
     if (!sc->sample_sizes)
         return AVERROR(ENOMEM);

-    num_bytes = (entries*field_size+4)>>3;
-
     buf = av_malloc(num_bytes+FF_INPUT_BUFFER_PADDING_SIZE);
     if (!buf) {
         av_freep(&sc->sample_sizes);