[FFmpeg-devel] [PATCH] avcodec_default_get_buffer should fail instead of returning nonsense for unsupported formats

[Reimar Döffinger]

Hello,
currently avcodec_default_get_buffer returns complete nonsense when
calling it e.g. to request a VDPAU format.
An obvious solution is to check the return value of ff_fill_pointer,
if it is -1 continuing (and treating -1 as size) obviously makes no
sense.
I suspect that some stuff might need to be freed, e.g.
(*picture_number)++;
undone but I still have hope that there is someone who knows the code
well enough to implement it correctly so I don't have to try to
understand all of it.
This is the change I had in mind and at least fixes a crash if
get_buffer is not overridden but a hardware-accelerated pixfmt selected:
Index: libavcodec/utils.c
===================================================================
--- libavcodec/utils.c  (revision 17530)
+++ libavcodec/utils.c  (working copy)
@@ -271,6 +271,8 @@
         }
 
         tmpsize = ff_fill_pointer(&picture, NULL, s->pix_fmt, h);
+        if (tmpsize < 0)
+            return -1;
 
         for (i=0; i<3 && picture.data[i+1]; i++)
             size[i] = picture.data[i+1] - picture.data[i];



Greetings,
Reimar D?ffinger