[FFmpeg-cvslog] h264: vdpau: fix crash with unsupported colorspace

[Uoti Urpala]

ffmpeg | branch: master | Uoti Urpala <uau at glyph.nonexistent.invalid> | Sat Aug 18 16:11:45 2012 +0200| [e70730045a22d053a64473df20886edfa4775e59] | committer: Michael Niedermayer

h264: vdpau: fix crash with unsupported colorspace

The h264_vdpau decoder crashed if output colorspace was not 8-bit 420.
Add a check to error out instead (current hardware does not support
other colorspaces, so successful decoding is not possible).

Check implemented at a different place by michael, thus blame for bugs goes to michael
Signed-off-by: Michael Niedermayer <michaelni at gmx.at>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=e70730045a22d053a64473df20886edfa4775e59
---

 libavcodec/h264.c |    9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/libavcodec/h264.c b/libavcodec/h264.c
index c07d10a..42de033 100644
--- a/libavcodec/h264.c
+++ b/libavcodec/h264.c
@@ -2494,6 +2494,15 @@ static int decode_slice_header(H264Context *h, H264Context *h0)
         s->avctx->sample_aspect_ratio = h->sps.sar;
         av_assert0(s->avctx->sample_aspect_ratio.den);
 
+        if (s->avctx->codec->capabilities & CODEC_CAP_HWACCEL_VDPAU
+            && (h->sps.bit_depth_luma != 8 ||
+                h->sps.chroma_format_idc > 1)) {
+            av_log(s->avctx, AV_LOG_ERROR,
+                   "VDPAU decoding does not support video "
+                   "colorspace\n");
+            return -1;
+        }
+
         if (s->avctx->bits_per_raw_sample != h->sps.bit_depth_luma ||
             h->cur_chroma_format_idc != h->sps.chroma_format_idc) {
             if (h->sps.bit_depth_luma >= 8 && h->sps.bit_depth_luma <= 14 && h->sps.bit_depth_luma != 11 && h->sps.bit_depth_luma != 13 &&