[FFmpeg-cvslog] avcodec/x86/mpegvideodsp: Fix signedness bug in need_emu

[Michael Niedermayer]

ffmpeg | branch: release/3.0 | Michael Niedermayer <michael at niedermayer.cc> | Mon Nov 13 20:47:48 2017 +0100| [b8a6b5602762e540fbac982cbc3689fa0d2768de] | committer: Michael Niedermayer

avcodec/x86/mpegvideodsp: Fix signedness bug in need_emu

Fixes: out of array read
Fixes: 3516/attachment-311488.dat

Found-by: Insu Yun, Georgia Tech.
Tested-by: wuninsu at gmail.com
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>
(cherry picked from commit 58cf31cee7a456057f337b3102a03206d833d5e8)
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=b8a6b5602762e540fbac982cbc3689fa0d2768de
---

 libavcodec/x86/mpegvideodsp.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/libavcodec/x86/mpegvideodsp.c b/libavcodec/x86/mpegvideodsp.c
index 941a8e2e4c..5dcfd76a61 100644
--- a/libavcodec/x86/mpegvideodsp.c
+++ b/libavcodec/x86/mpegvideodsp.c
@@ -53,8 +53,9 @@ static void gmc_mmx(uint8_t *dst, uint8_t *src,
     const int dyh = (dyy - (1 << (16 + shift))) * (h - 1);
     const int dxh = dxy * (h - 1);
     const int dyw = dyx * (w - 1);
-    int need_emu  =  (unsigned) ix >= width  - w ||
-                     (unsigned) iy >= height - h;
+    int need_emu  =  (unsigned) ix >= width  - w || width < w ||
+                     (unsigned) iy >= height - h || height< h
+                     ;
 
     if ( // non-constant fullpel offset (3% of blocks)
         ((ox ^ (ox + dxw)) | (ox ^ (ox + dxh)) | (ox ^ (ox + dxw + dxh)) |