[FFmpeg-devel] [PATCH 3/3] tiff: fix overflows when calling av_readuce

[Andreas Cadhalpun]

On 13.12.2016 01:32, Michael Niedermayer wrote:
> On Tue, Dec 13, 2016 at 12:50:19AM +0100, Andreas Cadhalpun wrote:
>> The arguments of av_reduce are signed, so the cast to uint64_t is misleading.
>>
>> Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun at googlemail.com>
>> ---
>>  libavcodec/tiff.c | 11 +++++++++--
>>  1 file changed, 9 insertions(+), 2 deletions(-)
>>
>> diff --git a/libavcodec/tiff.c b/libavcodec/tiff.c
>> index 4721e94..12ef419 100644
>> --- a/libavcodec/tiff.c
>> +++ b/libavcodec/tiff.c
>> @@ -772,9 +772,16 @@ static void set_sar(TiffContext *s, unsigned tag, unsigned num, unsigned den)
>>      int offset = tag == TIFF_YRES ? 2 : 0;
>>      s->res[offset++] = num;
>>      s->res[offset]   = den;
>> -    if (s->res[0] && s->res[1] && s->res[2] && s->res[3])
>> +    if (s->res[0] && s->res[1] && s->res[2] && s->res[3]) {
>> +        uint64_t num = s->res[2] * (uint64_t)s->res[1];
>> +        uint64_t den = s->res[0] * (uint64_t)s->res[3];
>> +        if (num > INT64_MAX || den > INT64_MAX) {
>> +            num = num >> 1;
>> +            den = den >> 1;
>> +        }
> 
> this can make one of them 0, in fact i think even if they arent 0
> the sample_aspect_ratio can be  after reduce
> should they be checked after all that instead of before ?

I've added a check for !s->avctx->sample_aspect_ratio.den after av_reduce.
The check before is still necessary to prevent sample_aspect_ratio from
becoming negative.

Best regards,
Andreas

-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-tiff-fix-overflows-when-calling-av_reduce.patch
Type: text/x-diff
Size: 1582 bytes
Desc: not available
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20161214/5032a923/attachment.patch>