[FFmpeg-devel] [PATCH] Check rc_buffer_size value using integer comparison

[Måns Rullgård]

Michael Niedermayer <michaelni at gmx.at> writes:

> On Tue, Sep 07, 2010 at 08:25:31PM +0100, Mans Rullgard wrote:
>> For certain input values, the floating-point value calculated
>> for bit_rate*time_base may, due to rounding, become minutely
>> larger than rc_buffer_size, thus causing the configuration to
>> be rejected in error.  Converting to int before the comparison
>> truncates any such overshoot.
>> 
>> This fixes the mxf regression test with gcc 4.5 on x86_32.
>> ---
>>  libavcodec/mpegvideo_enc.c |    2 +-
>>  1 files changed, 1 insertions(+), 1 deletions(-)
>> 
>> diff --git a/libavcodec/mpegvideo_enc.c b/libavcodec/mpegvideo_enc.c
>> index 9f8682c..f55883b 100644
>> --- a/libavcodec/mpegvideo_enc.c
>> +++ b/libavcodec/mpegvideo_enc.c
>> @@ -351,7 +351,7 @@ av_cold int MPV_encode_init(AVCodecContext *avctx)
>>          av_log(avctx, AV_LOG_INFO, "impossible bitrate constraints, this will fail\n");
>>      }
>>  
>> -    if(avctx->rc_buffer_size && avctx->bit_rate*av_q2d(avctx->time_base) > avctx->rc_buffer_size){
>> +    if(avctx->rc_buffer_size && (int)(avctx->bit_rate*av_q2d(avctx->time_base)) > avctx->rc_buffer_size){
>
> should be
>
> avctx->bit_rate*(int64_t)avctx->time_base.num > avctx->rc_buffer_size * (int64_t)avctx->time_base.den

Even better.  Applied like that.

-- 
M?ns Rullg?rd
mans at mansr.com