[MPlayer-dev-eng] [PATCH] fix aspect ratio calculations broken by new av_cmp_q

[Reimar Döffinger]

On Mon, Oct 18, 2010 at 03:11:44PM -0400, Andrew Wason wrote:
> av_cmp_q() changed in r25338/25340 ffmpeg - it can now return INT_MIN
> if one of the rationals is 0/0
> 
> av_cmp_q used to return:
>  * @return 0 if a==b, 1 if a>b and -1 if a<b
> but now returns:
>  * @return 0 if a==b, 1 if a>b, -1 if a<b, and INT_MIN if one of the
>  * values is of the form 0/0
> 
> So statements like "if (av_cmp_q(a,b))" behave differently now if one
> rational is 0/0
> 
> This breaks the display aspect ratio in mplayer for a number of
> samples because libmpcodecs/vd_ffmpeg.c initializes
> last_sample_aspect_ratio to 0/0 and we end up resetting sh->aspect to
> 0 in libmpcodecs/vd_ffmpeg.c:init_vo() when last_sample_aspect_ratio
> is used in av_cmp_q
> 
> Attached patch initializes last_sample_aspect_ratio before using it to
> avoid this.

I can see two issues in the current code, but I think your patch rather
adds a third actually.
Please check if below patch seems ok.
Index: libmpcodecs/vd_ffmpeg.c
===================================================================
--- libmpcodecs/vd_ffmpeg.c     (revision 32508)
+++ libmpcodecs/vd_ffmpeg.c     (working copy)
@@ -272,6 +272,7 @@
     if (!ctx)
         return 0;
     memset(ctx, 0, sizeof(vd_ffmpeg_ctx));
+    ctx->last_sample_aspect_ratio.den = 1;
 
     lavc_codec = avcodec_find_decoder_by_name(sh->codec->dll);
     if(!lavc_codec){
@@ -560,8 +561,9 @@
         set_format_params(avctx, avctx->pix_fmt);
         mp_msg(MSGT_DECVIDEO, MSGL_V, "[ffmpeg] aspect_ratio: %f\n", aspect);
         if (sh->aspect == 0 ||
+            (aspect != 0 &&
             av_cmp_q(avctx->sample_aspect_ratio,
-                     ctx->last_sample_aspect_ratio))
+                     ctx->last_sample_aspect_ratio)))
             sh->aspect = aspect;
         ctx->last_sample_aspect_ratio = avctx->sample_aspect_ratio;
         sh->disp_w = width;