[MPlayer-dev-eng] [PATCH] fix aspect ratio calculations broken by new av_cmp_q

[Reimar Döffinger]

On Mon, Oct 18, 2010 at 05:25:41PM -0400, Andrew Wason wrote:
> One of my samples has avctx->sample_aspect_ratio == {8, 9} and since
> ctx->last_sample_aspect_ratio == {0, 1} then av_cmp_q returns 1
> instead of 0 and the video is displayed with the wrong aspect ratio.

Why should it return 0?!?

> Hmm, I can't seem to upload a sample to mplayer incoming, but these
> are the values I'm seeing in gdb for this sample with your patch in
> libmpcodecs/vd_ffmpeg.c:init_vo
> 
> (gdb) p aspect
> $8 = 1.33333337
> (gdb) p sh->aspect
> $9 = 1.77777779
> (gdb) p avctx->sample_aspect_ratio
> $10 = {
>   num = 8,
>   den = 9
> }
> (gdb) p ctx->last_sample_aspect_ratio
> $11 = {
>   num = 0,
>   den = 1
> }
> 
> The video should be displayed with 1.7777 but is using 1.3333
> 
> Attached patch keeps your check for aspect!=0 but keeps my
> initialization of ctx->last_sample_aspect_ratio - this works with all
> the samples I'm testing with.

I'd say what you are seeing is that the bug that your patch adds
(or I guess restores to be pedantic) works in your favour by disabling
aspect updates in some cases (too early to be specific).
Maybe below patch works.
Index: libmpcodecs/vd_ffmpeg.c
===================================================================
--- libmpcodecs/vd_ffmpeg.c     (revision 32508)
+++ libmpcodecs/vd_ffmpeg.c     (working copy)
@@ -272,6 +272,7 @@
     if (!ctx)
         return 0;
     memset(ctx, 0, sizeof(vd_ffmpeg_ctx));
+    ctx->last_sample_aspect_ratio.den = 1;
 
     lavc_codec = avcodec_find_decoder_by_name(sh->codec->dll);
     if(!lavc_codec){
@@ -559,9 +560,8 @@
         // sets the value correctly in avcodec_open.
         set_format_params(avctx, avctx->pix_fmt);
         mp_msg(MSGT_DECVIDEO, MSGL_V, "[ffmpeg] aspect_ratio: %f\n", aspect);
-        if (sh->aspect == 0 ||
-            av_cmp_q(avctx->sample_aspect_ratio,
-                     ctx->last_sample_aspect_ratio))
+        // aspect may have changed even if sample_aspect_ratio is unmodified
+        if (aspect != 0)
             sh->aspect = aspect;
         ctx->last_sample_aspect_ratio = avctx->sample_aspect_ratio;
         sh->disp_w = width;