[FFmpeg-devel] [PATCH] avfiltergraph: fix format selection.

[Nicolas George]

The old code had two bugs:

For audio filters, the format was not set.

For video filters, if several links reference the same format list,
the same format must be selected in the end. This is done by
setting formats->format_count to 1: the other links sharing
the reference will therefore have only one choice.
If the heuristic does not pick the first format, the selected format
must also be moved to the first position.

Signed-off-by: Nicolas George <nicolas.george at normalesup.org>
---
 libavfilter/avfiltergraph.c |    6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)


The bug was introduced by the following commit:

0d115d7 avfiltergraph: More advanced heuristic to select colorspace.

It can be experienced using the following command:

./ffmpeg_g -vcodec pgmyuv -i tests/vsynth1/00.pgm \
  -vf format=yuv444p10,scale,crop,scale,format=yuv444p10 -f null -

Notice how the scale filters just before and just after crop do not have
matching colorspace: the first converts to rgb48be and the second from
yuv420p.

Regards,

-- 
  Nicolas George


diff --git a/libavfilter/avfiltergraph.c b/libavfilter/avfiltergraph.c
index ef597dd..ba4db6e 100644
--- a/libavfilter/avfiltergraph.c
+++ b/libavfilter/avfiltergraph.c
@@ -268,12 +268,12 @@ static void pick_format(AVFilterLink *link, AVFilterLink *ref)
                 enum PixelFormat p = link->in_formats->formats[i];
                 best= avcodec_find_best_pix_fmt2(best, p, ref->format, has_alpha, NULL);
             }
-            link->format = best;
-        }else
-            link->format = link->in_formats->formats[0];
+            link->in_formats->formats[0] = best;
+        }
     }
 
     link->in_formats->format_count = 1;
+    link->format = link->in_formats->formats[0];
     avfilter_formats_unref(&link->in_formats);
     avfilter_formats_unref(&link->out_formats);
 
-- 
1.7.2.5