[FFmpeg-devel] [RFC] Lavfi test system

Stefano Sabatini stefano.sabatini-lala
Sun Dec 13 01:52:41 CET 2009


On date Sunday 2009-11-29 13:15:17 +0100, Michael Niedermayer encoded:
> On Sun, Nov 29, 2009 at 02:05:08AM +0100, Stefano Sabatini wrote:
> [...]
> > seems to depend on the slicification of the image (yes at this point
> > is clear that we need some testing system for spotting all these
> > problems).
> 
> it was clear since years that lavfi tests are required to have a
> useable lavfi, its too complex and easy to break. But ive been
> ignored and didnt had time to write it myself
> 
> it would be nice if some regression tests would be added before
> further lavfi work is done.
> The minimum filter chain to test must contain all filter pairs,
> that makes ~ n^2 filters in it or n^2 tests for n distinct filters.

If we have n^2 tests for each filter, then we end-up with n^3 tests,
which doesn't look something we can manage already with a small number
of filters, that would mean already 343 tests with the currently 7
committed filters.

That's not easy to do it especially if we want to automate things, but
reference files need to be validated manually, and the same for the
options to be used to test the various filters (also a filter may fail
with some option set and not with other ones).

Also every filter addition will increase the number of tests of n^3 -
(n-1)^3, which also is not reasonable.

We already had had the experience of filters which only failed when
used with a particular format and with a special value for slices
height / input size, so I don't think that to design a system to test
all the possible cases would be something achievable.

Also when we consider filters with a more/less than 1 input and 1
output things get even more complicated (for example how to test when
a sink fails?).

A very simple alternative may be to simply implement for each filter a
set of carefully handcrafted tests which replicate all the possible
critical corner cases which may lead to a failure.

> and then each filter should be individually tested against all
> colorspaces.

Not every pixel format will be in general supported, so we may
manually add a list of supported formats. Even better would be to add
some interface to expose the set of supported inputs (though not sure
this is possible, especially considering the scale filter).

The the test may run like this:
$filter=foobar
$filter_opts=bar:baz:blah
for (pix_fmt in $supported_pix_fmts); do
    do_lavfi_unit "$filter-$pix_fmt"  "slicify=20,format=$pix_fmt,$filter=$filter_opts"  
done

For each of these tests we need a reference file. This shouldn't be
too hard to implement, each reference file can be validated by the
tester.

...

And BTW, in any case I believe need a more modular test system, it
should be easy to simply edit some list and test just a subset of the
complete test-set.

Comments are welcome, I'll start to implement part 2 since it looks
the simple one.

Regards.
-- 
FFmpeg = Fundamentalist and Forgiving Mortal Peaceless Eccentric Gadget



More information about the ffmpeg-devel mailing list