[FFmpeg-devel] [RFC] Lavfi test system

[Michael Niedermayer]

On Sun, Dec 13, 2009 at 01:52:41AM +0100, Stefano Sabatini wrote:
> On date Sunday 2009-11-29 13:15:17 +0100, Michael Niedermayer encoded:
> > On Sun, Nov 29, 2009 at 02:05:08AM +0100, Stefano Sabatini wrote:
> > [...]
> > > seems to depend on the slicification of the image (yes at this point
> > > is clear that we need some testing system for spotting all these
> > > problems).
> > 
> > it was clear since years that lavfi tests are required to have a
> > useable lavfi, its too complex and easy to break. But ive been
> > ignored and didnt had time to write it myself
> > 
> > it would be nice if some regression tests would be added before
> > further lavfi work is done.
> > The minimum filter chain to test must contain all filter pairs,
> > that makes ~ n^2 filters in it or n^2 tests for n distinct filters.
> 
> If we have n^2 tests for each filter, then we end-up with n^3 tests,

n tests for each filter makes n^2 which is what i meant


[...]
> We already had had the experience of filters which only failed when
> used with a particular format and with a special value for slices
> height / input size, so I don't think that to design a system to test
> all the possible cases would be something achievable.

we should test a widely spread mix of things ...


> 
> Also when we consider filters with a more/less than 1 input and 1
> output things get even more complicated (for example how to test when
> a sink fails?).

ask me when we have a sink


> 
> A very simple alternative may be to simply implement for each filter a
> set of carefully handcrafted tests which replicate all the possible
> critical corner cases which may lead to a failure.

Iam not sure if this is an alternative. Iam not even sure if what i think
you talk about is what you talk about.
A test filter that would feed another filter with pictures with differnt
sets of permissions, sizes, pix_fmts, slice order and size, interlacing
and progressive, misaligned and aligned data[], ....
is surely usefull, its also easy to test as its output should always be the
same (or very close for differing pix_fmts)
still we need actual tests of filter interactions of actual filter chains,
its not the same thing.


> 
> > and then each filter should be individually tested against all
> > colorspaces.
> 
> Not every pixel format will be in general supported, so we may
> manually add a list of supported formats. Even better would be to add
> some interface to expose the set of supported inputs (though not sure
> this is possible, especially considering the scale filter).

you are searching for query_formats() ?


[...]

> And BTW, in any case I believe need a more modular test system, it
> should be easy to simply edit some list and test just a subset of the
> complete test-set.

yes

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Complexity theory is the science of finding the exact solution to an
approximation. Benchmarking OTOH is finding an approximation of the exact
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20091213/1de2b932/attachment.pgp>