[FFmpeg-devel] [RFC] Lavfi test system

[Michael Niedermayer]

On Mon, Mar 08, 2010 at 10:40:02PM +0100, Stefano Sabatini wrote:
> On date Monday 2010-03-08 03:17:11 +0100, Michael Niedermayer encoded:
> > On Mon, Mar 08, 2010 at 02:36:14AM +0100, Stefano Sabatini wrote:
> > > On date Sunday 2010-03-07 17:44:20 +0100, Michael Niedermayer encoded:
> > > > On Sun, Mar 07, 2010 at 04:20:16PM +0100, Stefano Sabatini wrote:
> > > > > On date Sunday 2010-03-07 11:18:33 +0100, Stefano Sabatini encoded:
> > > > > > On date Thursday 2010-01-07 23:36:02 +0100, Stefano Sabatini encoded:
> > > > [...]
> > > > > Index: libavfilter-soc/ffmpeg/tests/lavfi-regression.sh
> > > > > ===================================================================
> > > > > --- libavfilter-soc.orig/ffmpeg/tests/lavfi-regression.sh	2010-03-07 16:15:22.000000000 +0100
> > > > > +++ libavfilter-soc/ffmpeg/tests/lavfi-regression.sh	2010-03-07 16:16:43.000000000 +0100
> > > > > @@ -20,6 +20,8 @@
> > > > >      vfilters=$2
> > > > >  
> > > > >      if [ -n "$test" ] ; then
> > > > > +        slicify_h=$(expr 8 + $(date +%N) % 24)
> > > > > +        vfilters=slicify=$slicify_h,$vfilters
> > > > >          do_video_encoding ${test_name}.nut "" "-vcodec rawvideo -vfilters $vfilters"
> > > > >      fi
> > > > >  }
> > > > 
> > > > this is a problem because its not reproduceable
> > > > 
> > > > an LCG started from a constant would be better
> > > 
> > > I believed the idea was to try different slicing at every run... what
> > > am I missing?
> > 
> > we dont want spurious unreproduceable failures
> > or other way around spurious unreproduceable successes in teh test wen
> > they should fail
> > it makes debuging hard ...
> 
> I agree but this way we are using only a limited subset of slice
> values.

true, could we postpone resolving this though.
having some regression tests is very important
testing all slice sizes is not that important


> 
> Another solution would be to iterate a few times each test with
> different slice height values taken from a pseudo random
> sequence. Would this be OK?

that slows it down proportionally
maybe we could add a "r" parameter to slicify to make it produce
deterministic but random slice sizes that would test more than a
single size for a video


> 
> > > Regards.
> > > -- 
> > > FFmpeg = Frenzy and Fostering Miracolous Problematic Erroneous Guru
> > 
> > >  lavfi-regression.sh |   10 ++++++++++
> > >  1 file changed, 10 insertions(+)
> > > b361118ffebf15ed75e0bac79f4aa27b4d20d371  randomize-lavfi-test-slice-sizes.patch
> > > Index: libavfilter-soc/ffmpeg/tests/lavfi-regression.sh
> > > ===================================================================
> > > --- libavfilter-soc.orig/ffmpeg/tests/lavfi-regression.sh	2010-03-07 16:17:19.000000000 +0100
> > > +++ libavfilter-soc/ffmpeg/tests/lavfi-regression.sh	2010-03-08 02:29:41.000000000 +0100
> > > @@ -14,12 +14,22 @@
> > >  rm -f "$logfile"
> > >  rm -f "$benchfile"
> > >  
> > > +get_lcg()
> > > +{
> > > +    echo "65539 * $1 % (2^31-1)" | bc
> > 
> > where does this generator come from?
> 
> Uh an experiment of mine, I did some little tests on it and it looked
> good enough for this application.

dont try to be smarter than scientists who spend years analyzing the
properties of these generators.


> 
> > and why dont you use the generator we use elsewhere in ffmpeg?
> 
> Do you mean av_lfg_get()? Are you suggesting that I should use a C
> program (lfg_get.c) for generating such a sequence?

use this:
state= state*1664525+1013904223

and dont forget to use *24 >>32, do not use %24


>  
> > also what should be used as random value is the scaled down value not the
> > modulo something value like you do later.
> 
> I have to say that I lack the understanding of the theory behind,
> please if you can give a very short explanation, I'm looking at the
> code using av_lfg_get() and most of the times a modulo operation is
> performed.

just use a proofen standard generator dont invent your own, someone
knuth? said something like, random functions are poor random number genrators
this one hits the key issue ...

If you want to design a better LCG, look at some prng test programs like
the diehard tests or TestU01
(note some of the soucre code [maybe debian package?] of diehard was in a
 very sorry state you wont get that one working correctly if its still in
 that condition)

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Why not whip the teacher when the pupil misbehaves? -- Diogenes of Sinope
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20100308/5fa49f38/attachment.pgp>