[FFmpeg-user] FATE tests: FFV1

[Peter B.]

On 11/03/2013 03:05 AM, Carl Eugen Hoyos wrote:
> Peter B. <pb <at> das-werkstatt.com> writes:
>
>> What is the official procedure for commiting these new tests
> $ git commit -a
> $ git format-patch HEAD^
> Send a mail to ffmpeg-devel

Alright. Thanks!
I'll try that.


> Consider also to send a patch to improve http://ffmpeg.org/developer.html

I was thinking about that already, since it took me some time to dig
myself through the existing tests to reverse-engineer how it could be
done ;)
I'll see what I can do.


>> - including 2 additional files for "fate-suite"?
> Send a mail to Michael explaining how he can get the samples.

Clear.


Open issues on my side:

---------------------------------
1) Most other test-sets are very, very limited:
---------------------------------
I was surprised to see that the tests of other lossless codecs (in
"lossless-video.mak") seem to only test decoding (e.g. Lagarith) for a
handful of samples. It seems that only popular sample use-cases are
tested, or for some (eg ffvhuff), it's even just encoded/decoded once.

I was thinking about also doing only the FATE-tests done in e.g.
Lagarith, but I've already found incompatibilities of FFV1.3 in libav
for certain pix_fmts which would not have been tested if I had sticked
to this very limited testset.


---------------------------------
2) Generate samples on the fly or add them to fate-suite?
---------------------------------
With the help of LCOV, I tried to target as many different use cases as
possible - including also the not-so-popular ones.
Almost all files are now generated, so fate-suite impact is very little,
but this means that the tests add computation time, each time FATE is
ran completely.

Furthermore, I saw that encoding tests (e.g. in "vcodecs.mak") for
lossless tests would actually require to have its own raw source file
for every tested parameter, like pix_fmt or resolution.
Resizing on the fly, or forcing the source to a different pix_fmt
results in "MAXDIFF > 0". If everything's encoded/decoded correctly,
that value should stay constant, but for lossless it'd actually be more
clearer to have "MAXDIFF == 0".


---------------------------------
3) Generating more vsynth variations?
---------------------------------
Right now, I'm using tests/data/vsynth1.yuv as input-source.
If I understood things correctly, vsynth[12].yuv are generated - so by
generating more versions (different pix_fmt, resolution) there could be
a source-video set, which would not have to be re-generated everytime
FATE's running, but also wouldn't clutter fate-suite samples.
What do you think?


Sorry if I'm completely off - and also sorry for all these questions,
but I'm already doing my best to stick to existing FATE tests and trying
to read minds ;)

Pb