[FFmpeg-devel] Adding FATE tests for FFV1 - revisited

[Peter B.]

On 08/31/2015 01:44 PM, Tobias Rapp wrote:
> On 30.08.2015 21:32, Michael Niedermayer wrote:
>>>      - Target "fate-vsynth%-*" tests default to sws_flags
>>> "accurate_rnd+bitexact". FFV1.3 tests have "neighbor+bitexact". Why?
>>
>> it makes more cases lossless IIRC
>> the default upscaling + default downscaling is not binary identical
>
> Indeed. See
> http://ffmpeg.org/pipermail/ffmpeg-devel/2015-June/174430.html and
> http://ffmpeg.org/pipermail/ffmpeg-devel/2015-June/174447.html for
> reference.

All clear now.
Thanks :)

You also mentioned the stddev not being 0 for RGB. I've also noticed
that yesterday.
In my previous FFV1 tests, I'm using the file generated by ENC,
therefore the pix_fmt of input/output matches and so the framecrc is
happy :)

// ------------------------------
fate-ffv1-enc-v3-bgr0:           ENCOPTS = -level 3 -g 1 -coder 0
-context 0 -slices 24 -slicecrc 0 -pix_fmt bgr0 $(FLAGS_FFV1_V3)
fate-ffv1-dec-v3-bgr0:           $(CMD = framecrc -i
$(DEC_SRC)/ffv1-enc-v3-bgr0.avi) fate-ffv1-enc-v3-bgr0
// ------------------------------

Would that be okay?


>>> 4) Add default argument "-g 1"
>
> This removes the test for the default GOP size.

True.
Will keep that in mind.


> I agree that having a full test suite for FFV1 would be a nice thing
> but am not sure if integration into FATE is the right framework for
> it. As far as I understand FATE tests need a good balance between
> processing time and code coverage.

I also understood it that way.
Wouldn't want to overdo it.
I'm not completely sure which things should definitely be tested in
order to avoid regressions in not-so-common cases?
What I've done previously is, to vary the coder/context/slices
parameters for different pix_fmts. So the pix_fmts are tested, but at
the same time coder/context/slices variations, too.

I've added that since we've seen bugs with e.g. certain slice numbers
during FFV1.3 development testing.


>
> Maybe it can be part of the IETF standardization task to create a FFV1
> test environment which tests every
> color-space/bit-depth/subsampling/... combination?

Definitely a good idea.
However, it still doesn't serve the purpose of being a FATE test to
avoid regressions.
Additionally, I'm also planning to submit the final FFV1 tests to LibAV,
so that LibAV stays compatible with changes/improvements implemented in
FFmpeg.


Regards,
Pb