[FFmpeg-devel] Adding FATE tests for FFV1 - revisited

[Tobias Rapp]

On 30.08.2015 21:32, Michael Niedermayer wrote:
> On Sun, Aug 30, 2015 at 07:06:44PM +0200, Peter B. wrote:
>> Hello,
>>
>> I've been working on FATE tests for FFV1 in the past already [1]. My
>> tests didn't work on all platforms and therefore never made it upstream.
>> I think it's better if I try to provide these new tests in smaller
>> chunks now :)
>>
>>
>> First of all, there are things I find inconsistent or confusing with the
>> current tests (vcodec.mak):
>>
>>      - ENCOPTS for FFV1.3 contain "-vcodec ffv1" instead of "CODEC=ffv1"
>> (this generates "-c ffv1.3" as parameter?)
>
>>      - Target "fate-vsynth%-*" tests default to sws_flags
>> "accurate_rnd+bitexact". FFV1.3 tests have "neighbor+bitexact". Why?
>
> it makes more cases lossless IIRC
> the default upscaling + default downscaling is not binary identical

Indeed. See 
http://ffmpeg.org/pipermail/ffmpeg-devel/2015-June/174430.html and 
http://ffmpeg.org/pipermail/ffmpeg-devel/2015-June/174447.html for 
reference.

>
>
>>      - ENCOPTS for "fate-vsynth%-ffv1" are "-slices 4", which is an
>> FFV1.3-only option.

I assume the FFV1 level/subversion is auto-negotiated if some explicit 
"-level" option is missing (e.g. see libavcodec/ffv1enc.c line 680).

>>      - What is "ffv1.0"?
>>
>>
>> My ideas/plans would be something like this:
>>
>> First steps:
>> 1) Clean the current FFV1 tests (naming, ENCDEC options, etc)
>> 2) Move FFV1 tests to its own file (ffv1.mak). Or at least to
>> "lossless-video.mak".
>> 3) Have separate tests for different FFV1 versions (1,3)
>>
>> Then:
>> 4) Add default argument "-g 1"

This removes the test for the default GOP size.

>> 5) Add tests to cover the following cases:
>>      - Color spaces YUV, RGB, GRAY
>>      - bits-per-component as currently supported
>>      - YUV subsampling 420, 422, 444
>>      - Alpha channel: YUVA, BGRA
>>
>> Maybe:
>> 6) Multiple coder/context options
>> 7) Multiple slices options
>> 8) Testing SliceCRC
>>
>>
>> This will produce quite a number of tests :(
>
>> I guess it is desired to keep the number of tests as low as necessary?
>
> avoiding redundant tests would be a good idea
>
>
>> I've attached my old test Makefile (ffv1.mak), for reference.
>> :D
>>
>>
>> What is the best way to proceed?
>
> probably, send patches
> and probably better few and small ones at once then wait and see
> in which direction reviewes go before spending too much time in some
> specific direction
>
> [...]

I agree that having a full test suite for FFV1 would be a nice thing but 
am not sure if integration into FATE is the right framework for it. As 
far as I understand FATE tests need a good balance between processing 
time and code coverage.

Maybe it can be part of the IETF standardization task to create a FFV1 
test environment which tests every color-space/bit-depth/subsampling/... 
combination?

Just my thoughts,
Tobias