[FFmpeg-devel] [PATCH] fate.sh: Allow overriding what targets to make for running the tests

[Martin Storsjö]

On Mon, 27 Nov 2023, Rémi Denis-Courmont wrote:

> Le maanantaina 27. marraskuuta 2023, 14.31.18 EET Martin Storsjö a écrit :
>> This can be useful if doing testing of uncommon CPU extensions by
>> running tests with QEMU (by configuring with e.g.
>> "target_exec=qemu-aarch64"), by only running the checkasm tests,
>> to get a reasonable test coverage without excessive test runtime.
>
> For the purpose of testing future or bleeding-edge CPU extensions on 
> emulator, you would normally want to be able to actually filter those 
> in. That is more of a matter of patching checkasm than FATE.

Sorry, can you elaborate on what you mean with "filter those in" here?

> Considering the poor coverage of checkasm, I fear that this just gives 
> the wrong impression, not to say a false sense of security. It feels 
> misleading to encourage or support that paradigm into FATE, in light of 
> that poor coverage. Afterall, if it's just about running checkasm, 
> anybody can just run `make tests/checkasm/checkasm && 
> tests/checkasm/checkasm`.

Yes, anybody can run that - but having those results posted continuously 
somewhere where other can see them can be valuable as well.

Anyway, the concrete case I'm considering, is that we've got AArch64 code 
merged, that uses the I8MM extensions. We don't have any FATE 
configuration that continuously test that. Whenever there are patches, I 
do spin up a cloud instance that supports this extension and test the 
patches there, but inbetween that we're pretty much blind.

While checkasm's coverage isn't fantastic, for this particular case I'm 
not merging any AArch64 code for new extensions unless that code is 
covered by checkasm.

The other AArch64 feature that we do have code for, which also is 
untested, is the assembly support for branch protection and pointer 
authentication. Also this is testable pretty easily with QEMU. It's of 
course more interesting to run the full fate suite, but if we're not 
looking for bugs in the compiler but only for bugs in our assembly, then 
checkasm should cover most of it.

Yes there's potential for QEMU bugs hiding real issues, but I'd rather 
have a regular run of QEMU+checkasm than not have it tested at all. And 
I'm volunteering HW+time for testing these cases with QEMU for whatever 
checkasm covers, but I'm not volunteering it for full fate runs with QEMU.

And sure, I can just run such configs privately, as I already do run a 
bunch of various regular builds for projects I care about - but as we do 
have FATE with the public status page, hooking it up to be reported there 
would feel like added value for everybody.

// Martin