[FFmpeg-devel] [PATCH] fate.sh: Allow overriding what targets to make for running the tests

[Alexander Strasser]

Hi Martin,

patch looks technically good to me.

On 2023-11-27 17:46 +0200, Rémi Denis-Courmont wrote:
> Le maanantaina 27. marraskuuta 2023, 14.31.18 EET Martin Storsjö a écrit :
> > This can be useful if doing testing of uncommon CPU extensions by
> > running tests with QEMU (by configuring with e.g.
> > "target_exec=qemu-aarch64"), by only running the checkasm tests,
> > to get a reasonable test coverage without excessive test runtime.
>
> For the purpose of testing future or bleeding-edge CPU extensions on emulator,
> you would normally want to be able to actually filter those in. That is more of
> a matter of patching checkasm than FATE.
>
> Considering the poor coverage of checkasm, I fear that this just gives the
> wrong impression, not to say a false sense of security. It feels misleading to
> encourage or support that paradigm into FATE, in light of that poor coverage.
> Afterall, if it's just about running checkasm, anybody can just run
> `make tests/checkasm/checkasm && tests/checkasm/checkasm`.

Agree, the practice should not be encouraged over cases where it is
necessary to be practical, but having fate clients run the tests and
submit the results seems still worthwhile to me.

We shouldn't get a false sense of security and it can be a problem.
Improving checkasm and qemu and maybe getting actual bleeding edge
hardware into fate could help with that. Not saying it is easy or
effortless...

Still having tests run and some failures detected seems better
than nothing to me. It would be only problematic if the false
positives or negatives weigh out the cases where it is helpful.

@Remi: Please take no offense. This does not look like a black
and white issue to me , thus I wanted to voice what I thought
when initially saw the patch and your response.


Best regards,
  Alexander


> Either way, this feels like a case of cart before horse.
>
> Also FWIW, RV broke due to misaligned accesses and illegal vector types that
> QEMU tolerated. That is rather an argument against QEMU than against this MR
> but still.