[FFmpeg-devel] [PATCH v4 2/2][GSoC 2024] tests/checkasm: Add check_vvc_sad to vvc_mc.c

[Martin Storsjö]

On Tue, 21 May 2024, Rémi Denis-Courmont wrote:

>
>
> Le 21 mai 2024 09:37:18 GMT+03:00, "Martin Storsjö" <martin at martin.st> a écrit :
>> On Tue, 21 May 2024, Rémi Denis-Courmont wrote:
>>
>>> Hi,
>>> 
>>> VVC benchmarks have increased checksam runtime by at least an order of 
>>> magnitude. It's become so prohibitively slow that I could not even get 
>>> to the end.
>>> 
>>> This is not an acceptable situation and impedes non-VVC assembler work
>>
>> I don't quite understand; whenever benchmarking anything in checkasm, I 
>> would always run e.g. "checkasm --test=ac3dsp 
>> --bench=ac3_sum_square_bufferfly_float", limiting the total running of 
>> tests to a specific module, and only benchmarking a subset of the run 
>> functions. (The --bench parameter specifies a prefix; only functions 
>> matching that prefix gets benchmarked.)
>
> Sure that's how you do it when you're working on a specific new 
> optimisation. Now we're trying to compare 128-bit and 256-bit vectors 
> for *all* existing functions to see which ones need to be reworked.
>
> That used to work (in 30 minutes on K230, 5 minutes on Zen 2, IIRC). Now 
> it's effectively broken and that's not acceptable'

Ah, I see. Ok, that's a reasonable thing to do I guess.

(It's of course possible to speed it up further by only testing specific 
--test=foo cases where you know you have riscv assembly worth 
benchmarking, but if it was doable in a tolerable amount of time before, 
that shouldn't be needed.)

>> Without limiting the scope with a --test parameter, checkasm 
>> benchmarking has always been prohibitively slow for me - so I don't 
>> think there's anything new here?
>
> As said, it seems to be literally an order of magnitude slower than 
> before if not worse.
>
>> That said I'm not familiar with the VVC tests in checkasm, perhaps they 
>> benchmark things excessively. But I don't see how that would impede 
>> work on other DSP functions in any way?
>
> James also complained about the same thing before I.

Indeed, the tests in vvc_alf group seem to do excessive benchmarking 
(benchmarking every width/height combination between 4 and 128, in 
increments of 4). I sent a patch to cut this down to a reasonable amount.

Overall, I would expect the vvc checkasm tests to take a notable amount of 
time. Dav1d's checkasm takes twice as long to run as ffmpeg's, and it's 
probably a reasonable to assume that vvc is roughly of the same level of 
complexity as av1, so it's probably expected that ffmpeg's checkasm 
runtime at least doubles, once all vvc routines are integrated in 
checkasm.

But the tests in vvc_alf indeed had an entirely unreasonable amount of 
benchmarking hooked up, and that should indeed be fixed, e.g. with the 
patch I just sent.

// Martin