[FFmpeg-devel] [PATCH v2 0/3] Make fate tests succeed with zlib-ng

[Leo Izen]

On 12/16/24 7:20 AM, Michael Niedermayer wrote:
> Hi
> 
> On Sat, Dec 14, 2024 at 11:09:00AM +0100, Anton Khirnov wrote:
>> Quoting Alexander Strasser via ffmpeg-devel (2024-12-01 21:13:56)
>>> This is a fixed up version of the series I sent before.
>>>
>>> This worked for me on Ubuntu 20.04 but probably will break
>>> with older zlib versions as Hendrik pointed out in the
>>> previous thread. Either we must update zlib on affected
>>> FATE clients or add more .alt files to them as well.
>>>
>>> We could also go the further the "no_file_checksums" as
>>> was demonstrated by James' series.
>>>
>>> I still prefer this way because it's simpler and retains
>>> the value of the tests.
>>
> 
> [...]
> 
> 
>>
>> We should not be testing for bitexact output from code that is not under
>> our control and does not guarantee bitexactness.
> 
> This is a odd statement, why is there
> "code that is not under our control" as a condition ?
> 

As far as I understand the issue in this case is that we do not have an 
internal zlib encoder, we just call zlib to encode for example PNG 
files. Even if two different versions/editions/etc. of zlib produce 
correct DEFLATEd blobs that decode to the same bits, that doesn't mean 
the blobs themselves are guaranteed to match bit-for-bit.

However, we're comparing the checksums of what are essentially gzipped 
files, which can be thrown off by differences in the zlib encoder. Those 
minor differences are outside of our control as it's an external library.

Essentially, we shouldn't be having FATE failures depend on the specific 
behavior of whatever zlib we link to, provided it's correct. DEFLATE 
promises certain things about its encodes, but one of the things DEFLATE 
does not promise is that the same uncompressed data always encodes to 
the same compressed data. When we check the CRC of e.g. PNG files, we 
erroneously assume this, and we just basically never got called on it 
because nearly every system uses a recent version of stock zlib.

Anton is basically saying that if we want to compare the CRC of zlib 
output, trying to compile a list of different zlib versions and making 
reference files of each one doesn't really solve the underlying problem, 
and it's kind of hacky.

Solutions I can think of to this problem that solve it forever would be to
(a): not compare the checksums of encoded files with zlib chunks
OR
(b): have some sort of tool that decodes relevant zlib chunks and then 
run a checksum on its output

However, (b) is much more complicated and probably not worth the effort.

- Leo Izen (Traneptora)