[FFmpeg-devel] [PATCH 1/2] ffbuild: compose linker response files in a loop

[Andreas Rheinhardt]

Gyan Doshi:
> 
> 
> On 2025-03-29 11:52 pm, Martin Storsjö wrote:
>> On Sat, 29 Mar 2025, Gyan Doshi wrote:
>>
>>>> Did you not try to use GNU make's flie function?
>>>
>>> I just benched this and it ranges from 1m28.093s to 1m29.971s (5%
>>> faster) for the lavc targets.
>>> However, this was added in make 4.0. Are we supporting older make?
>>
>> Yes, we generally do support older GNU make; macOS (even the latest
>> versions) only ships with GNU make 3.81.
>>
>> Regarding measuring the runtime cost of this change; measuring the
>> whole build time is quite uninteresting, the interesting bit is
>> measuring the time to build e.g. an .a library on its own. So after a
>> full build, I do "rm libavcodec/libavcodec.a; time make libavcodec/
>> libavcodec.a". This change raises that time from ~3.5 seconds to ~3.8
>> seconds. However do note that this is on a quite slow system in
>> itself; without the "rm", it takes make 2.3 seconds just to figure out
>> that nothing needs to be done.
>>
>> So on that level, the change indeed is mostly tolerable.
>>
>> However - this is very quick as long as "echo" is a shell builtin. If
>> "echo" turns out to be an external executable instead of the shell
>> builtin (which we can simulate by calling "/usr/bin/echo" instead of
>> "echo"), then this suddenly takes >16 seconds rather than the earlier
>> <4 seconds. And that's quite a steep price to pay.
>>
>> As noted before, this is only a fix for a potential, hypothetical
>> problem. The fix is inexpensive in the case of a builtin echo, where
>> we don't need the fix anyway. For the case of an external echo, where
>> we potentially could need the fix, the fix is quite expensive though.
>>
>> But even with the external /usr/bin/echo (on msys2), I still can
>> produce a very long (>32k) .objs file with only one single invocation
>> of /usr/bin/echo. So we don't actually have this problem even in that
>> case.
>>
>> So given that there are multiple concerns about the performance about
>> this, and the problem that it tries to fix is entirely hypothetical at
>> the moment, I would suggest that we skip this fix for now.
>>
>> If someone actually manages to hit the problem in some setup and can
>> tell us about it, we could reconsider of course.
> 
> Ok, I'll skip the piecewise patch.
> 
> But I'll note that just the linking step in isolation is not the
> relevant benchmark here. Most users who are not doing active ffmpeg
> development are building the whole thing. That means thousands of .o
> files. followed by linking external and internal libs.
> So what they will see with an echo utility is closer to 3m30s vs 3m42s
> than 4s vs 16s, which is a minimal change for someone not iterating app
> development.

Completely wrong: People doing active ffmpeg development matter a lot.
The incremental build is the relevant benchmark.

- Andreas