[FFmpeg-devel] [PATCH 1/2] ffbuild: compose linker response files in a loop

[Ramiro Polla]

On Sun, Mar 30, 2025 at 8:53 AM Gyan Doshi <ffmpeg at gyani.pro> wrote:
> On 2025-03-30 11:57 am, Andreas Rheinhardt wrote:
> > Gyan Doshi:
> >> On 2025-03-29 11:52 pm, Martin Storsjö wrote:
> >>> On Sat, 29 Mar 2025, Gyan Doshi wrote:
> >>>
> >>>>> Did you not try to use GNU make's flie function?
> >>>> I just benched this and it ranges from 1m28.093s to 1m29.971s (5%
> >>>> faster) for the lavc targets.
> >>>> However, this was added in make 4.0. Are we supporting older make?
> >>> Yes, we generally do support older GNU make; macOS (even the latest
> >>> versions) only ships with GNU make 3.81.
> >>>
> >>> Regarding measuring the runtime cost of this change; measuring the
> >>> whole build time is quite uninteresting, the interesting bit is
> >>> measuring the time to build e.g. an .a library on its own. So after a
> >>> full build, I do "rm libavcodec/libavcodec.a; time make libavcodec/
> >>> libavcodec.a". This change raises that time from ~3.5 seconds to ~3.8
> >>> seconds. However do note that this is on a quite slow system in
> >>> itself; without the "rm", it takes make 2.3 seconds just to figure out
> >>> that nothing needs to be done.
> >>>
> >>> So on that level, the change indeed is mostly tolerable.
> >>>
> >>> However - this is very quick as long as "echo" is a shell builtin. If
> >>> "echo" turns out to be an external executable instead of the shell
> >>> builtin (which we can simulate by calling "/usr/bin/echo" instead of
> >>> "echo"), then this suddenly takes >16 seconds rather than the earlier
> >>> <4 seconds. And that's quite a steep price to pay.
> >>>
> >>> As noted before, this is only a fix for a potential, hypothetical
> >>> problem. The fix is inexpensive in the case of a builtin echo, where
> >>> we don't need the fix anyway. For the case of an external echo, where
> >>> we potentially could need the fix, the fix is quite expensive though.
> >>>
> >>> But even with the external /usr/bin/echo (on msys2), I still can
> >>> produce a very long (>32k) .objs file with only one single invocation
> >>> of /usr/bin/echo. So we don't actually have this problem even in that
> >>> case.
> >>>
> >>> So given that there are multiple concerns about the performance about
> >>> this, and the problem that it tries to fix is entirely hypothetical at
> >>> the moment, I would suggest that we skip this fix for now.
> >>>
> >>> If someone actually manages to hit the problem in some setup and can
> >>> tell us about it, we could reconsider of course.
> >> Ok, I'll skip the piecewise patch.
> >>
> >> But I'll note that just the linking step in isolation is not the
> >> relevant benchmark here. Most users who are not doing active ffmpeg
> >> development are building the whole thing. That means thousands of .o
> >> files. followed by linking external and internal libs.
> >> So what they will see with an echo utility is closer to 3m30s vs 3m42s
> >> than 4s vs 16s, which is a minimal change for someone not iterating app
> >> development.

> > Completely wrong: People doing active ffmpeg development matter a lot.
> > The incremental build is the relevant benchmark.

Agreed.

> People doing active ffmpeg development i.e. those working on git master,
> should have modern shells with builtin echo
> or can opt to disable response files thus avoiding the issue altogether.

> The primary beneficiaries of response files are
> users of build scripts or binary providers like myself adding dozens of
> libraries.

Back when I used to do this on a daily basis, I found that it was
easier and faster to install virtualbox, install a linux distro, and
cross-compile to windows than it was to build everything natively. I
haven't tried building with msvc but I would not be surprised if it
was still easier and faster to cross-compile using wine.

Ramiro