[MPlayer-dev-eng] recursive make considered harmful (was: Re: The new Makefiles)

[Ivan Kalvachev]

On 6/3/08, Diego Biurrun <diego at biurrun.de> wrote:
> On Tue, Jun 03, 2008 at 12:17:05AM +0300, Ivan Kalvachev wrote:
>> On 6/2/08, Diego Biurrun <diego at biurrun.de> wrote:
>> > On Mon, Jun 02, 2008 at 07:07:22PM +0200, Nico Sabbi wrote:
>> >> Il Monday 02 June 2008 17:18:49 Diego Biurrun ha scritto:
>> >> > I did read your patch when you sent it to me.  However, I do not
>> >> > remember every single detail and it does not apply anymore.  By "I'll
>> >> > look at your patch" I just meant to say that I will investigate how
>> >> > it
>> >> > can be used to fix our linking problems, not that I wanted to look at
>> >> > it for the first time.
>> >>
>> >> sorry, but I really don't understand the reason for a single and
>> >> totally
>> >> flat
>> >> Makefile: IMO a Makefile per directory is a kind of modularization
>> >> (that
>> >> is generally considered "A good thing" ) , clean for its own nature.
>> >> Can you explain what we gain now? or in what respect a single Makefile
>> >> is cleaner?
>> >
>> > The single Makefile is shorter than the single-directory Makefiles
>> > combined, even with common infrastructure factorized into mpcommon.mak.
>> > Our Makefile is just above 1000 lines.  Given that it consists largely
>> > of long lists, it is neither complicated nor particularly long.  In such
>> > a case I prefer having all the information in one central place.
>> >
>> > At work I once had the misfortune of having to debug a recursive make
>> > system where everything was split into tiny snippets.  I continuously
>> > lost track of which part got included from where and after staring at it
>> > for a few hours, I rewrote it non-recursively in a fraction of that
>> > time.  The end result was a very simple, single Makefile with about 30%
>> > of the previous total line count.  It was also faster and correct.
>> >
>> > So unless you go over a certain threshold I generally prefer 60 lines
>> > over 6 x 10 lines.
>> >
>> > But this is not the main reason.  The problem is that recursive Make is
>> > incorrect.  What the Make program does is build a directed acyclic graph
>> > of all dependencies/targets and then perform all the necessary steps the
>> > target you request requires, but no more than those necessary steps.
>> >
>> > Recursive uses of make cut this graph into pieces.  Unfortunately this
>> > process is not lossless because inter-directory dependencies are left
>> > out.  This results in Make systems that continuously recompile things
>> > unnecessarily, fail to recompile necessary parts and generally have
>> > horrible performance.
>> >
>> > For a more in-depth explanation, read Peter Miller's short but
>> > informative paper "Recursive Make Considered Harmful":
>> >
>> > http://miller.emu.id.au/pmiller/books/rmch/
>> >
>> > I hope this clears it up.  If you have more questions, ask, but it's
>> > really been explained many times in many places, the best I know being
>> > the above-mentioned paper.
>>
>> Diego had found some article about good habits and implements it
>> without fully understanding it. He was inspired by the FFmpeg change
>> in same direction.
>
> I can claim with confidence to understand that paper (I read it twice)
> and I have read the GNU Make manual cover to cover.
>
> I understand the MPlayer build system, of which I built the Makefile
> single-handedly.  I also understand the FFmpeg build system and wrote
> large parts of it.  I also reworked some Make build system structures at
> work, making them non-recursive in the process.
>
> I have been planning to make MPlayer's (and FFmpeg's) build system
> non-recursive for years and started thinking and working on it long
> before Mans sat down to implement it for FFmpeg.
>
> You on the other hand kept asking me on IRC what the problems with
> recursive make really were.  You refused to google for the keywords I
> gave you.
>
> While I would never dare call myself a Make expert, I surely have solid
> knowledge and experience.  To claim that I do not understand these very
> simple issues is ridiculous.  Even more so coming from a person like
> you, who knows so little about Make as to be oblivious of the pitfalls
> of recursive usage.
>
>> Most of the drawbacks of recursive compilation do not apply to MPlayer.
>
> False.
>
>> The main problem solved in the paper is when files that are needed
>> for the compilation must be generated beforehand and they have/cause
>> dependences from other branches. This in extreme cases requiring
>> running make twice in the same directory/project.
>
> False.
>
>> The recursive makefile theoretically should be faster, because running
>> new instance of make is slower operation. However it also requires more
>> memory as it was exposed by make bug with memory handling that caused
>> FFmpeg to require newer version of make.
>> The computational slowdown caused by the larger lists could be negated
>> from not having to compute same stuff for every makefile instance. So
>> it must be benchmarked.
>
> False.
>
>> The biggest problem as it was already demonstrated, is that the
>> MPlayer Makefile cannot be fully made non-recursive. We do have
>> includes of different projects that are imported into MPlayer and we
>> must call them recursively.
>
> False.
>
> Of course I should know better than to discuss with you.  So far this
> has always been completely in vain.  I don't expect this to change in
> the future.  Unfortunately others might believe parts of what you say if
> it remains uncontradicted.

False.