[MPlayer-dev-eng] recursive make considered harmful (was: Re: The new Makefiles)

[Diego Biurrun]

On Tue, Jun 03, 2008 at 12:17:05AM +0300, Ivan Kalvachev wrote:
> On 6/2/08, Diego Biurrun <diego at biurrun.de> wrote:
> > On Mon, Jun 02, 2008 at 07:07:22PM +0200, Nico Sabbi wrote:
> >> Il Monday 02 June 2008 17:18:49 Diego Biurrun ha scritto:
> >> > I did read your patch when you sent it to me.  However, I do not
> >> > remember every single detail and it does not apply anymore.  By "I'll
> >> > look at your patch" I just meant to say that I will investigate how it
> >> > can be used to fix our linking problems, not that I wanted to look at
> >> > it for the first time.
> >> 
> >> sorry, but I really don't understand the reason for a single and totally
> >> flat
> >> Makefile: IMO a Makefile per directory is a kind of modularization (that
> >> is generally considered "A good thing" ) , clean for its own nature.
> >> Can you explain what we gain now? or in what respect a single Makefile
> >> is cleaner?
> >
> > The single Makefile is shorter than the single-directory Makefiles
> > combined, even with common infrastructure factorized into mpcommon.mak.
> > Our Makefile is just above 1000 lines.  Given that it consists largely
> > of long lists, it is neither complicated nor particularly long.  In such
> > a case I prefer having all the information in one central place.
> >
> > At work I once had the misfortune of having to debug a recursive make
> > system where everything was split into tiny snippets.  I continuously
> > lost track of which part got included from where and after staring at it
> > for a few hours, I rewrote it non-recursively in a fraction of that
> > time.  The end result was a very simple, single Makefile with about 30%
> > of the previous total line count.  It was also faster and correct.
> >
> > So unless you go over a certain threshold I generally prefer 60 lines
> > over 6 x 10 lines.
> >
> > But this is not the main reason.  The problem is that recursive Make is
> > incorrect.  What the Make program does is build a directed acyclic graph
> > of all dependencies/targets and then perform all the necessary steps the
> > target you request requires, but no more than those necessary steps.
> >
> > Recursive uses of make cut this graph into pieces.  Unfortunately this
> > process is not lossless because inter-directory dependencies are left
> > out.  This results in Make systems that continuously recompile things
> > unnecessarily, fail to recompile necessary parts and generally have
> > horrible performance.
> >
> > For a more in-depth explanation, read Peter Miller's short but
> > informative paper "Recursive Make Considered Harmful":
> >
> > http://miller.emu.id.au/pmiller/books/rmch/
> >
> > I hope this clears it up.  If you have more questions, ask, but it's
> > really been explained many times in many places, the best I know being
> > the above-mentioned paper.
> 
> Diego had found some article about good habits and implements it
> without fully understanding it. He was inspired by the FFmpeg change
> in same direction.

I can claim with confidence to understand that paper (I read it twice)
and I have read the GNU Make manual cover to cover.

I understand the MPlayer build system, of which I built the Makefile
single-handedly.  I also understand the FFmpeg build system and wrote
large parts of it.  I also reworked some Make build system structures at
work, making them non-recursive in the process.

I have been planning to make MPlayer's (and FFmpeg's) build system
non-recursive for years and started thinking and working on it long
before Mans sat down to implement it for FFmpeg.

You on the other hand kept asking me on IRC what the problems with
recursive make really were.  You refused to google for the keywords I
gave you.

While I would never dare call myself a Make expert, I surely have solid
knowledge and experience.  To claim that I do not understand these very
simple issues is ridiculous.  Even more so coming from a person like
you, who knows so little about Make as to be oblivious of the pitfalls
of recursive usage.

> Most of the drawbacks of recursive compilation do not apply to MPlayer.

False.

> The main problem solved in the paper is when files that are needed
> for the compilation must be generated beforehand and they have/cause
> dependences from other branches. This in extreme cases requiring
> running make twice in the same directory/project.

False.

> The recursive makefile theoretically should be faster, because running
> new instance of make is slower operation. However it also requires more
> memory as it was exposed by make bug with memory handling that caused
> FFmpeg to require newer version of make.
> The computational slowdown caused by the larger lists could be negated
> from not having to compute same stuff for every makefile instance. So
> it must be benchmarked.

False.

> The biggest problem as it was already demonstrated, is that the
> MPlayer Makefile cannot be fully made non-recursive. We do have
> includes of different projects that are imported into MPlayer and we
> must call them recursively.

False.

Of course I should know better than to discuss with you.  So far this
has always been completely in vain.  I don't expect this to change in
the future.  Unfortunately others might believe parts of what you say if
it remains uncontradicted.

Diego