Hi all, so the new flat build system is underway. Sadly IMHO it have a few shortcomings and there is some things that I think we should discuss here before going any further. The most important point IMHO, is do we really get rid of the .a or not? On the plus side I only see a few rule less in the Makefile. On the minus side the whole source list is required for any tool using the lib in question. Then there is the problem that now all sources are grouped together. It completely prevent any sane build system for the tools. Even if we drop the .a we need to keep a source list for each lib. Anything else is just too inflexible. If we keep separate lists we might as well put them in $LIB/Makefile and support building from the lib's dir. The latter is easy to achieve with some include. And still having separate Makefiles would make things easier for newcomers than having to dive in a 5000 lines Makefile. As an example, attached is a Makefile for osdep that can be included from the main Makefile or used "alone". Obviously it's just a draft as we still need to settle the above issues, and perhaps some more. Albeu
On Sat, Apr 12, 2008 at 4:31 PM, Alban Bedel <albeu@free.fr> wrote:
Hi all,
so the new flat build system is underway. Sadly IMHO it have a few shortcomings and there is some things that I think we should discuss here before going any further.
The most important point IMHO, is do we really get rid of the .a or not? On the plus side I only see a few rule less in the Makefile. On the minus side the whole source list is required for any tool using the lib in question.
Then there is the problem that now all sources are grouped together. It completely prevent any sane build system for the tools. Even if we drop the .a we need to keep a source list for each lib. Anything else is just too inflexible.
If we keep separate lists we might as well put them in $LIB/Makefile and support building from the lib's dir. The latter is easy to achieve with some include. And still having separate Makefiles would make things easier for newcomers than having to dive in a 5000 lines Makefile.
As an example, attached is a Makefile for osdep that can be included from the main Makefile or used "alone". Obviously it's just a draft as we still need to settle the above issues, and perhaps some more.
There is something else I wonder about the flat build system. Does it link objects faster that it links static libraries? Does it need less memory for linking?
On Sat, Apr 12, 2008 at 04:20:34PM +0300, Ivan Kalvachev wrote:
There is something else I wonder about the flat build system. Does it link objects faster that it links static libraries?
numbers welcome
Does it need less memory for linking?
Do you mean the linker program during the linking process or the resulting MPlayer binary? Diego
On Sat, Apr 12, 2008 at 03:31:36PM +0200, Alban Bedel wrote:
so the new flat build system is underway. Sadly IMHO it have a few shortcomings and there is some things that I think we should discuss here before going any further.
The most important point IMHO, is do we really get rid of the .a or not? On the plus side I only see a few rule less in the Makefile. On the minus side the whole source list is required for any tool using the lib in question.
We would just need the relevant object files. You do have a point that this can be a burden to maintain. Note that every time I have merged a subdirectory makefile into the top-level one, there was a net reduction in linecount.
Then there is the problem that now all sources are grouped together. It completely prevent any sane build system for the tools. Even if we drop the .a we need to keep a source list for each lib. Anything else is just too inflexible.
If we keep separate lists we might as well put them in $LIB/Makefile and support building from the lib's dir. The latter is easy to achieve with some include. And still having separate Makefiles would make things easier for newcomers than having to dive in a 5000 lines Makefile.
I don't buy the 50 small makefiles are simpler than a big one argument. If you look at the top-level makefile and disregard the cruft that is going to go away when recursive make is eliminated - then it is really quite simple. There are some long list of objects/sources in there, but long lists are not really more complicated than short lists. For FFmpeg it makes sense since some directories are used independent of the whole thing, but for MPlayer I think not. Diego
On Mon, 14 Apr 2008 01:34:26 +0200 Diego Biurrun <diego@biurrun.de> wrote:
On Sat, Apr 12, 2008 at 03:31:36PM +0200, Alban Bedel wrote:
so the new flat build system is underway. Sadly IMHO it have a few shortcomings and there is some things that I think we should discuss here before going any further.
The most important point IMHO, is do we really get rid of the .a or not? On the plus side I only see a few rule less in the Makefile. On the minus side the whole source list is required for any tool using the lib in question.
We would just need the relevant object files. You do have a point that this can be a burden to maintain.
IMHO if we don't have the .a anymore we should have an includeable Makefile with the corresponding file list. Otherwise anybody wanting to use this code (that is us in the case of the tools) must duplicate a good part of our build system, and keep up with any change in it.
Note that every time I have merged a subdirectory makefile into the top-level one, there was a net reduction in linecount.
No doubt there, but line count alone is not everything. A few lines more is well worth it if it cut down on maintenance, or improve reusability.
Then there is the problem that now all sources are grouped together. It completely prevent any sane build system for the tools. Even if we drop the .a we need to keep a source list for each lib. Anything else is just too inflexible.
If we keep separate lists we might as well put them in $LIB/Makefile and support building from the lib's dir. The latter is easy to achieve with some include. And still having separate Makefiles would make things easier for newcomers than having to dive in a 5000 lines Makefile.
I don't buy the 50 small makefiles are simpler than a big one argument. If you look at the top-level makefile and disregard the cruft that is going to go away when recursive make is eliminated - then it is really quite simple. There are some long list of objects/sources in there, but long lists are not really more complicated than short lists.
I agree. However when you compare: SRCS = liba/foo.c \ liba/bar.c \ libb/b.c \ libb/c.c \ main.c \ which is useless for any tool needing only liba, to: SRCS_LIBA = liba/foo.c \ liba/bar.c \ SRCS_LIBB = libb/b.c \ libb/c.c \ SRCS = $(SRCS_LIBA) \ $(SRCS_LIBB) \ main.c \ It's not like the second form make things much more verbose or complex. And such form would also limit the need for "double condition" variables.
For FFmpeg it makes sense since some directories are used independent of the whole thing, but for MPlayer I think not.
I dunno. There is many useful code in MPlayer, in particular stream, libao and libvo have no equivalent with such wide support AFAIK. If using them outside of MPlayer was simpler I'm pretty sure they would see some more use (== more bug fix/improvements for us). Albeu
Ping Diego .... any progress with this? Albeu
On Mon, Apr 14, 2008 at 03:08:49AM +0200, Alban Bedel wrote:
On Mon, 14 Apr 2008 01:34:26 +0200 Diego Biurrun <diego@biurrun.de> wrote:
On Sat, Apr 12, 2008 at 03:31:36PM +0200, Alban Bedel wrote:
so the new flat build system is underway. Sadly IMHO it have a few shortcomings and there is some things that I think we should discuss here before going any further.
The most important point IMHO, is do we really get rid of the .a or not? On the plus side I only see a few rule less in the Makefile. On the minus side the whole source list is required for any tool using the lib in question.
We would just need the relevant object files. You do have a point that this can be a burden to maintain.
IMHO if we don't have the .a anymore we should have an includeable Makefile with the corresponding file list. Otherwise anybody wanting to use this code (that is us in the case of the tools) must duplicate a good part of our build system, and keep up with any change in it.
I see no difference between SRCS = foo.c bar.c baz.c and SRCS += subdir/foo.c subdir/bar.c subdir/baz.c
Then there is the problem that now all sources are grouped together. It completely prevent any sane build system for the tools. Even if we drop the .a we need to keep a source list for each lib. Anything else is just too inflexible.
If we keep separate lists we might as well put them in $LIB/Makefile and support building from the lib's dir. The latter is easy to achieve with some include. And still having separate Makefiles would make things easier for newcomers than having to dive in a 5000 lines Makefile.
I don't buy the 50 small makefiles are simpler than a big one argument. If you look at the top-level makefile and disregard the cruft that is going to go away when recursive make is eliminated - then it is really quite simple. There are some long list of objects/sources in there, but long lists are not really more complicated than short lists.
I agree. However when you compare:
SRCS = liba/foo.c \ liba/bar.c \ libb/b.c \ libb/c.c \ main.c \
which is useless for any tool needing only liba, to:
SRCS_LIBA = liba/foo.c \ liba/bar.c \
SRCS_LIBB = libb/b.c \ libb/c.c \
SRCS = $(SRCS_LIBA) \ $(SRCS_LIBB) \ main.c \
It's not like the second form make things much more verbose or complex. And such form would also limit the need for "double condition" variables.
This is completely orthogonal to the question of using one big Makefile or splitting it up into multiple pieces. Just define a variable and then use it. A subdirectory Makefile basically does just that.
For FFmpeg it makes sense since some directories are used independent of the whole thing, but for MPlayer I think not.
I dunno. There is many useful code in MPlayer, in particular stream, libao and libvo have no equivalent with such wide support AFAIK. If using them outside of MPlayer was simpler I'm pretty sure they would see some more use (== more bug fix/improvements for us).
I don't see a separate Makefile as any sort of simplification for this. Note that the individual Makefiles depend on top-level Makefile snippets as well as configure anyway. Diego
On Fri, 18 Apr 2008 09:54:05 +0200 Diego Biurrun <diego@biurrun.de> wrote:
On Mon, Apr 14, 2008 at 03:08:49AM +0200, Alban Bedel wrote:
On Mon, 14 Apr 2008 01:34:26 +0200 Diego Biurrun <diego@biurrun.de> wrote:
On Sat, Apr 12, 2008 at 03:31:36PM +0200, Alban Bedel wrote:
so the new flat build system is underway. Sadly IMHO it have a few shortcomings and there is some things that I think we should discuss here before going any further.
The most important point IMHO, is do we really get rid of the .a or not? On the plus side I only see a few rule less in the Makefile. On the minus side the whole source list is required for any tool using the lib in question.
We would just need the relevant object files. You do have a point that this can be a burden to maintain.
IMHO if we don't have the .a anymore we should have an includeable Makefile with the corresponding file list. Otherwise anybody wanting to use this code (that is us in the case of the tools) must duplicate a good part of our build system, and keep up with any change in it.
I see no difference between
SRCS = foo.c bar.c baz.c
and
SRCS += subdir/foo.c subdir/bar.c subdir/baz.c
But if a tool need only subdir/* it must recreate that second list. With all the conditional used nowadays in the Makefiles that's far from as trivial as it look here. And what for? To spare _one_ line in the main Makefile.
Then there is the problem that now all sources are grouped together. It completely prevent any sane build system for the tools. Even if we drop the .a we need to keep a source list for each lib. Anything else is just too inflexible.
If we keep separate lists we might as well put them in $LIB/Makefile and support building from the lib's dir. The latter is easy to achieve with some include. And still having separate Makefiles would make things easier for newcomers than having to dive in a 5000 lines Makefile.
I don't buy the 50 small makefiles are simpler than a big one argument. If you look at the top-level makefile and disregard the cruft that is going to go away when recursive make is eliminated - then it is really quite simple. There are some long list of objects/sources in there, but long lists are not really more complicated than short lists.
I agree. However when you compare:
SRCS = liba/foo.c \ liba/bar.c \ libb/b.c \ libb/c.c \ main.c \
which is useless for any tool needing only liba, to:
SRCS_LIBA = liba/foo.c \ liba/bar.c \
SRCS_LIBB = libb/b.c \ libb/c.c \
SRCS = $(SRCS_LIBA) \ $(SRCS_LIBB) \ main.c \
It's not like the second form make things much more verbose or complex. And such form would also limit the need for "double condition" variables.
This is completely orthogonal to the question of using one big Makefile or splitting it up into multiple pieces. Just define a variable and then use it. A subdirectory Makefile basically does just that.
I know this is orthogonal to splitting or not. And I don't care much if it's split or not (although my preference is clear I think). All I want is not to have to duplicate half the (non-trivial to create) file lists when a tool use some of the lib.
For FFmpeg it makes sense since some directories are used independent of the whole thing, but for MPlayer I think not.
I dunno. There is many useful code in MPlayer, in particular stream, libao and libvo have no equivalent with such wide support AFAIK. If using them outside of MPlayer was simpler I'm pretty sure they would see some more use (== more bug fix/improvements for us).
I don't see a separate Makefile as any sort of simplification for this. Note that the individual Makefiles depend on top-level Makefile snippets as well as configure anyway.
Again I don't care much if the Makefiles are split or not. What I want is not to increase the work in maintaining the tools. Before the tools would break only when an _external_ dependency was added to one of the lib (which ideally nearly never happen). Now they will also break anytime a file is added or removed from a lib (which happen way more often). Now, how you can argue that the latter is better is just beyond me. I would suggest you try to merge the tools Makefile next. If you take the approach you suggest you will clearly see that the 3 lines you so bitterly want to spare become a couple dozen more and a clear burden on maintenance. Albeu
On Fri, Apr 18, 2008 at 02:01:02PM +0200, Alban Bedel wrote:
On Fri, 18 Apr 2008 09:54:05 +0200 Diego Biurrun <diego@biurrun.de> wrote:
On Mon, Apr 14, 2008 at 03:08:49AM +0200, Alban Bedel wrote:
On Mon, 14 Apr 2008 01:34:26 +0200 Diego Biurrun <diego@biurrun.de> wrote:
On Sat, Apr 12, 2008 at 03:31:36PM +0200, Alban Bedel wrote:
so the new flat build system is underway. Sadly IMHO it have a few shortcomings and there is some things that I think we should discuss here before going any further.
The most important point IMHO, is do we really get rid of the .a or not? On the plus side I only see a few rule less in the Makefile. On the minus side the whole source list is required for any tool using the lib in question.
We would just need the relevant object files. You do have a point that this can be a burden to maintain.
IMHO if we don't have the .a anymore we should have an includeable Makefile with the corresponding file list. Otherwise anybody wanting to use this code (that is us in the case of the tools) must duplicate a good part of our build system, and keep up with any change in it.
I see no difference between
SRCS = foo.c bar.c baz.c
and
SRCS += subdir/foo.c subdir/bar.c subdir/baz.c
But if a tool need only subdir/* it must recreate that second list. With all the conditional used nowadays in the Makefiles that's far from as trivial as it look here. And what for? To spare _one_ line in the main Makefile.
This is trivially solved by the use of variables.
I would suggest you try to merge the tools Makefile next. If you take the approach you suggest you will clearly see that the 3 lines you so bitterly want to spare become a couple dozen more and a clear burden on maintenance.
Done. Diego
On Sat, Apr 12, 2008 at 03:31:36PM +0200, Alban Bedel wrote:
[...]
OK, so the new Makefile is a reality, it's just below 1000 lines, two thirds of which are variable declarations. All in all I think it is quite simple for what it does. It seemed to me that the problem in this discussion was that we were talking about hypothetical issues without really knowing which would actually appear in the code. This is settled now. TOOLS/netstream and TOOLS/vivodump still do not link under any circumstances. Albeu, you said you knew a way to fix this. Let's hear it. Diego
On Tue, 27 May 2008 14:14:45 +0200 Diego Biurrun <diego@biurrun.de> wrote:
TOOLS/netstream and TOOLS/vivodump still do not link under any circumstances.
Albeu, you said you knew a way to fix this. Let's hear it.
I sent you a patch showing you what is needed to link them, then warned you from the start that the file list _MUST_ be kept slitted. You just ignored everything. Now the work needed is way more important as one must split your monster file list in logical parts :( I still plan to do it, however I'm quiet busy ATM and wont have a flat (again) for some times, so I doubt I'll be able to do it soon. However you have all the information needed in the patch I sent you. Apply the hunks on netstream.c and vivodump.c and all you need to do is fix the makefile. Albeu
On Tue, May 27, 2008 at 07:39:26PM +0200, Alban Bedel wrote:
On Tue, 27 May 2008 14:14:45 +0200 Diego Biurrun <diego@biurrun.de> wrote:
TOOLS/netstream and TOOLS/vivodump still do not link under any circumstances.
Albeu, you said you knew a way to fix this. Let's hear it.
I sent you a patch showing you what is needed to link them, then warned you from the start that the file list _MUST_ be kept slitted. You just ignored everything.
No, I just disagreed that a) splitting the lists is necessary; b) that if a) were true, it would be worth holding back my work on the non-recursive Makefile to fix the linkage of a few obscure tools.
Now the work needed is way more important as one must split your monster file list in logical parts :(
We will see if this is really so hard or if it cannot be done in another way.
However you have all the information needed in the patch I sent you. Apply the hunks on netstream.c and vivodump.c and all you need to do is fix the makefile.
I'll look at your patch if I find the time. Diego
On Tue, 27 May 2008 19:19:25 +0200 Diego Biurrun <diego@biurrun.de> wrote:
On Tue, May 27, 2008 at 07:39:26PM +0200, Alban Bedel wrote:
On Tue, 27 May 2008 14:14:45 +0200 Diego Biurrun <diego@biurrun.de> wrote:
TOOLS/netstream and TOOLS/vivodump still do not link under any circumstances.
Albeu, you said you knew a way to fix this. Let's hear it.
I sent you a patch showing you what is needed to link them, then warned you from the start that the file list _MUST_ be kept slitted. You just ignored everything.
No, I just disagreed that
a) splitting the lists is necessary;
OK, now I'm burning to find out how in hell you will manage to not split the list while not duplicating half of it.
b) that if a) were true, it would be worth holding back my work on the non-recursive Makefile to fix the linkage of a few obscure tools.
I never meant that! The only thing I insisted upon was to have: LIBA_SRC = foo.c bar.c LIBB_SRC = some.c MPLAYER_SRC = $(LIBA_SRC) $(LIBB_SRC) another.c Instead of one gigantic list. How that could in any way stop you from converting the Makefile to non-recursive, I just don't know.
Now the work needed is way more important as one must split your monster file list in logical parts :(
We will see if this is really so hard or if it cannot be done in another way.
Please remember that we don't want to have to maintain several file list.
However you have all the information needed in the patch I sent you. Apply the hunks on netstream.c and vivodump.c and all you need to do is fix the makefile.
I'll look at your patch if I find the time.
If that mean you never even looked at it I'm really disappointed. What's the point of asking my opinion and help if you anyway don't really look at my proposals and keep doing things strictly your way? That's just frustrating and a pure waste of time. And if you just had listened a bit, and instead of a single list simply converted each .a to a variable, which functionally is exactly the same as what you did. The problem would already be solved, and you would have spared quiet some configure changes. Albeu
On Wed, May 28, 2008 at 07:11:48PM +0200, Alban Bedel wrote:
On Tue, 27 May 2008 19:19:25 +0200 Diego Biurrun <diego@biurrun.de> wrote:
On Tue, May 27, 2008 at 07:39:26PM +0200, Alban Bedel wrote:
On Tue, 27 May 2008 14:14:45 +0200 Diego Biurrun <diego@biurrun.de> wrote:
TOOLS/netstream and TOOLS/vivodump still do not link under any circumstances.
Albeu, you said you knew a way to fix this. Let's hear it.
I sent you a patch showing you what is needed to link them, then warned you from the start that the file list _MUST_ be kept slitted. You just ignored everything.
No, I just disagreed that
a) splitting the lists is necessary;
OK, now I'm burning to find out how in hell you will manage to not split the list while not duplicating half of it.
Attached is a hackish patch that makes vivodump link. A big problem is the massive code duplication between mplayer.c and mencoder.c. Better suggestions are welcome.
b) that if a) were true, it would be worth holding back my work on the non-recursive Makefile to fix the linkage of a few obscure tools.
I never meant that! The only thing I insisted upon was to have:
LIBA_SRC = foo.c bar.c LIBB_SRC = some.c
MPLAYER_SRC = $(LIBA_SRC) $(LIBB_SRC) another.c
Instead of one gigantic list. How that could in any way stop you from converting the Makefile to non-recursive, I just don't know.
If this turns out to be necessary, it can still be done without being much extra work.
However you have all the information needed in the patch I sent you. Apply the hunks on netstream.c and vivodump.c and all you need to do is fix the makefile.
I'll look at your patch if I find the time.
If that mean you never even looked at it I'm really disappointed.
I did read your patch when you sent it to me. However, I do not remember every single detail and it does not apply anymore. By "I'll look at your patch" I just meant to say that I will investigate how it can be used to fix our linking problems, not that I wanted to look at it for the first time. Diego
Il Monday 02 June 2008 17:18:49 Diego Biurrun ha scritto:
I did read your patch when you sent it to me. However, I do not remember every single detail and it does not apply anymore. By "I'll look at your patch" I just meant to say that I will investigate how it can be used to fix our linking problems, not that I wanted to look at it for the first time.
Diego
sorry, but I really don't understand the reason for a single and totally flat Makefile: IMO a Makefile per directory is a kind of modularization (that is generally considered "A good thing" ) , clean for its own nature. Can you explain what we gain now? or in what respect a single Makefile is cleaner?
On Mon, Jun 02, 2008 at 07:07:22PM +0200, Nico Sabbi wrote:
Il Monday 02 June 2008 17:18:49 Diego Biurrun ha scritto:
I did read your patch when you sent it to me. However, I do not remember every single detail and it does not apply anymore. By "I'll look at your patch" I just meant to say that I will investigate how it can be used to fix our linking problems, not that I wanted to look at it for the first time.
sorry, but I really don't understand the reason for a single and totally flat Makefile: IMO a Makefile per directory is a kind of modularization (that is generally considered "A good thing" ) , clean for its own nature. Can you explain what we gain now? or in what respect a single Makefile is cleaner?
The single Makefile is shorter than the single-directory Makefiles combined, even with common infrastructure factorized into mpcommon.mak. Our Makefile is just above 1000 lines. Given that it consists largely of long lists, it is neither complicated nor particularly long. In such a case I prefer having all the information in one central place. At work I once had the misfortune of having to debug a recursive make system where everything was split into tiny snippets. I continuously lost track of which part got included from where and after staring at it for a few hours, I rewrote it non-recursively in a fraction of that time. The end result was a very simple, single Makefile with about 30% of the previous total line count. It was also faster and correct. So unless you go over a certain threshold I generally prefer 60 lines over 6 x 10 lines. But this is not the main reason. The problem is that recursive Make is incorrect. What the Make program does is build a directed acyclic graph of all dependencies/targets and then perform all the necessary steps the target you request requires, but no more than those necessary steps. Recursive uses of make cut this graph into pieces. Unfortunately this process is not lossless because inter-directory dependencies are left out. This results in Make systems that continuously recompile things unnecessarily, fail to recompile necessary parts and generally have horrible performance. For a more in-depth explanation, read Peter Miller's short but informative paper "Recursive Make Considered Harmful": http://miller.emu.id.au/pmiller/books/rmch/ I hope this clears it up. If you have more questions, ask, but it's really been explained many times in many places, the best I know being the above-mentioned paper. Diego
On Mon, Jun 02, 2008 at 08:06:26PM +0200, Diego Biurrun wrote:
On Mon, Jun 02, 2008 at 07:07:22PM +0200, Nico Sabbi wrote:
sorry, but I really don't understand the reason for a single and totally flat Makefile: IMO a Makefile per directory is a kind of modularization (that is generally considered "A good thing" ) , clean for its own nature. Can you explain what we gain now? or in what respect a single Makefile is cleaner?
The single Makefile is shorter than the single-directory Makefiles combined, even with common infrastructure factorized into mpcommon.mak. Our Makefile is just above 1000 lines. Given that it consists largely of long lists, it is neither complicated nor particularly long. In such a case I prefer having all the information in one central place.
But this is not the main reason. The problem is that recursive Make is incorrect. What the Make program does is build a directed acyclic graph of all dependencies/targets and then perform all the necessary steps the target you request requires, but no more than those necessary steps.
Recursive uses of make cut this graph into pieces. Unfortunately this process is not lossless because inter-directory dependencies are left out. This results in Make systems that continuously recompile things unnecessarily, fail to recompile necessary parts and generally have horrible performance.
Note that it is possible to split Makefiles into pieces without the build process becoming recursive. You can use the 'include' Makefile directive to incorporate Makefile snippets from subdirectories into the top-level Makefile. It is even possible to build non-recursive build systems with Make where you can invoke Make from subdirectories, like Mans did for FFmpeg. But this does require a considerable amount of Make trickery. For MPlayer I did not see much added benefit to make up for the complication. In FFmpeg it makes sense to build single libraries separately. In MPlayer, you are interested in the binaries, which are built from the top-level. Diego
On 6/2/08, Diego Biurrun <diego@biurrun.de> wrote:
On Mon, Jun 02, 2008 at 07:07:22PM +0200, Nico Sabbi wrote:
Il Monday 02 June 2008 17:18:49 Diego Biurrun ha scritto:
I did read your patch when you sent it to me. However, I do not remember every single detail and it does not apply anymore. By "I'll look at your patch" I just meant to say that I will investigate how it can be used to fix our linking problems, not that I wanted to look at it for the first time.
sorry, but I really don't understand the reason for a single and totally flat Makefile: IMO a Makefile per directory is a kind of modularization (that is generally considered "A good thing" ) , clean for its own nature. Can you explain what we gain now? or in what respect a single Makefile is cleaner?
The single Makefile is shorter than the single-directory Makefiles combined, even with common infrastructure factorized into mpcommon.mak. Our Makefile is just above 1000 lines. Given that it consists largely of long lists, it is neither complicated nor particularly long. In such a case I prefer having all the information in one central place.
At work I once had the misfortune of having to debug a recursive make system where everything was split into tiny snippets. I continuously lost track of which part got included from where and after staring at it for a few hours, I rewrote it non-recursively in a fraction of that time. The end result was a very simple, single Makefile with about 30% of the previous total line count. It was also faster and correct.
So unless you go over a certain threshold I generally prefer 60 lines over 6 x 10 lines.
But this is not the main reason. The problem is that recursive Make is incorrect. What the Make program does is build a directed acyclic graph of all dependencies/targets and then perform all the necessary steps the target you request requires, but no more than those necessary steps.
Recursive uses of make cut this graph into pieces. Unfortunately this process is not lossless because inter-directory dependencies are left out. This results in Make systems that continuously recompile things unnecessarily, fail to recompile necessary parts and generally have horrible performance.
For a more in-depth explanation, read Peter Miller's short but informative paper "Recursive Make Considered Harmful":
http://miller.emu.id.au/pmiller/books/rmch/
I hope this clears it up. If you have more questions, ask, but it's really been explained many times in many places, the best I know being the above-mentioned paper.
There is Bulgarian saying "The frog saw that farrier is shoeing a horse and riced leg to get iron shoe". Diego had found some article about good habits and implements it without fully understanding it. He was inspired by the FFmpeg change in same direction. Most of the drawbacks of recursive compilation do not apply to MPlayer. The main problem solved in the paper is when files that are needed for the compilation must be generated beforehand and they have/cause dependences from other branches. This in extreme cases requiring running make twice in the same directory/project. In MPlayer such cases are 2, mp_msg.h and codecs-conf* . I can't remember any of these causing any major problems. (I'm not sure about vidix it may create some file) The recursive makefile theoretically should be faster, because running new instance of make is slower operation. However it also requires more memory as it was exposed by make bug with memory handling that caused FFmpeg to require newer version of make. The computational slowdown caused by the larger lists could be negated from not having to compute same stuff for every makefile instance. So it must be benchmarked. The paper also suggest that there is nothing bad in including per-directory makefiles.inc files, if developers prefer to do so. The biggest problem as it was already demonstrated, is that the MPlayer Makefile cannot be fully made non-recursive. We do have includes of different projects that are imported into MPlayer and we must call them recursively.
Ivan Kalvachev a écrit :
There is Bulgarian saying "The frog saw that farrier is shoeing a horse and riced leg to get iron shoe".
There is French saying 'Shut the hell up" ... Now what, you need quotations to deprecate Diego's work ?
The biggest problem as it was already demonstrated, is that the MPlayer Makefile cannot be fully made non-recursive. We do have includes of different projects that are imported into MPlayer and we must call them recursively.
No, the biggest problem is your attitude discouraging any positive development. It is always easy to criticize people's work when you don't do anything. For sure, I do have more stuff to complain about on Diego's work than on yours for obvious reasons, I barely ever have seen a commit from you. No, in fact, that's not true, looking at MAINTAINERS file, I do have more stuff to complain about on files you maintain than on the one Diego does. Ben
Benjamin Zores a écrit :
For sure, I do have more stuff to complain about on Diego's work than on yours for obvious reasons, I barely ever have seen a commit from you. No, in fact, that's not true, looking at MAINTAINERS file, I do have more stuff to complain about on files you maintain than on the one Diego does.
And just to sum up, lookign at MAINTAINERS file, you maintain: - XviD codec => we use ffmpeg only and you probably never wrote any code from libxvid - vo_svga.c => you did a rewrote of existing code - vo_xvmc.c => copy/paste of vo_xv.c code, nothing's factorized, full of bugs and crahes with libmenu (but you don't care about as no one ever wrote in rules file that code had to work ...). So you maintain 3 _really_ important files in MPlayer. Also: # svn log -v > commits.txt # grep " iive " commits.txt | wc -l 194 # grep " ben " commits.txt | wc -l 315 # grep " diego " commits.txt | wc -l 4840 Are you really sure you have any right to criticize him ? I certainly do not. The whole point is that your contribution is meaningless by comparison to the pain you've caused. Also, your behavior is pissing me up and it's probably the same for other (silent) devels. You wanna blame Diego because Uoti's account hasn't been removed, that's ridiculous, but that's fine. However please not that if you intend to play this stupid game any longer and if ever Uoti's (or anyone else in fact) account is removed, than I'd like to vote for your own account removal (you can do the same for me, even if it is, I really do not fucking care). Ben
On Mon, Jun 02, 2008 at 11:52:04PM +0200, Benjamin Zores wrote:
Benjamin Zores a écrit :
For sure, I do have more stuff to complain about on Diego's work than on yours for obvious reasons, I barely ever have seen a commit from you. No, in fact, that's not true, looking at MAINTAINERS file, I do have more stuff to complain about on files you maintain than on the one Diego does.
And just to sum up, lookign at MAINTAINERS file, you maintain: - XviD codec => we use ffmpeg only and you probably never wrote any code from libxvid
Correct, v[de]_xvid4.c was written by Edouard Gomez, v[de]_xvid.c by Alban Bedel and Kim Minh Kaplan.
- vo_xvmc.c => copy/paste of vo_xv.c code, nothing's factorized, full of bugs and crahes with libmenu (but you don't care about as no one ever wrote in rules file that code had to work ...).
This is badly maintained. The xvmc_render.h header is misplaced at the top of the source tree and it is duplicated in libavcodec/. The issue has never been adressed although pointed out multiple times. xvmc was never properly hooked up in configure. It defaults to disabled waiting for autodetection to be finished since many years. Documentation describing how to enable it is missing. Diego
On Mon, Jun 02, 2008 at 11:52:04PM +0200, Benjamin Zores wrote:
Benjamin Zores a écrit :
For sure, I do have more stuff to complain about on Diego's work than on yours for obvious reasons, I barely ever have seen a commit from you. No, in fact, that's not true, looking at MAINTAINERS file, I do have more stuff to complain about on files you maintain than on the one Diego does.
And just to sum up, lookign at MAINTAINERS file, you maintain: - XviD codec => we use ffmpeg only and you probably never wrote any code from libxvid - vo_svga.c => you did a rewrote of existing code - vo_xvmc.c => copy/paste of vo_xv.c code, nothing's factorized, full of bugs and crahes with libmenu (but you don't care about as no one ever wrote in rules file that code had to work ...).
So you maintain 3 _really_ important files in MPlayer.
Also: # svn log -v > commits.txt # grep " iive " commits.txt | wc -l 194 # grep " ben " commits.txt | wc -l 315 # grep " diego " commits.txt | wc -l 4840
Are you really sure you have any right to criticize him ?
Article 19. of the universal declaration of human rights by the United Nations Everyone has the right to freedom of opinion and expression; this right includes freedom to hold opinions without interference and to seek, receive and impart information and ideas through any media and regardless of frontiers. ---- Article 10 of the European Convention on Human Rights 1. Everyone has the right to freedom of expression. This right shall include freedom to hold opinions and to receive and impart information and ideas without interference by public authority and regardless of frontiers. This article shall not prevent States from requiring the licensing of broadcasting, television or cinema enterprises. 2. The exercise of these freedoms, since it carries with it duties and responsibilities, may be subject to such formalities, conditions, restrictions or penalties as are prescribed by law and are necessary in a democratic society, in the interests of national security, territorial integrity or public safety, for the prevention of disorder or crime, for the protection of health or morals, for the protection of the reputation or rights of others, for preventing the disclosure of information received in confidence, or for maintaining the authority and impartiality of the judiciary. ---- So while you may belive he does not have the right, frankly his right is even legally enforceable and has constitutional status in many european countries. [This of course does not necccesarily apply to lies, insults and such ... ]
I certainly do not.
You do as well
The whole point is that your contribution is meaningless by comparison to the pain you've caused.
Thats your oppinion and an insult, did iive insult you? He surely did diego and diego him ... I really think we should all try to return to a insult free form of communication, flaming about technical parts is one thing but honestly ... calling fellow developers dog, frog, comparing them to sheep, and telling them to "shut the hell up" is IMHO going too far. I dont really mind it in principle but i have the feeling some people here take it more serious than they should.
Also, your behavior is pissing me up and it's probably the same for other (silent) devels.
You wanna blame Diego because Uoti's account hasn't been removed, that's ridiculous, but that's fine.
Diego is the one "in charge" ATM so he is the one to blame about account manageent issues. Like the finance minister of a country is responsible for the finances ... The difference is mainly that the later likely is somehow democratically elected at least indirectly.
However please not that if you intend to play this stupid game any longer and if ever Uoti's (or anyone else in fact) account is removed, than I'd like to vote for your own account removal (you can do the same for me, even if it is, I really do not fucking care).
Do you have any reasons beyond personal anger to ask for iives account removal? Uoti has broken the rules repeatly, said he doesnt care about the rules, ignored requests from a near unanimous majorty of developers, ... [...] -- Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB I hate to see young programmers poisoned by the kind of thinking Ulrich Drepper puts forward since it is simply too narrow -- Roman Shaposhnik
On Thu, Jun 05, 2008 at 03:48:45AM +0200, Michael Niedermayer wrote:
On Mon, Jun 02, 2008 at 11:52:04PM +0200, Benjamin Zores wrote:
Benjamin Zores a écrit :
For sure, I do have more stuff to complain about on Diego's work than on yours for obvious reasons, I barely ever have seen a commit from you. No, in fact, that's not true, looking at MAINTAINERS file, I do have more stuff to complain about on files you maintain than on the one Diego does.
And just to sum up, lookign at MAINTAINERS file, you maintain: - XviD codec => we use ffmpeg only and you probably never wrote any code from libxvid - vo_svga.c => you did a rewrote of existing code - vo_xvmc.c => copy/paste of vo_xv.c code, nothing's factorized, full of bugs and crahes with libmenu (but you don't care about as no one ever wrote in rules file that code had to work ...).
So you maintain 3 _really_ important files in MPlayer.
Also: # svn log -v > commits.txt # grep " iive " commits.txt | wc -l 194 # grep " ben " commits.txt | wc -l 315 # grep " diego " commits.txt | wc -l 4840
Are you really sure you have any right to criticize him ? I certainly do not. The whole point is that your contribution is meaningless by comparison to the pain you've caused.
Thats your oppinion and an insult, did iive insult you?
Benjamin's opinion is shared by many others (and not just me). As Dominik said, Ivan is out of control. He has contributed nothing but flames in a long time. The little code he contributed over the years has multiple long-standing issues that do not get addressed. It's very sad to have to say this, but indeed his positive contributions are overshadowed by the flaming, trolling and the obstructions he puts in everybody's way.
I really think we should all try to return to a insult free form of communication, flaming about technical parts is one thing but honestly ...
It's interesting that you say that to all people except the main offender.
calling fellow developers dog, frog, comparing them to sheep, and telling them to "shut the hell up" is IMHO going too far.
Telling somebody who steps out of line repeatedly, trolls and seems incapable of civil communication to "shut up", even to "shut the hell up" is perfectly acceptable. It's not like Ivan has responded to softer methods to get this point across in the past. Unless people articulate their opinion about his behavior at some point he will keep believing he does nothing wrong.
However please not that if you intend to play this stupid game any longer and if ever Uoti's (or anyone else in fact) account is removed, than I'd like to vote for your own account removal (you can do the same for me, even if it is, I really do not fucking care).
Do you have any reasons beyond personal anger to ask for iives account removal?
Is that a serious question? Ivan is a destructive force in this project. He is a troll, flames constantly, keeps other people from doing useful work, poisons the atmosphere and contributes nothing to offset this. He reverts other people's commits without prior notice, which threatens to incite commit wars. He never accepts that a discussion has ended or that he has been overruled. You left Ivan out of NUT development yourself, because he was unreasonable. Now you surely do not reserve the right to ask for account suspension just for you and not for Benjamin, do you? Diego
On Thu, Jun 05, 2008 at 10:25:44AM +0200, Diego Biurrun wrote:
On Thu, Jun 05, 2008 at 03:48:45AM +0200, Michael Niedermayer wrote:
On Mon, Jun 02, 2008 at 11:52:04PM +0200, Benjamin Zores wrote:
Benjamin Zores a écrit :
For sure, I do have more stuff to complain about on Diego's work than on yours for obvious reasons, I barely ever have seen a commit from you. No, in fact, that's not true, looking at MAINTAINERS file, I do have more stuff to complain about on files you maintain than on the one Diego does.
And just to sum up, lookign at MAINTAINERS file, you maintain: - XviD codec => we use ffmpeg only and you probably never wrote any code from libxvid - vo_svga.c => you did a rewrote of existing code - vo_xvmc.c => copy/paste of vo_xv.c code, nothing's factorized, full of bugs and crahes with libmenu (but you don't care about as no one ever wrote in rules file that code had to work ...).
So you maintain 3 _really_ important files in MPlayer.
Also: # svn log -v > commits.txt # grep " iive " commits.txt | wc -l 194 # grep " ben " commits.txt | wc -l 315 # grep " diego " commits.txt | wc -l 4840
Are you really sure you have any right to criticize him ? I certainly do not. The whole point is that your contribution is meaningless by comparison to the pain you've caused.
Thats your oppinion and an insult, did iive insult you?
Benjamin's opinion is shared by many others (and not just me). As
We know, the silent majority is always on diegos side ...
Dominik said, Ivan is out of control. He has contributed nothing but flames in a long time. The little code he contributed over the years has multiple long-standing issues that do not get addressed.
It's very sad to have to say this, but indeed his positive contributions are overshadowed by the flaming, trolling and the obstructions he puts in everybody's way.
I really think we should all try to return to a insult free form of communication, flaming about technical parts is one thing but honestly ...
It's interesting that you say that to all people except the main offender.
? I said it to all involved, you, iive and ben at least, surely ben is least involved
calling fellow developers dog, frog, comparing them to sheep, and telling them to "shut the hell up" is IMHO going too far.
Telling somebody who steps out of line repeatedly, trolls and seems incapable of civil communication to "shut up", even to "shut the hell up" is perfectly acceptable.
no it is not.
It's not like Ivan has responded to softer methods to get this point across in the past. Unless people articulate their opinion about his behavior at some point he will keep believing he does nothing wrong.
However please not that if you intend to play this stupid game any longer and if ever Uoti's (or anyone else in fact) account is removed, than I'd like to vote for your own account removal (you can do the same for me, even if it is, I really do not fucking care).
Do you have any reasons beyond personal anger to ask for iives account removal?
Is that a serious question?
Ivan is a destructive force in this project. He is a troll, flames constantly, keeps other people from doing useful work, poisons the atmosphere and contributes nothing to offset this.
s/ivan/uoti/
He reverts other people's commits without prior notice, which threatens to incite commit wars.
You steped on iives personal property (XviD code in mplayer) he chased you off, he went too far onto your property and you chased him back. Both of you fuelled this fight equally. Neither has done anything to deescalate it. And its not a attack by one countered by mere defense by another. Its a consensual fight by 2 stubborn angered people. One who started and the other who instead of just defending his own Went and attacked the others as well ...
He never accepts that a discussion has ended or that he has been overruled.
There never was a discussion, you even said that you prefer if it isnt discussed which spelling is correct. You said you just want it consistant even if wrong. Well if you want your house and the neighbors painted in the same color you have to repaint yours. You cannot repaint the neighbors, also you cannot even force your neighbor to repaint yours. You want consistancy so its yours to achive it within your property ...
You left Ivan out of NUT development yourself, because he was unreasonable.
Now you surely do not reserve the right to ask for account suspension just for you and not for Benjamin, do you?
Everyone can ask for account suspension, but there should be a reason behind the request, personal anger is a weak reason. [...] -- Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB When the tyrant has disposed of foreign enemies by conquest or treaty, and there is nothing more to fear from them, then he is always stirring up some war or other, in order that the people may require a leader. -- Plato
On Thursday, 05 June 2008 at 15:16, Michael Niedermayer wrote:
On Thu, Jun 05, 2008 at 10:25:44AM +0200, Diego Biurrun wrote: [...]
Ivan is a destructive force in this project. He is a troll, flames constantly, keeps other people from doing useful work, poisons the atmosphere and contributes nothing to offset this.
s/ivan/uoti/
Well, yes, except that Uoti does contribute more. Both should learn a lesson in teamwork.
He reverts other people's commits without prior notice, which threatens to incite commit wars.
You steped on iives personal property (XviD code in mplayer) he chased you off, he went too far onto your property and you chased him back.
I think it is petty and childish to make a big mess out of someone making spelling changes in files maintained by someone else. [...]
Everyone can ask for account suspension, but there should be a reason behind the request, personal anger is a weak reason.
I'd much rather resolve this without anyone losing their write access. Regards, R. -- MPlayer http://mplayerhq.hu | Livna http://rpm.livna.org There should be a science of discontent. People need hard times and oppression to develop psychic muscles. -- from "Collected Sayings of Muad'Dib" by the Princess Irulan
On Thu, Jun 05, 2008 at 04:03:17PM +0200, Dominik 'Rathann' Mierzejewski wrote:
On Thursday, 05 June 2008 at 15:16, Michael Niedermayer wrote:
On Thu, Jun 05, 2008 at 10:25:44AM +0200, Diego Biurrun wrote: [...]
Ivan is a destructive force in this project. He is a troll, flames constantly, keeps other people from doing useful work, poisons the atmosphere and contributes nothing to offset this.
s/ivan/uoti/
Well, yes, except that Uoti does contribute more. Both should learn a lesson in teamwork.
true [...]
[...]
Everyone can ask for account suspension, but there should be a reason behind the request, personal anger is a weak reason.
I'd much rather resolve this without anyone losing their write access.
Me too, but we are trying that since how long? It didnt work ... , why not try some other tactic? Like account suspension? uoti can still send patches, if they are all ok for a while he can have his account back. And if they are not ok they simply wont be applied, no flames, no fights. [...] -- Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB Frequently ignored awnser#1 FFmpeg bugs should be sent to our bugtracker. User questions about the command line tools should be sent to the ffmpeg-user ML. And questions about how to use libav* should be sent to the libav-user ML.
Michael Niedermayer wrote:
Me too, but we are trying that since how long? It didnt work ... , why not try some other tactic? Like account suspension? uoti can still send patches, if they are all ok for a while he can have his account back. And if they are not ok they simply wont be applied, no flames, no fights.
At the same time get iive account locked so he cannot send anything but patches. I got more annoyed at his destructive behavior, than Uoti's constructive misbehavior. lu -- Luca Barbato Gentoo Council Member Gentoo/linux Gentoo/PPC http://dev.gentoo.org/~lu_zero
On Thu, Jun 05, 2008 at 03:16:17PM +0200, Michael Niedermayer wrote:
On Thu, Jun 05, 2008 at 10:25:44AM +0200, Diego Biurrun wrote:
On Thu, Jun 05, 2008 at 03:48:45AM +0200, Michael Niedermayer wrote:
On Mon, Jun 02, 2008 at 11:52:04PM +0200, Benjamin Zores wrote:
Benjamin Zores a écrit :
For sure, I do have more stuff to complain about on Diego's work than on yours for obvious reasons, I barely ever have seen a commit from you. No, in fact, that's not true, looking at MAINTAINERS file, I do have more stuff to complain about on files you maintain than on the one Diego does.
And just to sum up, lookign at MAINTAINERS file, you maintain: - XviD codec => we use ffmpeg only and you probably never wrote any code from libxvid - vo_svga.c => you did a rewrote of existing code - vo_xvmc.c => copy/paste of vo_xv.c code, nothing's factorized, full of bugs and crahes with libmenu (but you don't care about as no one ever wrote in rules file that code had to work ...).
So you maintain 3 _really_ important files in MPlayer.
Also: # svn log -v > commits.txt # grep " iive " commits.txt | wc -l 194 # grep " ben " commits.txt | wc -l 315 # grep " diego " commits.txt | wc -l 4840
Are you really sure you have any right to criticize him ? I certainly do not. The whole point is that your contribution is meaningless by comparison to the pain you've caused.
Thats your oppinion and an insult, did iive insult you?
Benjamin's opinion is shared by many others (and not just me). As
We know, the silent majority is always on diegos side ...
You are the one who keeps talking about silent majorities and you are the one saying it is on your side. Dominik, Benjamin, Luca, Uoti and myself are on the record for saying that Ivan is out of control. That's more than you can claim.
I really think we should all try to return to a insult free form of communication, flaming about technical parts is one thing but honestly ...
It's interesting that you say that to all people except the main offender.
? I said it to all involved, you, iive and ben at least, surely ben is least involved
You never chided Ivan explicitly. Well, I'm glad you at least include him indirectly here.
calling fellow developers dog, frog, comparing them to sheep, and telling them to "shut the hell up" is IMHO going too far.
Telling somebody who steps out of line repeatedly, trolls and seems incapable of civil communication to "shut up", even to "shut the hell up" is perfectly acceptable.
no it is not.
Uh, for somebody who flames as hard as you sometimes do, that sure is strict criterion... Some of those "shut ups" were simply very heart-felt. Well, let's have people respectfully request for Ivan to desist from flaming then :)
He reverts other people's commits without prior notice, which threatens to incite commit wars.
You steped on iives personal property (XviD code in mplayer) he chased you off, he went too far onto your property and you chased him back.
The spelling of xvid in MPlayer is not Ivan's personal property. And you forget that I am the documentation maintainer and spelling maintainer if you wish to call it so. I have done similar changes many times before without ever encountering this kind of resistance. Ivan, you're out of line. You've let your personal feelings cloud your judgement. You've blown this issue out of proportion. Diego has done nothing wrong in this case. He is the maintainer of the documentation, so it is within his compentence to make such changes (that is, keeping spelling consistent).
One who started and the other who instead of just defending his own Went and attacked the others as well ...
I did not touch his precious code. Since you insist on this idea of private property, that should make a great deal of difference to you. I can be reasoned with. Mans and Ramiro did it in FFmpeg, Ivan could have done it in MPlayer. I do not insist on this silly bikeshed issue. Ivan is free to change it to his preferred spelling if he does it right.
He never accepts that a discussion has ended or that he has been overruled.
There never was a discussion,
I'm not talking about this silly xvid bikeshed topic, but in general. Just remember how you left Ivan out of NUT design discussions because he was unreasonable and would not give in. Ever.
you even said that you prefer if it isnt discussed which spelling is correct. You said you just want it consistant even if wrong.
False.
Now you surely do not reserve the right to ask for account suspension just for you and not for Benjamin, do you?
Everyone can ask for account suspension, but there should be a reason behind the request, personal anger is a weak reason.
True. I'm trying to get this message across in the Uoti case. So far with very little success unfortunately. Diego
On Tue, Jun 03, 2008 at 12:17:05AM +0300, Ivan Kalvachev wrote:
On 6/2/08, Diego Biurrun <diego@biurrun.de> wrote:
On Mon, Jun 02, 2008 at 07:07:22PM +0200, Nico Sabbi wrote:
Il Monday 02 June 2008 17:18:49 Diego Biurrun ha scritto:
I did read your patch when you sent it to me. However, I do not remember every single detail and it does not apply anymore. By "I'll look at your patch" I just meant to say that I will investigate how it can be used to fix our linking problems, not that I wanted to look at it for the first time.
sorry, but I really don't understand the reason for a single and totally flat Makefile: IMO a Makefile per directory is a kind of modularization (that is generally considered "A good thing" ) , clean for its own nature. Can you explain what we gain now? or in what respect a single Makefile is cleaner?
The single Makefile is shorter than the single-directory Makefiles combined, even with common infrastructure factorized into mpcommon.mak. Our Makefile is just above 1000 lines. Given that it consists largely of long lists, it is neither complicated nor particularly long. In such a case I prefer having all the information in one central place.
At work I once had the misfortune of having to debug a recursive make system where everything was split into tiny snippets. I continuously lost track of which part got included from where and after staring at it for a few hours, I rewrote it non-recursively in a fraction of that time. The end result was a very simple, single Makefile with about 30% of the previous total line count. It was also faster and correct.
So unless you go over a certain threshold I generally prefer 60 lines over 6 x 10 lines.
But this is not the main reason. The problem is that recursive Make is incorrect. What the Make program does is build a directed acyclic graph of all dependencies/targets and then perform all the necessary steps the target you request requires, but no more than those necessary steps.
Recursive uses of make cut this graph into pieces. Unfortunately this process is not lossless because inter-directory dependencies are left out. This results in Make systems that continuously recompile things unnecessarily, fail to recompile necessary parts and generally have horrible performance.
For a more in-depth explanation, read Peter Miller's short but informative paper "Recursive Make Considered Harmful":
http://miller.emu.id.au/pmiller/books/rmch/
I hope this clears it up. If you have more questions, ask, but it's really been explained many times in many places, the best I know being the above-mentioned paper.
Diego had found some article about good habits and implements it without fully understanding it. He was inspired by the FFmpeg change in same direction.
I can claim with confidence to understand that paper (I read it twice) and I have read the GNU Make manual cover to cover. I understand the MPlayer build system, of which I built the Makefile single-handedly. I also understand the FFmpeg build system and wrote large parts of it. I also reworked some Make build system structures at work, making them non-recursive in the process. I have been planning to make MPlayer's (and FFmpeg's) build system non-recursive for years and started thinking and working on it long before Mans sat down to implement it for FFmpeg. You on the other hand kept asking me on IRC what the problems with recursive make really were. You refused to google for the keywords I gave you. While I would never dare call myself a Make expert, I surely have solid knowledge and experience. To claim that I do not understand these very simple issues is ridiculous. Even more so coming from a person like you, who knows so little about Make as to be oblivious of the pitfalls of recursive usage.
Most of the drawbacks of recursive compilation do not apply to MPlayer.
False.
The main problem solved in the paper is when files that are needed for the compilation must be generated beforehand and they have/cause dependences from other branches. This in extreme cases requiring running make twice in the same directory/project.
False.
The recursive makefile theoretically should be faster, because running new instance of make is slower operation. However it also requires more memory as it was exposed by make bug with memory handling that caused FFmpeg to require newer version of make. The computational slowdown caused by the larger lists could be negated from not having to compute same stuff for every makefile instance. So it must be benchmarked.
False.
The biggest problem as it was already demonstrated, is that the MPlayer Makefile cannot be fully made non-recursive. We do have includes of different projects that are imported into MPlayer and we must call them recursively.
False. Of course I should know better than to discuss with you. So far this has always been completely in vain. I don't expect this to change in the future. Unfortunately others might believe parts of what you say if it remains uncontradicted. Diego
On 6/3/08, Diego Biurrun <diego@biurrun.de> wrote:
On Tue, Jun 03, 2008 at 12:17:05AM +0300, Ivan Kalvachev wrote:
On 6/2/08, Diego Biurrun <diego@biurrun.de> wrote:
On Mon, Jun 02, 2008 at 07:07:22PM +0200, Nico Sabbi wrote:
Il Monday 02 June 2008 17:18:49 Diego Biurrun ha scritto:
I did read your patch when you sent it to me. However, I do not remember every single detail and it does not apply anymore. By "I'll look at your patch" I just meant to say that I will investigate how it can be used to fix our linking problems, not that I wanted to look at it for the first time.
sorry, but I really don't understand the reason for a single and totally flat Makefile: IMO a Makefile per directory is a kind of modularization (that is generally considered "A good thing" ) , clean for its own nature. Can you explain what we gain now? or in what respect a single Makefile is cleaner?
The single Makefile is shorter than the single-directory Makefiles combined, even with common infrastructure factorized into mpcommon.mak. Our Makefile is just above 1000 lines. Given that it consists largely of long lists, it is neither complicated nor particularly long. In such a case I prefer having all the information in one central place.
At work I once had the misfortune of having to debug a recursive make system where everything was split into tiny snippets. I continuously lost track of which part got included from where and after staring at it for a few hours, I rewrote it non-recursively in a fraction of that time. The end result was a very simple, single Makefile with about 30% of the previous total line count. It was also faster and correct.
So unless you go over a certain threshold I generally prefer 60 lines over 6 x 10 lines.
But this is not the main reason. The problem is that recursive Make is incorrect. What the Make program does is build a directed acyclic graph of all dependencies/targets and then perform all the necessary steps the target you request requires, but no more than those necessary steps.
Recursive uses of make cut this graph into pieces. Unfortunately this process is not lossless because inter-directory dependencies are left out. This results in Make systems that continuously recompile things unnecessarily, fail to recompile necessary parts and generally have horrible performance.
For a more in-depth explanation, read Peter Miller's short but informative paper "Recursive Make Considered Harmful":
http://miller.emu.id.au/pmiller/books/rmch/
I hope this clears it up. If you have more questions, ask, but it's really been explained many times in many places, the best I know being the above-mentioned paper.
Diego had found some article about good habits and implements it without fully understanding it. He was inspired by the FFmpeg change in same direction.
I can claim with confidence to understand that paper (I read it twice) and I have read the GNU Make manual cover to cover.
I understand the MPlayer build system, of which I built the Makefile single-handedly. I also understand the FFmpeg build system and wrote large parts of it. I also reworked some Make build system structures at work, making them non-recursive in the process.
I have been planning to make MPlayer's (and FFmpeg's) build system non-recursive for years and started thinking and working on it long before Mans sat down to implement it for FFmpeg.
You on the other hand kept asking me on IRC what the problems with recursive make really were. You refused to google for the keywords I gave you.
While I would never dare call myself a Make expert, I surely have solid knowledge and experience. To claim that I do not understand these very simple issues is ridiculous. Even more so coming from a person like you, who knows so little about Make as to be oblivious of the pitfalls of recursive usage.
Most of the drawbacks of recursive compilation do not apply to MPlayer.
False.
The main problem solved in the paper is when files that are needed for the compilation must be generated beforehand and they have/cause dependences from other branches. This in extreme cases requiring running make twice in the same directory/project.
False.
The recursive makefile theoretically should be faster, because running new instance of make is slower operation. However it also requires more memory as it was exposed by make bug with memory handling that caused FFmpeg to require newer version of make. The computational slowdown caused by the larger lists could be negated from not having to compute same stuff for every makefile instance. So it must be benchmarked.
False.
The biggest problem as it was already demonstrated, is that the MPlayer Makefile cannot be fully made non-recursive. We do have includes of different projects that are imported into MPlayer and we must call them recursively.
False.
Of course I should know better than to discuss with you. So far this has always been completely in vain. I don't expect this to change in the future. Unfortunately others might believe parts of what you say if it remains uncontradicted.
False.
I'd like to point out a serious flaw in Miller's paper. It's somewhat of a strawman argument, as he describes half a dozen ways to do recursive make incorrectly and concludes that recursive make is bad. To do recursive make correctly, you have to invoke make on every dependency that is in another directory. Make either rebuilds the dependency or determines that it doesn't have to be rebuilt. So I maintain recursive make is not incorrect per se. The drawback is a whole lot of makes (not builds, just makes) of the same thing in a single run. And it takes extra rules to make this happen and I've seen plenty of projects that don't have them. It's worth noting that Miller's paper was fairly well known 10 years ago, and I believe the use of recursive make has only increased since then. So he apparently didn't convince a lot of people. I'm not arguing in favor of recursive make for Mplayer. Just offering a rebuttal to the paper for the record.
On Wed, Jun 04, 2008 at 02:37:17AM +0000, Bryan Henderson wrote:
I'd like to point out a serious flaw in Miller's paper. It's somewhat of a strawman argument, as he describes half a dozen ways to do recursive make incorrectly and concludes that recursive make is bad.
It is bad. Why would you want to run make recursively in the first place?
To do recursive make correctly, you have to invoke make on every dependency that is in another directory. Make either rebuilds the dependency or determines that it doesn't have to be rebuilt.
Good luck implementing that correctly. Besides, the multiple invocations will slow things down considerably.
So I maintain recursive make is not incorrect per se.
The drawback is a whole lot of makes (not builds, just makes) of the same thing in a single run. And it takes extra rules to make this happen and I've seen plenty of projects that don't have them.
This would be both a performance and maintenance disaster. You're basically saying that you can either do things correctly or fix a broken strategy with a lot of kludging and then hope for the best.
It's worth noting that Miller's paper was fairly well known 10 years ago, and I believe the use of recursive make has only increased since then. So he apparently didn't convince a lot of people.
It is not quite as well-known as it should be, yes. However, this excuses nothing. People have simply gotten accustomed to running make clean or distclean all the time because they have come to distrust all the broken make systems. It's a real shame. Diego
Diego Biurrun wrote:
It is bad. Why would you want to run make recursively in the first place?
Usually since you partition your problem better and usually because you just depend on getting the subdir built in order to build what is in the parent dir. Having .mk or .inc doing the same makes this issue pointless, beside the fact is nice doing "cd foodir ; make".
It's worth noting that Miller's paper was fairly well known 10 years ago, and I believe the use of recursive make has only increased since then. So he apparently didn't convince a lot of people.
It is not quite as well-known as it should be, yes. However, this excuses nothing. People have simply gotten accustomed to running make clean or distclean all the time because they have come to distrust all the broken make systems. It's a real shame.
Most of the people are accustomed to autotools that got that right (probably just that), many of the people that got the Miller's paper just put it in the "Considered Harmful" basket (is usually round and full of other trash if you are wondering which is it), some found something good in it and applied in real life case with good results to a point. Having a jihad open just because somebody found the paper useful and made a better build system from practices suggested by it isn't the smartest thing to do. Thinking that non-recursive is the holy and only way seems as stupid to me. Backing statements by hard numbers seems too obvious and un-creative than using flawed logic rhetoric? lu -- Luca Barbato Gentoo Council Member Gentoo/linux Gentoo/PPC http://dev.gentoo.org/~lu_zero
On Thu, Jun 05, 2008 at 04:43:00AM +0200, Luca Barbato wrote:
Diego Biurrun wrote:
It is bad. Why would you want to run make recursively in the first place?
Usually since you partition your problem better and usually because you just depend on getting the subdir built in order to build what is in the parent dir. Having .mk or .inc doing the same makes this issue pointless, beside the fact is nice doing "cd foodir ; make".
Just add one cross-dependency and you haven't partitioned your problem better, but broken your graph into non-equivalent pieces instead.
It's worth noting that Miller's paper was fairly well known 10 years ago, and I believe the use of recursive make has only increased since then. So he apparently didn't convince a lot of people.
It is not quite as well-known as it should be, yes. However, this excuses nothing. People have simply gotten accustomed to running make clean or distclean all the time because they have come to distrust all the broken make systems. It's a real shame.
Most of the people are accustomed to autotools that got that right
I have seen so many broken auto* build systems... Diego
Diego Biurrun wrote:
On Thu, Jun 05, 2008 at 04:43:00AM +0200, Luca Barbato wrote:
It is bad. Why would you want to run make recursively in the first place? Usually since you partition your problem better and usually because you just depend on getting the subdir built in order to build what is in the
Diego Biurrun wrote: parent dir. Having .mk or .inc doing the same makes this issue pointless, beside the fact is nice doing "cd foodir ; make".
Just add one cross-dependency and you haven't partitioned your problem better, but broken your graph into non-equivalent pieces instead.
You broke the topology, you may ask yourself if that cross-dep couldn't be remapped otherwise. Having recursive makefiles requires you to keep the paths following the deps topology ^^
I have seen so many broken auto* build systems...
I also see many broken plain makefiles, nonetheless the tools aren't that bad if used correctly. Still I think having the non recursive makefile could be an improvement. lu -- Luca Barbato Gentoo Council Member Gentoo/linux Gentoo/PPC http://dev.gentoo.org/~lu_zero
On Thu, Jun 05, 2008 at 10:30:19AM +0200, Luca Barbato wrote:
Diego Biurrun wrote:
On Thu, Jun 05, 2008 at 04:43:00AM +0200, Luca Barbato wrote:
It is bad. Why would you want to run make recursively in the first place? Usually since you partition your problem better and usually because you just depend on getting the subdir built in order to build what is in the
Diego Biurrun wrote: parent dir. Having .mk or .inc doing the same makes this issue pointless, beside the fact is nice doing "cd foodir ; make".
Just add one cross-dependency and you haven't partitioned your problem better, but broken your graph into non-equivalent pieces instead.
You broke the topology, you may ask yourself if that cross-dep couldn't be remapped otherwise. Having recursive makefiles requires you to keep the paths following the deps topology ^^
It can be remapped manually of course, but that is really Make's job... You should not have to worry about the dependency graph, the tool should do the job for you. Generating the dependency graph and acting according to it is what Make was designed to do. If you cut the graph apart and try to stitch it back together manually, that is an error-prone and maintenance-intensive process. Diego
On 6/4/08, Diego Biurrun <diego@biurrun.de> wrote:
On Wed, Jun 04, 2008 at 02:37:17AM +0000, Bryan Henderson wrote:
I'd like to point out a serious flaw in Miller's paper. It's somewhat of a strawman argument, as he describes half a dozen ways to do recursive make incorrectly and concludes that recursive make is bad.
It is bad.
You missed my point. I did not say recursive make is not bad. I criticized the logic of the paper.
Why would you want to run make recursively in the first place?
I didn't say I (or one) did.
To do recursive make correctly, you have to invoke make on every dependency that is in another directory. Make either rebuilds the dependency or determines that it doesn't have to be rebuilt.
Good luck implementing that correctly.
Doesn't require luck; just skill. I've done it many times, both from scratch and correcting other people's failed (or maybe just approximate) attempts.
You're basically saying that you can either do things correctly or fix a broken strategy with a lot of kludging and then hope for the best.
No, I said you can do it correctly with or without recursive make. Unless by "correct" you mean "the proper way," in which case I didn't state an opinion.
On Thu, Jun 05, 2008 at 03:45:47AM +0000, Bryan Henderson wrote:
On 6/4/08, Diego Biurrun <diego@biurrun.de> wrote:
On Wed, Jun 04, 2008 at 02:37:17AM +0000, Bryan Henderson wrote:
To do recursive make correctly, you have to invoke make on every dependency that is in another directory. Make either rebuilds the dependency or determines that it doesn't have to be rebuilt.
Good luck implementing that correctly.
Doesn't require luck; just skill. I've done it many times, both from scratch and correcting other people's failed (or maybe just approximate) attempts.
And you are sure you got it correct? And it was not more complicated than the non-recursive solution? Because the non-recursive way has been simpler and more correct every time I did it.
You're basically saying that you can either do things correctly or fix a broken strategy with a lot of kludging and then hope for the best.
No, I said you can do it correctly with or without recursive make. Unless by "correct" you mean "the proper way," in which case I didn't state an opinion.
The proper way is non-recursively of course :) Diego
On Tue, Jun 03, 2008 at 02:31:07AM +0200, Diego Biurrun wrote:
On Tue, Jun 03, 2008 at 12:17:05AM +0300, Ivan Kalvachev wrote:
On 6/2/08, Diego Biurrun <diego@biurrun.de> wrote:
On Mon, Jun 02, 2008 at 07:07:22PM +0200, Nico Sabbi wrote:
Il Monday 02 June 2008 17:18:49 Diego Biurrun ha scritto:
I did read your patch when you sent it to me. However, I do not remember every single detail and it does not apply anymore. By "I'll look at your patch" I just meant to say that I will investigate how it can be used to fix our linking problems, not that I wanted to look at it for the first time.
sorry, but I really don't understand the reason for a single and totally flat Makefile: IMO a Makefile per directory is a kind of modularization (that is generally considered "A good thing" ) , clean for its own nature. Can you explain what we gain now? or in what respect a single Makefile is cleaner?
The single Makefile is shorter than the single-directory Makefiles combined, even with common infrastructure factorized into mpcommon.mak. Our Makefile is just above 1000 lines. Given that it consists largely of long lists, it is neither complicated nor particularly long. In such a case I prefer having all the information in one central place.
At work I once had the misfortune of having to debug a recursive make system where everything was split into tiny snippets. I continuously lost track of which part got included from where and after staring at it for a few hours, I rewrote it non-recursively in a fraction of that time. The end result was a very simple, single Makefile with about 30% of the previous total line count. It was also faster and correct.
So unless you go over a certain threshold I generally prefer 60 lines over 6 x 10 lines.
But this is not the main reason. The problem is that recursive Make is incorrect. What the Make program does is build a directed acyclic graph of all dependencies/targets and then perform all the necessary steps the target you request requires, but no more than those necessary steps.
Recursive uses of make cut this graph into pieces. Unfortunately this process is not lossless because inter-directory dependencies are left out. This results in Make systems that continuously recompile things unnecessarily, fail to recompile necessary parts and generally have horrible performance.
For a more in-depth explanation, read Peter Miller's short but informative paper "Recursive Make Considered Harmful":
http://miller.emu.id.au/pmiller/books/rmch/
I hope this clears it up. If you have more questions, ask, but it's really been explained many times in many places, the best I know being the above-mentioned paper.
Diego had found some article about good habits and implements it without fully understanding it. He was inspired by the FFmpeg change in same direction.
I can claim with confidence to understand that paper (I read it twice) and I have read the GNU Make manual cover to cover.
I understand the MPlayer build system, of which I built the Makefile single-handedly. I also understand the FFmpeg build system and wrote large parts of it. I also reworked some Make build system structures at work, making them non-recursive in the process.
I have been planning to make MPlayer's (and FFmpeg's) build system non-recursive for years and started thinking and working on it long before Mans sat down to implement it for FFmpeg.
I just wanted to say that i belive that having the whole dependancy graph available to make is the better thing than the splitted up recursive calling. Though IMHO libmp* should be buildable seperately. It was IIRC the way it was intended. Not that i need either a non recursive make nor seperately build libmp*. [...] -- Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB Complexity theory is the science of finding the exact solution to an approximation. Benchmarking OTOH is finding an approximation of the exact
On Mon, Jun 02, 2008 at 05:18:49PM +0200, Diego Biurrun wrote:
On Wed, May 28, 2008 at 07:11:48PM +0200, Alban Bedel wrote:
On Tue, 27 May 2008 19:19:25 +0200 Diego Biurrun <diego@biurrun.de> wrote:
On Tue, May 27, 2008 at 07:39:26PM +0200, Alban Bedel wrote:
On Tue, 27 May 2008 14:14:45 +0200 Diego Biurrun <diego@biurrun.de> wrote:
TOOLS/netstream and TOOLS/vivodump still do not link under any circumstances.
Albeu, you said you knew a way to fix this. Let's hear it.
I sent you a patch showing you what is needed to link them, then warned you from the start that the file list _MUST_ be kept slitted. You just ignored everything.
No, I just disagreed that
a) splitting the lists is necessary;
OK, now I'm burning to find out how in hell you will manage to not split the list while not duplicating half of it.
Attached is a hackish patch that makes vivodump link. A big problem is the massive code duplication between mplayer.c and mencoder.c. Better suggestions are welcome.
Here is an updated patch for both vivodump and netstream. It's hackish in the sense that it builds a separate object from mplayer.c, but I don't see a better way except for massively refactoring mplayer.c. This would be a considerable undertaking and IMO not worth the trouble. Any objections? Otherwise I will commit. Diego
On Wed, Jun 04, 2008 at 12:13:35PM +0200, Diego Biurrun wrote:
Here is an updated patch for both vivodump and netstream. It's hackish in the sense that it builds a separate object from mplayer.c, but I don't see a better way except for massively refactoring mplayer.c. This would be a considerable undertaking and IMO not worth the trouble.
Any objections? Otherwise I will commit.
Applied. Diego
participants (9)
-
Alban Bedel -
Benjamin Zores -
Bryan Henderson -
Diego Biurrun -
Dominik 'Rathann' Mierzejewski -
Ivan Kalvachev -
Luca Barbato -
Michael Niedermayer -
Nico Sabbi