[FFmpeg-devel] Upgrade Trouble

Fri Dec 11 15:08:03 CET 2009

On Fri, Dec 11, 2009 at 09:19:31AM +0100, Reinhard Tartler wrote:
> 
> Michael Niedermayer <michaelni at gmx.at> writes:
> 
> > Are you sure its not worthwhile to fix the linker? This dependancy & conflict
> > dance has to be done each time a project wants to add versioning.
> > The linker should do the following (and it does not)
> > 1. Prefer a symbol with matching version over an unversioned symbol.
> >    Not just pick the first that does not have a mismatching version.
> > 2. do a breadth first search from the object where a symbol is referenced
> >    Not a breadth first search from the application.
> > 3. Not fail assertions.
> 
> As indicated above, I've now done more research on your idea. I've
> talked to a colleague at my department, who has programmed an linker for
> embedded systems, and fetched Levine's excellent book "Linkers and
> Loaders" from the library.
> 
> Let's recapitulate roughly what the runtime linker does at load time:
> Conceptually, the runtime linker builds at bootstrap phase of a program
> a global symbol table containing all symbols of all DSO the application
> is linked to. This symbol table is only conceptually global. In
> practice, there are actually two (one for code, one for data), and there
> are actually tables for each and every library. At bootstrap phase,
> these tables are combined in a linked list. (there are a bit more
> optimizations going on here like lazy lookups for function, but that
> doesn't matter for this discussion, for details please see Levine's
> book, chapter 10.3, "Loading a Dynamically Linked Program" and chapter
> 10.4, "Lazy Procedure Linking with the PLT").
> 
> What you now propose is to introduce "hierarchic symbol tables", as my
> colleague called it. I would rather call that "introducing namespace"
> that are created on load time of a library. This way the runtime linker
> would have to obey the library in which the search for symbol it was
> started.
> 
> I think this is not a good idea for a few reasons:
> 
>  - you could of course change the list in which the symbol tables are
>    linked to a graph to implement the "hierarchic symbol table" conecpt,
>    but this increases the complexity of the runtime linker considerably!

The alternative puts the complexity on all the package maintainers who
maintain libs that can be referenced more than once in an applications
dependancy tree.
I always am in favor of fixing the root cause of problems not to let
everyone have to work around it, that being a O(1) vs. O(N) thing.
But then iam also not really expecting that the linker will be changed,
though it really would be nice if a lib or application could enable
such behavior for its dependancies ...

as is though i must admit i can very well understand that many developers
of many libs just refuse to deal with the issue and leave the troubble
on the distro maintainers. -> IMO people just instinctively feel its not
their job to workaround insane gnu linker behavior

> 
>  - this would probably only work for functions, not for code (would have
>    to think more about this)

iam not sure why you think so ...

> 
>  - it would totally break the existing behavior of LD_PRELOAD and
>    dlopen()-tricks that overshadow existing symbols. A lot of really
>    useful applications rely on this.

i see no reason why it would break LD_PRELOAD
dlopen() of course could not be used to override symbols with the different
search order but one could add a RTLD_PRELOAD to restore this feature
all that being theory of course

> 
> So all in all, I'm now pretty sure that changing the existing linker is
> an option for solving the problem at hand.

was that really what you wanted to write? ;)

also your above arguments only address the second point ive raised,
Point 1 and 3 still stand, i guess you arent arguing that failing with
an assertion is correct behavior?
Or that prefering an unversioned symbols over a versioned one
with matching version is correct?

If one combines these 2 you even end up with a funnier case where a
unversioned symbol is always prefered over the correct versioned one
but the removial of the never used versioned one leads to hard failure
Its really not sane and its also not what the sun linker does if i understood
the things i read correctly (but having no solaris here i cant test it)

Now if  1+3  where fixed without the "hierarchic symbol tables"
as your colleague called my point 2 that still would be a huge simplification
for distro packagers.
The reasons should be obvious

* currently introducing versioning one needs to make sure that no application
  that has been build with the new versioned lib can end up being linked to
  the older lib without versioning but otherwise 100% identical ABI.
  Because ld.so goes bye bye with assert(0) in that case

* The other issue that would be fixed is that if a versioned symbol is
  preferably bound to a symbol with matching version and a unversioned
  symbol is preferably bound to a unversioned symbol.
  Then you could just introduce a new libavutil.50 with versioning enabled
  and thats it, it would work. Now you need a complicated net of conflicts
  to prevent any application to end up with a libavutil with and without
  versioning because the linker can mis-bind the symbols in this case.

> 
> In parallel, I asked the Debian glibc maintainers for their opinion. I
> was pointed to the -Bsymbolic ld flag, which, alters the symbol lookup
> behavior of the runtime linker as well. 
> 
> At first, I was sceptic on that idea, and frankly, I still have to
> actually experiment with this approach and to think more about this if
> it would really helps to fix the problem.  On the second thought,
> *might* indeed be an alternative to symbol versioning.  However, it
> would require to create static shared libraries, e.g. to link libavutil
> statically into the dynamically linked libavcodec and libavformat, and
> libavcodec statically in the dynamically linked libavformat.  It would
> effectively prevent to LD_PRELOAD existing symbols, e.g. in libavutil. I
> guess (and hope) that nobody does that!

If i understand you correctly that would make libavformat quite a bit bigger
namely by the size of libavcodec and libavutil. Iam not sure if this is
a good idea.

If you really want an "alternative" to versioning and to fixing the linker
that would be simply #including an header that appended _50 to all libavutil
symbols. This wont assert(0) and it wont miss-bind symbols to the wrong
version so it should avoid most of the conflict&dependancy dance. But one
thing it cant do, you cant add _49 to the existing lib obviously.
Iam not sure if this is a good idea but at least considering how poor
the gnu linker handles versioning simply hard renaming symbols seems quite a
bit more robust.

[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Incandescent light bulbs waste a lot of energy as heat so the EU forbids them.
Their replacement, compact fluorescent lamps, much more expensive, dont fit in
many old lamps, flicker, contain toxic mercury, produce a fraction of the light
that is claimed and in a unnatural spectrum rendering colors different than
in natural light. Ah and we now need to turn the heaters up more in winter to
compensate the lower wasted heat. Who wins? Not the environment, thats for sure
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20091211/1827d34d/attachment.pgp>