[FFmpeg-devel] Upgrade Trouble

[Reinhard Tartler]

Michael Niedermayer <michaelni at gmx.at> writes:

> Are you sure its not worthwhile to fix the linker? This dependancy & conflict
> dance has to be done each time a project wants to add versioning.
> The linker should do the following (and it does not)
> 1. Prefer a symbol with matching version over an unversioned symbol.
>    Not just pick the first that does not have a mismatching version.
> 2. do a breadth first search from the object where a symbol is referenced
>    Not a breadth first search from the application.
> 3. Not fail assertions.

As indicated above, I've now done more research on your idea. I've
talked to a colleague at my department, who has programmed an linker for
embedded systems, and fetched Levine's excellent book "Linkers and
Loaders" from the library.

Let's recapitulate roughly what the runtime linker does at load time:
Conceptually, the runtime linker builds at bootstrap phase of a program
a global symbol table containing all symbols of all DSO the application
is linked to. This symbol table is only conceptually global. In
practice, there are actually two (one for code, one for data), and there
are actually tables for each and every library. At bootstrap phase,
these tables are combined in a linked list. (there are a bit more
optimizations going on here like lazy lookups for function, but that
doesn't matter for this discussion, for details please see Levine's
book, chapter 10.3, "Loading a Dynamically Linked Program" and chapter
10.4, "Lazy Procedure Linking with the PLT").

What you now propose is to introduce "hierarchic symbol tables", as my
colleague called it. I would rather call that "introducing namespace"
that are created on load time of a library. This way the runtime linker
would have to obey the library in which the search for symbol it was
started.

I think this is not a good idea for a few reasons:

 - you could of course change the list in which the symbol tables are
   linked to a graph to implement the "hierarchic symbol table" conecpt,
   but this increases the complexity of the runtime linker considerably!

 - this would probably only work for functions, not for code (would have
   to think more about this)

 - it would totally break the existing behavior of LD_PRELOAD and
   dlopen()-tricks that overshadow existing symbols. A lot of really
   useful applications rely on this.

So all in all, I'm now pretty sure that changing the existing linker is
an option for solving the problem at hand.

In parallel, I asked the Debian glibc maintainers for their opinion. I
was pointed to the -Bsymbolic ld flag, which, alters the symbol lookup
behavior of the runtime linker as well. 

At first, I was sceptic on that idea, and frankly, I still have to
actually experiment with this approach and to think more about this if
it would really helps to fix the problem.  On the second thought,
*might* indeed be an alternative to symbol versioning.  However, it
would require to create static shared libraries, e.g. to link libavutil
statically into the dynamically linked libavcodec and libavformat, and
libavcodec statically in the dynamically linked libavformat.  It would
effectively prevent to LD_PRELOAD existing symbols, e.g. in libavutil. I
guess (and hope) that nobody does that!

Implementing this might be more challenging for me though, because it
would require more invasive changes and increased complexity to the
Makefiles and the configure script.  In detail, the Makefile would need
to know exactly which libraries to link statically an which dynamically
in what way. So, how do the makefile maintainers feel about this
approach?


Still, this approach wouldn't get us the other benefits of symbol
versions. I guess both approaches can be combined, though.

-- 
Gruesse/greetings,
Reinhard Tartler, KeyID 945348A4