[FFmpeg-devel] Upgrade Trouble

[Michael Niedermayer]

On Fri, Dec 04, 2009 at 09:24:46PM +0100, Reinhard Tartler wrote:
> Michael Niedermayer <michaelni at gmx.at> writes:
> 
> > what i suggest is that the linker behaves sanely without bandaids.
> > what it does is a breadth first search from the application, what it
> > should do is a breadth first search from the object where a symbol is
> > used.
> > iam no expert in all these corner cases of shared lib ABI compatibility
> > but then its clear the others in this thread arent either.
> > Possibly RTLD_DEEPBIND could solve the problem. Drepper is quite vocal
> > on arguing against it which is a sign that this might be the right
> > direction.
> 
> RTLD_DEEPBIND is a flag for dlopen(). We are not talking about
> applications that load ffmpeg this way but are dynamically linked.

for the case that
application -> libA
application -> libB.0
libA -> libB.1

application will have their references bound to libB.0 before libB.1 
similarly symbols referenced from libA will be bound to libB.0 not libB.1
(this is required by the ELF spec and not particularly sane, after all libA
 specifies a dependancy on libB.1 not .0)


> 
> Moreover, that flag wouldn't help even if we would find some way to make
> the dynamic linker (ld-linux.so or similar) use that flag. Since it
> applies to intra-library symbol lookups only, it does not help solving
> the problems caused by transitive dependency problems that we are
> currently discussing in this thread.

The way i understand it the application is linked to the correct libavutil
rather libavcodec is linked to the wrong one.


> 
> > Either way, ffmpeg isnt the only thing that seems to have been hit by this
> > A->B1,C1 B1->C2 symbol shadowing bug so i really think that a generic
> > solution at the linker/loader level would be best. It might solve this
> > problem once and for all.
> 
> Correct. other libraries that got hit by this problem started
> implementing proper symbol versioning.  Prominent examples for this
> include the gnu libc and libdb (berkely db).  A more complete explanation
> why symbol versioning helps in the situation of is explained in Dreppers
> DSO howto[1], section "3.3 ABI Versioning", p.35f.

i dont like drepper


> 
> > Of course iam not against other solutions if a linker/loader based solution
> > is not possible. symbol versioning seems like an option, ...
> 
> Indeed, however, proper symbol versioning has additionally the advantage
> that the ffmpeg libraries could properly prevent exporting non-public
> symbols.  This has been previously attempted by Diego, but his patch was
> unfortunately never finished.
> 
> Let's recapitulate, what is symbol versioning after all?  It basically
> means that a special linker script is used to assign "version tags" to
> symbols in an ELF DSO. That version script classifies symbols either as
> "global" or "local":
> 
>   * local symbols are marked as 'hidden' and cannot be used outside the
>     DSO.  This is an effective way to hide private symbols
> 
>   * global symbols are part of the public ABI
> 
> The "assignment" of tags is similar to mechanically adding prefixes to
> symbol names. This effectively makes the assigned symbol part of the
> symbol name and thus, part of the ABI! This means that symbol versioning
> should be done carefully.
> 
> > ... though here some extra work is needed to manually patch up the
> > corners like symbols that have been moved from lavf -> lavc.
> 
> Indeed. The approach to statically prefix the SONAME to each symbol is
> trivial to implement, but leads to the situation that moving symbols
> from lavf to lavc effectively results in an ABI break that would require

> an SONAME bump.  This might be an acceptable compromise, but can only be
> avoided by using a more sophisticated approach for symbol versioning.

For moving symbols between lavf->lavc we would need to keep the symbol with
the old version in lavf until the next bump but that would have to be under
some kind of #if ELF_SO_VERSIONING otherwise we could end up with 2 symbols
of the same name that might have some sideeffects


> 
> The safest approach to avoid this would probably be a version script,
> in that each maintainer needs to carfully maintain version numbers for
> each publicly exported symbol.
> 
> > Bumping tha soname of a lib that has not changed its interface is something
> > id like to avoid.
> 
> I absolutely agree.  This is the only way applications can benefit from
> improvments in the library without needing to be recompiled!
> 

> > I dont think lavc/f exports parts of lavu, if they do that would need a
> > soname bump of course.
> > That said iam open to all solutions if the nicer ones are exhausted, the
> > problem is real and a solution needed.
> 
> I'm happy to read that you take this problem seriously.  I'd be willing
> to help with implementing symbol versioning in FFmpeg, if you can give
> me hints what kind of work would be helpful.

If we do have to use versioning, then it seems changing libavutil to use
it would solve the problem at hand, that also would avoid any problems
with lavc->lavf moves for the moment.

also symbols not starting with av/ff should be local (though this is a
seperate issue and will need some work in the form of exceptions being
added due to historic symbols which didnt had these prefixes but need to be
exported)


> 
> > Also there is another aspect that should be mentioned, debians dependancy
> > system is buggy if it does not detect this.
> > I mean if the linker/loader used by debian cannot handle 2 libs with
> > different major abi versions linked together it should not allow this
> > situation to arise for any application.
> >
> > something like:
> > lavu=123
> > conflicts_when_loaded_togeher: lavu!=123
> 
> That's trivial to implement by making a potential libavutil50 package
> declare a 'Breaks' or 'Conflicts' relationship with libavutil49.  Please

theres a difference between
!(A && B)
vs.
!(A && B && C) for all C in Packages for which C depends on A && B through
any path


> note that no packaging system can have the clairvoyance skills for
> this.

What i suggested is probably difficult to implement in practice
and i doubt it will be done by anyone anytime soon also you are
correct that its not a solution on its own ...
but it surely is unambigous and well defined and a hypothetical
packaging system could do it
its hard because not only could a change to the actual lib cause a
conflict, but quite many actions that appear unrelated could too
when 2 ABI versions of the lib are installed. This would make
conflict resolution quite "interresting"

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Democracy is the form of government in which you can choose your dictator
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20091205/ce0d8cde/attachment.pgp>