[FFmpeg-devel] Upgrade Trouble

Reinhard Tartler siretart
Fri Dec 25 23:28:49 CET 2009


Michael Niedermayer <michaelni at gmx.at> writes:

>> I've not addressed points 1 and 3 because I haven't managed (yet) to
>> reproduce the symptoms. With the testcase I've written earlier, I've
>> encountered not an assertion failure, but a failure to find a symbol. So
>> I think you're trying something more obscure than I'm doing.
>

Okay, I've succeeded now at reproducing your test case. So let's see:

>
> gcc -shared -fPIC -c libA0.c -o libA0.o
> ld -shared libA0.o -o libA0.so
>
> gcc -shared -fPIC -c libA1.c -o libA1.o
> ld -shared libA1.o -o libA1.so
>
> gcc -shared -fPIC -c libB0.c -o libB0.o
> ld -shared libB0.o -o libB0.so -L. -lA0
>
> gcc app.c -o app -L. -lB0 -lA0

So we're here leaving out details like using proper SONAMEs for the
libraries, which makes objdump output a bit unfamiliar, but for this
example it doesn't matter

>
> ./app
> #libA0
> #B libB0
> #libA0
> #libA0
> #        (correct)
>
> gcc -shared -fPIC -c libA1.c -o libA1.o

this is pretty  unnecessary, it was already compiled above

> ld -shared --version-script libA1.v libA1.o -o libA1.so

now libA1 is introduced *with* symbol versioning. This is IMO wrong,
what I propose would be to first introduce symbol versioning in
libA0.so, recompile libB.so and app, and then introduce symbol
versioning in libA1.so.

But for the sake of this example, let's proceed.

> gcc -shared -fPIC -c libB0.c -o libB0.o

this makes libB0.c pickup the versioned symbol of libA1.

> ld -shared libB0.o -o libB0.so -L. -lA1
>
> ./app
> #libA0
> #B libB0
> #libA0
> #libA0
> #        (libB0 is linked by the linker to the wrong lib here)

This mixes a library with both versioned and unversioned symbols. The
"Base" symbol seems to be treated as a corner case by ld, that's why all
applications needs to be rebuilt to pickup the versioned symbols.

>
> gcc -shared -fPIC -c libA0.c -o libA0.o

again, superfluous

> ld -shared --version-script libA0.v libA0.o -o libA0.so

now we introduce symbol versioning to the 'old' lib, which removes the
@Base symbol from the application's view. This unsurprisingly results in:

> ./app
> #libA0
> #B libB0
> #libA1
> #libA0 
> #        (correct)
>

the expected behavior!

> gcc app.c -o app -L. -lB0 -lA0

now app picks up the versioned symbol from libA0. This should have been
done earlier.

> ./app
> #libA0
> #B libB0
> #libA1
> #libA0 
> #        (correct)

yupp

> gcc -shared -fPIC -c libA0.c -o libA0.o
> ld -shared libA0.o -o libA0.so

so now you remove the version tags. This essentially breaks the ABI and
results in:

>
> ./app
> #./app: ./libA0.so: no version information available (required by ./app)
> #Inconsistency detected by ld.so: do-lookup.h: 115: check_match: Assertion `version->filename == ((void *)0) || ! _dl_name_match_p (version->filename, map)' failed!

The error message above correctly indicates the problem. The assertion
failure could be probably seen as a very minor cosmetic issue.

> gcc -shared -fPIC -c libA0.c -o libA0.o
> ld -shared --version-script libA0.v libA0.o -o libA0.so

so symbol versioning in libA0.so is restored, the application should
work again.

> gcc -shared -fPIC -c libA1.c -o libA1.o
> ld -shared libA1.o -o libA1.so

now the version tags from libA1 is removed, which again results in
unsurprising behavior:

>
> ./app
> #./app: ./libA1.so: no version information available (required by ./libB0.so)
> #libA0
> #B libB0
> #Inconsistency detected by ld.so: do-lookup.h: 115: check_match: Assertion `version->filename == ((void *)0) || ! _dl_name_match_p (version->filename, map)' failed!
>

the runtime linker now fails to find the versioned symbols for libA1,
against which libB0 is linked.

I fail to see any bug in ld here, this example demonstrates that broken
usage of ld can produce some confusing error messages.

I'd suggest to just use symbol versioning as it is intended:

 - first introduce symbol tags
 - then recompile applications
 - do not remove symbol versioning afterwards without at least a SONAME
   bump (preferably never)

>> I think that when introducing symbol versioning all applications need
>> to be rebuilt in order to pick up the new versioned symbols and the
>> problem doesn't arise in the first place.
>
> An application that is not rebuild links to the correct symbols
> A not rebuilt lib can link to the wrong symbols, you can rebuild all libs
> But does this ensure that the user doesnt upgrade just a subset?

By using versioned dependencies in application packages, it is
impossible to install a rebuilt application without also installing a
rebuilt library. We use the dpkg-shlibdeps mechanism to automate this,
so nothing to worry about that here.

OTOH, updating the library without the application is no problem either,
because the runtime linker will use the versioned symbol if the
application doesn't specify any particular version. So no problem here
either.

NB: I do not intend to update the FFmpeg package from 0.5 to current
trunk yet, so the transition in Debian does not have to deal with the
SONAME change yet. The FFmpeg "trunk" package will be provided as add-on
package. So in this specific situation, you could indeed force by
partial upgrades a situation with that the runtime linker would pickup a
wrong symbol. But this is a corner case I can live with at the moment.
And even if I couldn't, I do have a plan how to tackle this. I'd rather
avoid that at this point in order to make the transition not more even
more complicated (I would need to rename binary packages and so on...)


BTW, the reason why this mail is so delayed is because I didn't manage
to find someone in the release team to track this issue, now I did and
got the green light that my proposed approach indeed makes sense. See
http://article.gmane.org/gmane.linux.debian.devel.release/32768 and the
answer to that mail.

>> > If one combines these 2 you even end up with a funnier case where a
>> > unversioned symbol is always prefered over the correct versioned one
>> > but the removial of the never used versioned one leads to hard failure
>> 
>> I don't see how this can become a problem in practice. If I upload a new
>> FFmpeg 0.5 package with versioned symbols, it will instantly replace all
>> copies of FFmpeg 0.5 libraries with unversioned symbols.
>
> on the debian servers sure but if an application gets rebuild against the
> new versioned libs cant the user upgrade the application without upgrading
> the lib? (the soname didnt change here, does something still add a dependancy
> to ensure this upgrade?)

yes, as indicated above, we use automatically calculated versioned
dependencies on library packages in Debian and ubuntu.

>> > Now if  1+3  where fixed without the "hierarchic symbol tables"
>> > as your colleague called my point 2 that still would be a huge simplification
>> > for distro packagers.
>> > The reasons should be obvious
>> >
>> > * currently introducing versioning one needs to make sure that no application
>> >   that has been build with the new versioned lib can end up being linked to
>> >   the older lib without versioning but otherwise 100% identical ABI.
>> >   Because ld.so goes bye bye with assert(0) in that case
>> 
>> That's why we have a documented and recommended migration strategy for
>> that. [1]
>
> That guide also says all libs depending on one that changed the soname must
> themself change the soname (thats just a ridiculously extreem way to work
> around linker bugs).
> You didnt plan to do that, but what matters more, is the rest of the guide
> still correct if one doesnt do that?

yes.

>> I guess I'll just go ahead and start working on the plan with
>> introducing symbol versioning.
>
> fine with me

Okay, so here we go with the initial patches:

The first patch adds the versioning scripts. random thoughts on them:

 - I generated them automatically from configure at first, but dropped
   that idea because it doesn't make too much sense, IMO.
 - should they be moved to some subdirectory? if yes to which?
 - I did leave the symbols for libavformat and libavcodec rather
   unrestricted to not break application that use private
   symbols. However it is pretty straightforward how to maintain it and
   make it more restrictive. This should be done by individual file
   maintainers who have more insight which symbols should be public and
   which shouldn't




More information about the ffmpeg-devel mailing list