[FFmpeg-devel] release

[Robert Swain]

2009/1/30 Robert Swain <robert.swain at gmail.com>:
> 2009/1/30 Ronald S. Bultje <rsbultje at gmail.com>:
>> On Fri, Jan 30, 2009 at 9:28 AM, Benjamin Larsson <banan at ludd.ltu.se> wrote:
>>> have releases - silence the major complaint we have about that
>>
>> And how will it silence the complaints if it doesn't address them?
>> Your definition of "release" is "make a tarball", and most of the
>> complainers might have higher expectations. Have you asked the
>> complainers what they're hoping to achieve by having ffmpeg do a
>> "release"?
>
> Indeed. Releases aren't for us, we'll be using the development code
> regardless of releases I expect. Releases are for users. If we don't
> support/maintain the releases, what is to be gained from doing them?
>
> Diego: I assumed, perhaps wrongly, that you were invigorated to make a
> release quickly to get the release ball rolling for good reasons -
> what do you want to achieve by conducting releases?

I went back to the beginning of the FOMS thread to check the initial concerns.

2009/1/22 Peter Ross <pross at xvid.org>:
> 1. CONCERN: Quality assurance
>   "It is difficult to determine which revision of the FFmpeg tree is
>    suitable for distributing and/or linking against.."

So, releases need to instil confidence in the quality of the FFmpeg
code being used. But what will achieve increased confidence? (I'll be
commenting as I go along and summarising with bullet points.)

>   Yes, the head revision isn't always suitable for use due to regressions.
>   Recent examples include the Aug'08 WMA regression (now fixed), and audio
>   resampling between different sample formats (my fault, still broken).

The WMA stuff was my fault. Float issues that differed across
platforms of which I was unaware.

>   Mike Melanson's FATE effort was discussed, and seen as a good step forward
>   in improving QA.

Most definitely. The issues above can/could be caught by FATE. If we
know FATE is happy with the selected revision to be used for a
release, that should provide some confidence that we haven't screwed
anything up.

>   There are many outstanding bugs in roundup.
>   Bugs get dealt with when *somebody* fixes them (stating the obvious).

This has lead to the proposal of a bug squashing weekend, regular such
occurrences, especially timed before releases so that we can possibly
garner community support to test and report bugs when we're building
up to a release, should also provide confidence.

>   Regression tests to stress the behaviour of the FFmpeg API, not just
>   bitstream regressions, are lacking.

Agreed. This should be ranked highly on the wish list for those
concerned about improving confidence in the stability of the FFmpeg
code base. Not only expanding the scope of the regressions tests, but
helping Mike to expand the scope of FATE as well, as has been noted on
his blog.

>   Formal releases have been discussed at length by the FFmpeg community and
>   have not been pursued due to lack of time & effort. 'Somebody' needs to
>   do the hard work.

It will happen.

>   Other projects employ bug squashing events to improve quality.

See above.

>   GOAL:        Improve QA processes
>   OBJECTIVES:  Extend FFmpeg regression test scripts and test cases.
>                Contribute FATE test cases.
>                Fix more bugs.

Indeed.

> 2. CONCERN: API stability
>   "The FFmpeg API keeps changing..."
>
>   API is not backwards compatible between major API version bumps. Stuff
>   gets deprecated, API behaviour changes.

That's the point of major version bumps.

>   This makes upgrading the libav* packages on a distribution difficult,
>   because often the application also needs to be upgraded.

This was discussed but I'm not sure if any useful change in our
procedure was considered that could resolve the issue.

>   As long as new codecs, containers and concepts are being added to FFmpeg,
>   the API will continue to change.

I'm pretty sure this was completely discredited. Adding a codec or
container does not alter the public API and I'm not sure what a
'concept' is in this context.

>   Ensuring backwards compatibility is a lot of work. There are perhaps more
>   important things to be concerned about.

It's a trade off that is considered every time it needs to happen.

>   Do we need a mechanism to inform users of FFmpeg about API changes?

Definitely. As discussed, it should be maintained as and when the API
changes occur along with information as to how to update software to
work with the changed API.


What follows are some of my own thoughts regarding improving
confidence in the stability of the code base and otherwise.

Our documentation for API and CLI users is lacking. We should either
add a wiki to ffmpeg.org or add a link to an FFmpeg section of
wiki.multimedia.cx. I believe the latter already exists so I shall go
ahead and do this shortly if it's not already there. If we can get the
community working on documentation for us, that's less work for us and
less complaints from those who don't know how to use the API/CLI as
well as some of us do.

I just had the idea that maybe alongside bug squashing weekends we
could have documentation weekends. Or maybe they could be simultaneous
so that those who can't fix bugs can contribute to the cause. Whatever
is most productive.

Another item of note is that external projects seem to lack confidence
in using our libavformat API. I think it would be good if some of our
developers who are knowledgeable in this area could take the time to
assist the major third party projects and conduct some sort of audit
of their lavf wrapper code to check that they are "doing it right"
(tm).

We may well complain when people in each of the projects expend effort
rewriting de/muxers rather than helping to improve the implementations
in FFmpeg but, we need to tackle the source of that problem. If these
people were happier with the way libavformat functioned in their
projects, I would expect a higher proportion of contributions from the
outside to libavformat. This would definitely be good for FFmpeg and
could lead to libavformat being held in as high regard as libavcodec.

Also, if when doing this people could make some notes on the things
that they were doing wrong and make note of before and after and the
changes made, such issues could be documented and avoided by API users
in the future.

So, to summarise I'll split the ideas into two sections. Those
pertaining to releases and those unrelated but very worthwhile.

Releases:
- The goal is to increase confidence in a particular FFmpeg revision's
API and general stability.
- Releases should not contain known regressions.
  - Confidence can be increased in this area by contributing to the
regressions tests and to FATE.
- We should aim to reduce the known bug count to a minimum between releases.
  - Regular bug squashing events, particularly before releases
(coupled with community testing triggered by "we're thinking of
releasing soon, please test heavily and report bugs") as well as the
usual steady trickle of bug fixes should help here.

Not release-related:
- Document API/ABI changes.
  - This should be done when the change occurs.
  - Ideally the documentation should not just contain that a change
has occurred but also what the change was, how to update to follow the
change and any other considerations imposed by the change.
- API/CLI documentation is lacking.
  - Wiki?!
  - API examples need lots of comments and should be tutorial like.
  - API examples need to be more bite-sized. Some external tutorials
have been written that could possibly be pulled onto the wiki with the
authors' consent and be maintained as official API tutorials.
  - Documentation events.
- Audit libav* wrappers in external projects (GStreamer, VLC, xine and
so on) to improve their implementations and document common mistakes
as well as the "right way" (tm).

I hope reading this helps to re-affirm the goals for the release and
some other goals to improve the project.

Best regards,
Rob