[MPlayer-advusers] Long-standing encoding bug: ff_rate_estimate_qscale: Assertion `pict_type == rce->new_pict_type' failed

[Hans Meine]

Hi everybody!

I have been using mplayer and mencoder for a long time, and still am impressed 
by it's speed / efficiency and quality (both when decoding and encoding).
I even developed a small python GUI for mencoder that I plan(ned) to release 
some day.. ;-)

However, I have been having problems for several months now with mencoder 
stopping during the second (lavc-divx) pass with this assertion failing:
mencoder: ratecontrol.c:635: ff_rate_estimate_qscale: Assertion `pict_type == 
rce->new_pict_type' failed.

I found some old messages concerning this assertion in the WWW, but nothing 
really related.  The bug appeared _after_ 1.0pre4 IIRC, and remains until 
today's CVS.  It was not trivial to prepare a testcase, but I finally found 
the time to strip a file and the failing commands down to this:

This triggers the assertion on 1.0pre6-3.3.5, 1.0pre6-3.3.5-20050130 (Gentoo 
ebuild) and dev-CVS-050509-09:37-3.3.5-20050130.

As mentioned, the bug was hard to nail down, because often, it would go away 
when I tried to encode only small parts of the original file (even when 
choosing the part where it failed before!), and the testcase that I uploaded 
now (ff_rate_estimate_qscale_assertion.* in mplayerhq.hu/MPlayer/incoming/) 
is heavily stripped down, but the assertion disappears when removing either 
the cropping, the deinterlacing, or the B frame parameters! ;-}

I sincerely hope the testcase is useful and someone finds the bug! ;-)

Last-minute update: Now that I have the testcase, I found out the day CVS 
broke (by binary search), it's the 15th of October 2004, with a commit to 
libavcodec/mpegfile.c concerning B frame handling, see for yourself:
   cvs diff -D20041014 -D20041015 mpegvideo.c

In other words: updating everything to CVS from 20041015 still shows the 
error, but reversing the version of mpegvideo.c to 1.440 (which is that from 
20041014), the encoding works again.
However, this does of course not mean that I know a fix.. ;-/

Nice greetings,
Ciao, /  /
     /--/
    /  / ANS