[MPlayer-advusers] Long-standing encoding bug: ff_rate_estimate_qscale: Assertion `pict_type == rce->new_pict_type' failed

[Loren Merritt]

On Tue, 10 May 2005, RC wrote:
> On Tue, 10 May 2005 13:20:22  Hans Meine wrote:
>
>> now (ff_rate_estimate_qscale_assertion.* in
>> mplayerhq.hu/MPlayer/incoming/) is heavily stripped down, but the
>> assertion disappears when removing either the cropping, the
>> deinterlacing, or the B frame parameters! ;-}
>
> I've seen the same thing when encoding multiple-passes with B-frames
> enabled.  Funny that you have a hard time reproducing it, when I'm
> having the problem with every video I try, even with no cropping,
> scaling, deinterlacing, etc. Bugs in mencoder seem to search me out...

While I never tracked it down, I'm suspicious of vb_strategy. It modifies 
frametypes, and I think it tries to run again in the 2nd pass.

--Loren Merritt
-------------- next part --------------
Index: libavcodec/mpegvideo.c
===================================================================
RCS file: /cvsroot/ffmpeg/ffmpeg/libavcodec/mpegvideo.c,v
retrieving revision 1.481
diff -u -r1.481 mpegvideo.c
--- libavcodec/mpegvideo.c	8 May 2005 20:15:42 -0000	1.481
+++ libavcodec/mpegvideo.c	11 May 2005 03:07:28 -0000
@@ -2188,7 +2188,7 @@
             if(s->avctx->b_frame_strategy==0){
                 b_frames= s->max_b_frames;
                 while(b_frames && !s->input_picture[b_frames]) b_frames--;
-            }else if(s->avctx->b_frame_strategy==1){
+            }else if(s->avctx->b_frame_strategy==1 && !(s->flags&CODEC_FLAG_PASS2)){
                 for(i=1; i<s->max_b_frames+1; i++){
                     if(s->input_picture[i] && s->input_picture[i]->b_frame_score==0){
                         s->input_picture[i]->b_frame_score=