[Ffmpeg-devel] Re: mjpeg with trellis quant segaults ... to be expected?

[Michael Niedermayer]

Hi

On Thu, Oct 26, 2006 at 09:22:13AM +0200, Karl H. Beckers wrote:
> Hi,
> dunno if trellis quant with mjpeg makes any sense, but at any rate,
> ffmpeg should treat this gracefully. The error occurs in mpegvideo.c at
> line 5920 and the following patch seems to be the safe (if possibly not
> the most resource conservative) approach for avoiding it.
> 
> mjpeg.c doesn't seem to set intra_ac_vlc_length on the MpegEncContext.
> 
> Cheers,
> 
> Karl.
> 
> svn diff mpegvideo.c
> Index: mpegvideo.c
> ===================================================================
> --- mpegvideo.c (Revision 6791)
> +++ mpegvideo.c (Arbeitskopie)
> @@ -5916,8 +5916,13 @@
>              level+=64;
>              if((level&(~127)) == 0){
>                  for(j=survivor_count-1; j>=0; j--){
> +                   int score;
>                      int run= i - survivor[j];
> -                    int score= distoration +
> length[UNI_AC_ENC_INDEX(run, level)]*lambda;
> +
> +//av_log(NULL, AV_LOG_INFO, "here distoration %i lambda %i run %i level
> %i\n\tindex %i\n", distoration, lambda, run, level,
> UNI_AC_ENC_INDEX(run, level));
> +                   if (!length) break;
> +                    score= distoration + length[UNI_AC_ENC_INDEX(run,
> level)]*lambda;
> +//av_log(NULL, AV_LOG_INFO, "here2\n");
>                      score += score_tab[i-run];
> 
>                      if(score < best_score){

the indention of the new code is wrong, the patch has random linebreaks
in the midde (=its malformed and not applyable)
besides checking for this in the innermost loop is not acceptable
this can and should be checked during init or trellis quant should be
changed to support mjpeg

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

In the past you could go to a library and read, borrow or copy any book
Today you'd get arrested for mere telling someone where the library is