[Ffmpeg-devel] mjpeg with trellis quant segaults ... to be expected

[Loren Merritt]

On Thu, 26 Oct 2006, Michael Niedermayer wrote:

> On Thu, Oct 26, 2006 at 07:23:44PM +0200, Karl H. Beckers wrote:
>>> the indention of the new code is wrong, the patch has random linebreaks
>>> in the midde (=its malformed and not applyable)
>>> besides checking for this in the innermost loop is not acceptable
>>> this can and should be checked during init or trellis quant should be
>>> changed to support mjpeg
>>
>> Michael,
>> I only sent the diff to highlight the issue not to suggest that this
>> should be the fix or applied as a patch. I'm also all with you about
>> adding the feature (though that won't be me ;S),
>
> why not? its not that hard, first just fill the length array with the
> correct vlc lengths see  mjpeg.c:encode_block() then ensure that the
> assumed dequantization and assumed number of bits in trellis_quant
> is correct ...

The assumed dequantization is not correct. Even if the length array is 
full of zeros, trellis quantization produces lower bitrate and much lower 
psnr than non-trellis.
I haven't looked in detail where it goes wrong... The most obvious error 
would be that h263 uses qadd=.5 and jpeg uses qadd=0, but fixing that 
didn't help measurably.

--Loren Merritt