[FFmpeg-cvslog] vc2enc_dwt: use 32 bit coefficients by default

[Rostislav Pehlivanov]

ffmpeg | branch: master | Rostislav Pehlivanov <atomnuker at gmail.com> | Sat May  7 18:05:25 2016 +0100| [b6c207f5358181f64e8cfadc929d61f0ed25266c] | committer: Rostislav Pehlivanov

vc2enc_dwt: use 32 bit coefficients by default

The problem is that with particularly complex images and especially at
high bit depths and 5-level transforms the coefficients would overflow,
causing huge artifacts to appear. This was discovered thanks to the fate
tests, which will have to be redone as this fixes a multitude of
problems and increases PSNR.

There is a slight performance drop associated with this change, making
the encoder slower by 1.15 times, however this is necessary in order to
avoid undefined behavior and overflows.

It would be worth to template the transforms to keep the performance for
8 bit images as 32 bit coefficients are unnecessary for that case, but
the primary use of the encoder is to encode video at 10 bits.

Reviewed-by: Christophe Gisquet <christophe.gisquet at gmail.com>
Reviewed-by: Michael Niedermayer <michael at niedermayer.cc>
Signed-off-by: Rostislav Pehlivanov <atomnuker at gmail.com>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=b6c207f5358181f64e8cfadc929d61f0ed25266c
---

 libavcodec/vc2enc_dwt.h |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/libavcodec/vc2enc_dwt.h b/libavcodec/vc2enc_dwt.h
index af6fe3e..7fbbfbe 100644
--- a/libavcodec/vc2enc_dwt.h
+++ b/libavcodec/vc2enc_dwt.h
@@ -25,7 +25,7 @@
 #include <stddef.h>
 #include <stdint.h>
 
-typedef int16_t dwtcoef;
+typedef int32_t dwtcoef;
 
 enum VC2TransformType {
     VC2_TRANSFORM_9_7    = 0,   /* Deslauriers-Dubuc (9,7)  */