[FFmpeg-devel] [PATCH 1/2] vc2enc: prevent random data

[Christophe Gisquet]

The slice prefix is 0 in the reference encoder and the decoder ignores it.
Writing 0 there seems like the best temporary solution.

The padding could have contained uninitialized data, but reference VC2
encoders put 0xFF there, hence the memset value.

Overall this allows producing bistreams with no random data for use by fate.
---
 libavcodec/vc2enc.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/libavcodec/vc2enc.c b/libavcodec/vc2enc.c
index 6d24552..bbbeaa0 100644
--- a/libavcodec/vc2enc.c
+++ b/libavcodec/vc2enc.c
@@ -777,7 +777,10 @@ static int encode_hq_slice(AVCodecContext *avctx, void *arg)
     uint8_t quants[MAX_DWT_LEVELS][4];
     int p, level, orientation;
 
+    /* The reference decoder ignores it, and its typical length is 0 */
+    memset(put_bits_ptr(pb), 0, s->prefix_bytes);
     skip_put_bytes(pb, s->prefix_bytes);
+
     put_bits(pb, 8, quant_idx);
 
     /* Slice quantization (slice_quantizers() in the specs) */
@@ -809,6 +812,8 @@ static int encode_hq_slice(AVCodecContext *avctx, void *arg)
         }
         pb->buf[bytes_start] = pad_s;
         flush_put_bits(pb);
+        /* vc2-reference uses that padding that decodes to '0' coeffs */
+        memset(put_bits_ptr(pb), 0xFF, pad_c);
         skip_put_bytes(pb, pad_c);
     }
 
-- 
2.8.1