[FFmpeg-cvslog] avcodec/utvideodec/enc: Fix edge case of creating Huffman table

[Andreas Rheinhardt]

ffmpeg | branch: master | Andreas Rheinhardt <andreas.rheinhardt at gmail.com> | Thu Sep 24 16:19:03 2020 +0200| [099feb941147fbf10d1ae941b96bf94c4b4dbd1c] | committer: Andreas Rheinhardt

avcodec/utvideodec/enc: Fix edge case of creating Huffman table

The Ut Video format stores Huffman tables in its bitstream by coding
the length of a given symbol; it does not code the actual code directly,
instead this is to be inferred by the rule that a symbol is to the left
of every shorter symbol in the Huffman tree and that for symbols of the
same length the symbol is descending from left to right. With one
exception, this is also what our de- and encoder did.

The exception only matters when there are codes of length 32, because
in this case the first symbol of this length did not get the code 0,
but 1; this is tantamount to pretending that there is a (nonexistent)
leaf of length 32. This is simply false. The reference software agrees
with this [1].

[1]: https://github.com/umezawatakeshi/utvideo/blob/2700a471a78402e5c340150b38e8a793ef3676f1/utv_core/HuffmanCode.cpp#L280

Reviewed-by: Paul B Mahol <onemda at gmail.com>
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt at gmail.com>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=099feb941147fbf10d1ae941b96bf94c4b4dbd1c
---

 libavcodec/utvideodec.c | 4 ++--
 libavcodec/utvideoenc.c | 2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/libavcodec/utvideodec.c b/libavcodec/utvideodec.c
index c07636d435..b3c4c3519b 100644
--- a/libavcodec/utvideodec.c
+++ b/libavcodec/utvideodec.c
@@ -70,7 +70,7 @@ static int build_huff10(const uint8_t *src, VLC *vlc, int *fsym)
         return -1;
     }
 
-    code = 1;
+    code = 0;
     for (i = last; i >= 0; i--) {
         codes[i] = code >> (32 - he[i].len);
         bits[i]  = he[i].len;
@@ -113,7 +113,7 @@ static int build_huff(const uint8_t *src, VLC *vlc, int *fsym)
     if (he[last].len > 32)
         return -1;
 
-    code = 1;
+    code = 0;
     for (i = last; i >= 0; i--) {
         codes[i] = code >> (32 - he[i].len);
         bits[i]  = he[i].len;
diff --git a/libavcodec/utvideoenc.c b/libavcodec/utvideoenc.c
index f1b9d11c96..05a9614036 100644
--- a/libavcodec/utvideoenc.c
+++ b/libavcodec/utvideoenc.c
@@ -346,7 +346,7 @@ static void calculate_codes(HuffEntry *he)
     while (he[last].len == 255 && last)
         last--;
 
-    code = 1;
+    code = 0;
     for (i = last; i >= 0; i--) {
         he[i].code  = code >> (32 - he[i].len);
         code       += 0x80000000u >> (he[i].len - 1);