[FFmpeg-cvslog] pgssubdec: fix subpicture output colorspace and range

[Jan Ekström]

ffmpeg | branch: release/2.7 | Jan Ekström <jeebjp at gmail.com> | Sun Apr 24 17:30:56 2016 +0300| [3f69797e9b2f2d7060c5f7833479ab085bbca50f] | committer: Michael Niedermayer

pgssubdec: fix subpicture output colorspace and range

Functionality used before didn't widen the values from limited to
full range. Additionally, now the decoder uses BT.709 where it
should be used according to the video resolution.

Default for not yet set colorimetry is BT.709 due to most observed
HDMV content being HD.

BT.709 coefficients were gathered from the first two parts of BT.709
to BT.2020 conversion guide in ARIB STD-B62 (Pt. 1, Chapter 6.2.2).
They were additionally confirmed by manually calculating values.

Fixes #4637
(cherry picked from commit 9779b6262471d553c1ed811ff7312564e39d8adf)

Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=3f69797e9b2f2d7060c5f7833479ab085bbca50f
---

 libavcodec/pgssubdec.c |   10 ++++++++--
 libavutil/colorspace.h |   10 ++++++++++
 2 files changed, 18 insertions(+), 2 deletions(-)

diff --git a/libavcodec/pgssubdec.c b/libavcodec/pgssubdec.c
index e567f53..6ff6cc4 100644
--- a/libavcodec/pgssubdec.c
+++ b/libavcodec/pgssubdec.c
@@ -354,8 +354,14 @@ static int parse_palette_segment(AVCodecContext *avctx,
         cb        = bytestream_get_byte(&buf);
         alpha     = bytestream_get_byte(&buf);
 
-        YUV_TO_RGB1(cb, cr);
-        YUV_TO_RGB2(r, g, b, y);
+        /* Default to BT.709 colorimetry. In case of <= 576 height use BT.601 */
+        if (avctx->height <= 0 || avctx->height > 576) {
+            YUV_TO_RGB1_CCIR_BT709(cb, cr);
+        } else {
+            YUV_TO_RGB1_CCIR(cb, cr);
+        }
+
+        YUV_TO_RGB2_CCIR(r, g, b, y);
 
         ff_dlog(avctx, "Color %d := (%d,%d,%d,%d)\n", color_id, r, g, b, alpha);
 
diff --git a/libavutil/colorspace.h b/libavutil/colorspace.h
index 826ffd5..7d3f711 100644
--- a/libavutil/colorspace.h
+++ b/libavutil/colorspace.h
@@ -41,6 +41,16 @@
     b_add = FIX(1.77200*255.0/224.0) * cb + ONE_HALF;\
 }
 
+#define YUV_TO_RGB1_CCIR_BT709(cb1, cr1)\
+{\
+    cb = (cb1) - 128;\
+    cr = (cr1) - 128;\
+    r_add = FIX(1.5747*255.0/224.0) * cr + ONE_HALF;\
+    g_add = - FIX(0.1873*255.0/224.0) * cb - FIX(0.4682*255.0/224.0) * cr + \
+            ONE_HALF;\
+    b_add = FIX(1.8556*255.0/224.0) * cb + ONE_HALF;\
+}
+
 #define YUV_TO_RGB2_CCIR(r, g, b, y1)\
 {\
     y = ((y1) - 16) * FIX(255.0/219.0);\