[FFmpeg-cvslog] avcodec/tiffenc: properly compute packet size

[Christophe Gisquet]

ffmpeg | branch: release/2.4 | Christophe Gisquet <christophe.gisquet at gmail.com> | Sun Oct 12 21:10:54 2014 +0200| [30a0622a5dbf2753553deddbfbf2410f7d7799e6] | committer: Michael Niedermayer

avcodec/tiffenc: properly compute packet size

The bytes per row is a better indication of it.

Helps resolving ticket #3874 by fixing ffmpeg's encoder and transforming
the issue in a issue with non-compliant decoders. ffmpeg's one is ok,
but unfortunately, many others aren't handling correctly unusual chroma
samplings.

Signed-off-by: Michael Niedermayer <michaelni at gmx.at>
(cherry picked from commit 0e8bfd3c934768f9812dd20d71fa4709de54186d)

Signed-off-by: Michael Niedermayer <michaelni at gmx.at>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=30a0622a5dbf2753553deddbfbf2410f7d7799e6
---

 libavcodec/tiffenc.c |    6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/libavcodec/tiffenc.c b/libavcodec/tiffenc.c
index 5a61f1a..138d214 100644
--- a/libavcodec/tiffenc.c
+++ b/libavcodec/tiffenc.c
@@ -305,7 +305,9 @@ static int encode_frame(AVCodecContext *avctx, AVPacket *pkt,
 
     strips = (s->height - 1) / s->rps + 1;
 
-    packet_size = avctx->height * ((avctx->width * s->bpp + 7) >> 3) * 2 +
+    bytes_per_row = (((s->width - 1) / s->subsampling[0] + 1) * s->bpp *
+                     s->subsampling[0] * s->subsampling[1] + 7) >> 3;
+    packet_size = avctx->height * bytes_per_row * 2 +
                   avctx->height * 4 + FF_MIN_BUFFER_SIZE;
 
     if ((ret = ff_alloc_packet2(avctx, pkt, packet_size)) < 0)
@@ -333,8 +335,6 @@ static int encode_frame(AVCodecContext *avctx, AVPacket *pkt,
         goto fail;
     }
 
-    bytes_per_row = (((s->width - 1) / s->subsampling[0] + 1) * s->bpp *
-                     s->subsampling[0] * s->subsampling[1] + 7) >> 3;
     if (is_yuv) {
         av_fast_padded_malloc(&s->yuv_line, &s->yuv_line_size, bytes_per_row);
         if (s->yuv_line == NULL) {