[FFmpeg-devel] [PATCH v2] avformat/rtpdec_rfc4175: Fix incorrect copy_offset calculation

[Jacob Siddall]

The previous calculation code did not account for the fact that the
copy_offset for the start of the frame array is at index 0, yet the
scan line number from the rfc4175 RTP header starts at 1.
This caused 2 issues to appear:
- The first scan line was being copied into the array where the second
  scan line should be. This caused the resulting video to have a green
  line at the top of it.
- Since the packet containing the last scan line would fail the
  calculation, the packet with the RTP marker would not be processed
  which caused a log message saying "Missed previous RTP marker" to be
  outputted for each frame.

Signed-off-by: Jacob Siddall <kobe at live.com.au>
---
Changes in v2:
  - Don't handle packet if the line number is less than 1

Section 12 in the VSF technical recommendation TR-03 specifies that the
video scan line numbers should start at 1.
http://www.videoservicesforum.org/download/technical_recommendations/VSF_TR-03_2015-11-12.pdf

 libavformat/rtpdec_rfc4175.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/libavformat/rtpdec_rfc4175.c b/libavformat/rtpdec_rfc4175.c
index e9c62c1..490db87 100644
--- a/libavformat/rtpdec_rfc4175.c
+++ b/libavformat/rtpdec_rfc4175.c
@@ -205,8 +205,11 @@ static int rfc4175_handle_packet(AVFormatContext *ctx, PayloadContext *data,
         if (length > payload_len)
             length = payload_len;
 
+        if (line < 1)
+            return AVERROR_INVALIDDATA;
+
         /* prevent ill-formed packets to write after buffer's end */
-        copy_offset = (line * data->width + offset) * data->pgroup / data->xinc;
+        copy_offset = ((line - 1) * data->width + offset) * data->pgroup / data->xinc;
         if (copy_offset + length > data->frame_size)
             return AVERROR_INVALIDDATA;
 
-- 
2.20.1