[FFmpeg-devel] [PATCH] potential fix for memory underread issue in h264 predict

[Ronald S. Bultje]

Hi,

On Tue, Jan 18, 2011 at 9:32 PM, Daniel Kang <daniel.d.kang at gmail.com> wrote:
> A VLC user (GomGom on freenode) has brought up a bug. h264's
> pred8x8l_horizontal may underread a buffer in certain cases (i.e. if the
> top row does not exist). The check for this has been moved up so the
> buffer is not underread. The patch passes
> fate-(h264|svq3|vp8|real-rv40), the tests relevant to this patch.
>
> I believe this occurs when the flag emu_edge is set.
>
> I cannot replicate the issue, so I am not sure if it is a correct fix.

attached is a better version of this patch. Passes make fate-h264 and
fixes at least some valgrind errors with -flags emu_edge on one of the
fate samples that's failing with that flag set. It still breaks, I'm
working on the other errors to fix it for good so we can enable this
on fate and track it automatically.

Ronald
-------------- next part --------------
A non-text attachment was scrubbed...
Name: horizontal_simd.patch
Type: application/octet-stream
Size: 2421 bytes
Desc: not available
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20110119/751eb9b9/attachment.obj>