[FFmpeg-devel] [PATCH V4 1/2] lavfi/nlmeans: Checking number precision when computing integral images

[Jun Zhao]

From: Jun Zhao <barryjzhao at tencent.com>

accumulation of 8-bits uint_8 (uint8_t *src) into 32-bits (uint32_t *ii)
data type, it will have a risk of an integral value becoming larger than
the 32-bits integer capacity and resulting in an integer overflow. For
this risk, add a checking with warning message.

Signed-off-by: Jun Zhao <barryjzhao at tencent.com>
---
 libavfilter/vf_nlmeans.c |   11 +++++++++++
 1 files changed, 11 insertions(+), 0 deletions(-)

diff --git a/libavfilter/vf_nlmeans.c b/libavfilter/vf_nlmeans.c
index dcb5a03..8d47f9d 100644
--- a/libavfilter/vf_nlmeans.c
+++ b/libavfilter/vf_nlmeans.c
@@ -477,6 +477,17 @@ static int filter_frame(AVFilterLink *inlink, AVFrame *in)
     NLMeansContext *s = ctx->priv;
     AVFilterLink *outlink = ctx->outputs[0];
 
+    // accumulation of 8-bits uint_8 into 32-bits data type, it will have
+    // a risk of an integral value becoming larger than the 32-bits integer
+    // capacity and resulting in an integer overflow, so limit the image size
+    if ((UINT32_MAX / (uint64_t)inlink->w) < (255 * (uint64_t)inlink->h)) {
+        av_log(ctx, AV_LOG_ERROR,
+               "image size (%d x %d) integral value may overflow.\n",
+               inlink->w, inlink->h);
+        av_frame_free(&in);
+        return AVERROR(EINVAL);
+    }
+
     AVFrame *out = ff_get_video_buffer(outlink, outlink->w, outlink->h);
     if (!out) {
         av_frame_free(&in);
-- 
1.7.1