[FFmpeg-devel] [PATCH V2] lavfi/nlmeans: Checking number precision when computing integral images

[Jun Zhao]

From: Jun Zhao <barryjzhao at tencent.com>

accumulation of 8-bits uint_8 (uint8_t *src) into 32-bits (uint32_t *ii)
data type, it will have a risk of an integral value becoming larger than
the 32-bits integer capacity and resulting in an integer overflow. For
this risk, add a checking with warning message.

Signed-off-by: Jun Zhao <barryjzhao at tencent.com>
---
 libavfilter/vf_nlmeans.c |    7 +++++++
 1 files changed, 7 insertions(+), 0 deletions(-)

diff --git a/libavfilter/vf_nlmeans.c b/libavfilter/vf_nlmeans.c
index dcb5a03..e7015cd 100644
--- a/libavfilter/vf_nlmeans.c
+++ b/libavfilter/vf_nlmeans.c
@@ -236,6 +236,13 @@ static void compute_ssd_integral_image(const NLMeansDSPContext *dsp,
     // adjusted end x position of the safe area after width of the safe area gets aligned
     const int endx_safe = startx_safe + safe_pw;
 
+    // accumulation of 8-bits uint_8 (uint8_t *src) into 32-bits (uint32_t *ii)
+    // data type, it will have a risk of an integral value becoming larger than
+    // the 32-bits integer capacity and resulting in an integer overflow.
+    if ((UINT32_MAX / (uint64_t)w) < (UINT8_MAX * (uint64_t)h))
+        av_log(NULL, AV_LOG_WARNING,
+               "image (%d x %d) integral value maybe overflow.\n", w ,h);
+
     // top part where only one of s1 and s2 is still readable, or none at all
     compute_unsafe_ssd_integral_image(ii, ii_linesize_32,
                                       0, 0,
-- 
1.7.1