[FFmpeg-cvslog] avcodec/svq1enc: Use unsigned for parameter >= 0 to workaround GCC bug

[Andreas Rheinhardt]

ffmpeg | branch: master | Andreas Rheinhardt <andreas.rheinhardt at outlook.com> | Mon Jul 11 00:46:16 2022 +0200| [1cb7fd317c84117bbb13b14851d62f77f57bb9ce] | committer: Andreas Rheinhardt

avcodec/svq1enc: Use unsigned for parameter >= 0 to workaround GCC bug

encode_block() in svq1enc.c looks like the following:

static int encode_block(int block[7][256], int level)
{
    int best_score = 0;

    for (unsigned x = 0; x < level; x++) {
        int v = block[1][x];
        block[level][x] = 0;
        best_score      += v * v;
    }

    if (level > 0 && best_score > 64) {
        int score = 0;

        score += encode_block(block, level - 1);
        score += encode_block(block, level - 1);

        if (score < best_score) {
            best_score = score;
        }
    }

    return best_score;
}

When called from outside of encode_block(), it is always called with
level == 5.

This triggers a bug [1] in GCC: On -O3, it creates eight clones of
encode_block with different values of level inlined into it. The clones
with negative values are of course useless*, but they also lead to
-Warray-bounds warnings, because they access block[-1].

This has been mitigated in GCC 12: It no longer creates clones
for parameters that it knows are impossible. Somehow switching levels
to unsigned makes GCC know this. Therefore this commit does this.
(For GCC 11, this changes the warning to "array subscript 4294967295 is
above array bounds" from "array subscript -1 is below array bounds".)

[1]: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102513

*: These clones can actually be discarded when compiling with
-ffunction-sections.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt at outlook.com>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=1cb7fd317c84117bbb13b14851d62f77f57bb9ce
---

 libavcodec/svq1enc.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/libavcodec/svq1enc.c b/libavcodec/svq1enc.c
index 37a111a1ec..f188111699 100644
--- a/libavcodec/svq1enc.c
+++ b/libavcodec/svq1enc.c
@@ -91,7 +91,7 @@ static int ssd_int8_vs_int16_c(const int8_t *pix1, const int16_t *pix2,
 }
 
 static int encode_block(SVQ1EncContext *s, uint8_t *src, uint8_t *ref,
-                        uint8_t *decoded, int stride, int level,
+                        uint8_t *decoded, int stride, unsigned level,
                         int threshold, int lambda, int intra)
 {
     int count, y, x, i, j, split, best_mean, best_score, best_count;