[FFmpeg-devel] [PATCH 2/2] swscale/output: Bias 16bps output calculations to improve non overflowing range for GBRP16/GBRPF32

[Michael Niedermayer]

On Wed, Nov 02, 2022 at 10:16:57PM +0100, Andreas Rheinhardt wrote:
> Michael Niedermayer:
> > On Wed, Nov 02, 2022 at 10:02:39PM +0100, Michael Niedermayer wrote:
> >> Fixes: integer overflow
> >> Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>
> >> ---
> >>  libswscale/output.c       | 25 +++++++++++--------------
> >>  libswscale/x86/output.asm | 16 +++++++++++++++-
> >>  2 files changed, 26 insertions(+), 15 deletions(-)
> > 
> > Note, these corner case overflows could affect some of the similary
> > implemented cases that are not depth 16 too
> > 
> > ill fix them if issues are replicated
> > 
> 
> The checkasm-sw_gbrp runs into many overflows (when run under UBSan);
> e.g. fate.ffmpeg.org tells me of an issue in line 2289. Said line is not
> touched in your commits.

checkasm-sw_gbrp feeds random data widely outside sane ranges in.
the test certainly makes no sense for testing asm. There is no point
in matching C for widely invalid cases. Of cousre we shouldnt overflow
if any of this can be triggered with valid and real input (which probably
can be done in some cases)

thx

[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Elect your leaders based on what they did after the last election, not
based on what they say before an election.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20221102/a6c38665/attachment.sig>