[FFmpeg-devel] [PATCH] avcodec/x86/hpeldsp: fix half pel interpolation

[Michael Niedermayer]

On Fri, Apr 27, 2018 at 04:47:13PM +0200, Jerome Borsboom wrote:
> The assembly optimized half pel interpolation in some cases rounds the
> interpolated value when no rounding is requested. The result is a off by one
> error when one of the pixel values is zero.
> 
> Signed-off-by: Jerome Borsboom <jerome.borsboom at carpalis.nl>
> ---
> In the put_no_rnd_pixels functions, the psubusb instruction subtracts one from each
> unsigned byte to correct for the rouding that the PAVGB instruction performs. The psubusb
> instruction, however, uses saturation when the value does not fit in the operand type,
> i.e. an unsigned byte. In this particular case, this means that when the value of a pixel
> is 0, the psubusb instruction will return 0 instead of -1 as this value does not fit in
> an unsigned byte and is saturated to 0. The result is that the interpolated value is not
> corrected for the rounding that PAVGB performs and that the result will be off by one.
> 
> The corrections below solved the issues for me, but I do not a lot of experience in optimizing
> assembly. A good check for the correctness of the solution might be advisable. Furthermore,
> I have not checked the other assembly, but there may be more cases where the psubusb
> instruction does not provide the desired results. A good check by the owner/maintainer of
> the assembly code might be appropriate.
> 
>  libavcodec/x86/hpeldsp.asm | 38 ++++++++++++++++++++++++++++++++------
>  1 file changed, 32 insertions(+), 6 deletions(-)

This patch is not correct.

this code is not used if AV_CODEC_FLAG_BITEXACT is set, because it is
not bit exact ...

Also the case where the off by 1 error occurs is a rare corner case,
Compared to the errors introduced by the IDCT this is not significant

If you want to optimize the bit exact version, feel free to do so.
It may be faster to use xor -1 before and after avg for this though

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Into a blind darkness they enter who follow after the Ignorance,
they as if into a greater darkness enter who devote themselves
to the Knowledge alone. -- Isha Upanishad
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: not available
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20180428/0ae0b980/attachment.sig>