[FFmpeg-devel] [PATCH v2] avcodec/libjxlenc: support negative linesizes

[Andreas Rheinhardt]

Leo Izen:
> libjxl doesn't support negative strides, but JPEG XL has an orientation
> flag inside the codestream. We can use this to work around the library
> limitation, by taking the absolute value of the negative row stride,
> sending the image up-side-down, and telling the library that the image
> has a vertical-flip orientation.
> 
> Signed-off-by: Leo Izen <leo.izen at gmail.com>
> ---
>  libavcodec/libjxlenc.c | 12 ++++++++++--
>  1 file changed, 10 insertions(+), 2 deletions(-)
> 
> diff --git a/libavcodec/libjxlenc.c b/libavcodec/libjxlenc.c
> index 67be8a01ca..4ddd5f9f2c 100644
> --- a/libavcodec/libjxlenc.c
> +++ b/libavcodec/libjxlenc.c
> @@ -259,6 +259,7 @@ static int libjxl_encode_frame(AVCodecContext *avctx, AVPacket *pkt, const AVFra
>      size_t available = ctx->buffer_size;
>      size_t bytes_written = 0;
>      uint8_t *next_out = ctx->buffer;
> +    const uint8_t *data;
>  
>      ret = libjxl_init_jxl_encoder(avctx);
>      if (ret) {
> @@ -303,6 +304,7 @@ static int libjxl_encode_frame(AVCodecContext *avctx, AVPacket *pkt, const AVFra
>  
>      /* bitexact lossless requires there to be no XYB transform */
>      info.uses_original_profile = ctx->distance == 0.0;
> +    info.orientation = frame->linesize[0] >= 0 ? JXL_ORIENT_IDENTITY : JXL_ORIENT_FLIP_VERTICAL;
>  
>      if (JxlEncoderSetBasicInfo(ctx->encoder, &info) != JXL_ENC_SUCCESS) {
>          av_log(avctx, AV_LOG_ERROR, "Failed to set JxlBasicInfo\n");
> @@ -383,9 +385,15 @@ static int libjxl_encode_frame(AVCodecContext *avctx, AVPacket *pkt, const AVFra
>      }
>  
>      jxl_fmt.endianness = JXL_NATIVE_ENDIAN;
> -    jxl_fmt.align = frame->linesize[0];
> +    if (frame->linesize[0] >= 0) {
> +        jxl_fmt.align = frame->linesize[0];
> +        data = frame->data[0];
> +    } else {
> +        jxl_fmt.align = -frame->linesize[0];
> +        data = frame->data[0] - jxl_fmt.align * (info.ysize - 1);

Please don't rely on the type of jxl_fmt.align here (which is out of our
control). E.g. in the future it may be that libjxl supports only 32bit
align values (i.e. uses uint32_t or so for it), but that we support
64bit (ptrdiff_t) linesizes and allocations, so that jxl_fmt.align *
(info.ysize - 1) may overflow this even when -linesize fits into
jxl_fmt.align.
(Very unlikely given that align is size_t and they will likely not go
back from this, but it could happen.)

> +    }
>  
> -    if (JxlEncoderAddImageFrame(ctx->options, &jxl_fmt, frame->data[0], jxl_fmt.align * info.ysize) != JXL_ENC_SUCCESS) {
> +    if (JxlEncoderAddImageFrame(ctx->options, &jxl_fmt, data, jxl_fmt.align * info.ysize) != JXL_ENC_SUCCESS) {
>          av_log(avctx, AV_LOG_ERROR, "Failed to add Image Frame\n");
>          return AVERROR_EXTERNAL;
>      }