[FFmpeg-devel] [PATCH]lavc/dds: Fix GRAY8A decoding

[Paul B Mahol]

On 4/14/16, Carl Eugen Hoyos <cehoyos at ag.or.at> wrote:
> Hi!
>
> Attached patch fixes the original sample of ticket #4667 for me.
>
> Please comment, Carl Eugen
>

> diff --git a/libavcodec/dds.c b/libavcodec/dds.c
> index 9577b67..f364de7 100644
> --- a/libavcodec/dds.c
> +++ b/libavcodec/dds.c
> @@ -356,6 +356,10 @@ static int parse_pixel_format(AVCodecContext *avctx)
>          /* 16 bpp */
>          else if (bpp == 16 && r == 0xff && g == 0 && b == 0 && a == 0xff00)
>              avctx->pix_fmt = AV_PIX_FMT_YA8;
> +        else if (bpp == 16 && r == 0xff00 && g == 0 && b == 0 && a == 0xff) {
> +            avctx->pix_fmt = AV_PIX_FMT_YA8;
> +            ctx->postproc = DDS_SWAP_ALPHA;

Why? You disabled postproc lines bellow.

> +        }
>          else if (bpp == 16 && r == 0xffff && g == 0 && b == 0 && a == 0)
>              avctx->pix_fmt = AV_PIX_FMT_GRAY16LE;
>          else if (bpp == 16 && r == 0xf800 && g == 0x7e0 && b == 0x1f && a == 0)
> @@ -387,8 +391,6 @@ static int parse_pixel_format(AVCodecContext *avctx)
>          ctx->postproc = DDS_NORMAL_MAP;
>      else if (ycocg_classic && !ctx->compressed)
>          ctx->postproc = DDS_RAW_YCOCG;
> -    else if (avctx->pix_fmt == AV_PIX_FMT_YA8)
> -        ctx->postproc = DDS_SWAP_ALPHA;
>
>      /* ATI/NVidia variants sometimes add swizzling in bpp. */
>      switch (bpp) {
> @@ -697,8 +699,7 @@ static int dds_decode(AVCodecContext *avctx, void *data,
>      if (avctx->pix_fmt == AV_PIX_FMT_BGRA ||
>          avctx->pix_fmt == AV_PIX_FMT_RGBA ||
>          avctx->pix_fmt == AV_PIX_FMT_RGB0 ||
> -        avctx->pix_fmt == AV_PIX_FMT_BGR0 ||
> -        avctx->pix_fmt == AV_PIX_FMT_YA8)
> +        avctx->pix_fmt == AV_PIX_FMT_BGR0)
>          run_postproc(avctx, frame);
>
>      /* Frame is ready to be output. */
> diff --git a/tests/ref/fate/dds-ya b/tests/ref/fate/dds-ya
> index d6527c6..01f7894 100644
> --- a/tests/ref/fate/dds-ya
> +++ b/tests/ref/fate/dds-ya
> @@ -1,2 +1,2 @@
>  #tb 0: 1/25
> -0,          0,          0,        1,    28160, 0xd3981fcb
> +0,          0,          0,        1,    28160, 0x5aa41fcb

I believe this shouldn't change, do you have explanation why it should change?