[FFmpeg-devel] [PATCH] d3d11va: use the proper slice index

[James Almer]

On 10/4/2016 11:47 AM, Steve Lhomme wrote:
> The slice index expected by D3D11VA is the one from the texture not from the
> array or texture/slices.
> 
> In VLC the slices we provide the decoder don't start from 0 and thus pictures
> appear in bogus order. With possible crashes and corruptions when using an
> invalid index.
> ---
>  libavcodec/dxva2.c | 9 +++++++++
>  1 file changed, 9 insertions(+)
> 
> diff --git a/libavcodec/dxva2.c b/libavcodec/dxva2.c
> index f68df86..f8801c9 100644
> --- a/libavcodec/dxva2.c
> +++ b/libavcodec/dxva2.c
> @@ -43,7 +43,16 @@ unsigned ff_dxva2_get_surface_index(const AVCodecContext *avctx,
>  
>      for (i = 0; i < DXVA_CONTEXT_COUNT(avctx, ctx); i++)
>          if (DXVA_CONTEXT_SURFACE(avctx, ctx, i) == surface)
> +#if CONFIG_D3D11VA
> +        {
> +            ID3D11VideoDecoderOutputView *pOut = DXVA_CONTEXT_SURFACE(avctx, ctx, i);
> +            D3D11_VIDEO_DECODER_OUTPUT_VIEW_DESC viewDesc;
> +            ID3D11VideoDecoderOutputView_GetDesc( pOut, &viewDesc );
> +            return viewDesc.Texture2D.ArraySlice;
> +        }
> +#elif CONFIG_DXVA2

This will break DXVA2 if D3D11 is also enabled.
The rest of the file supports both cases by doing something like

#if CONFIG_D3D11VA
        if (avctx->pix_fmt == AV_PIX_FMT_D3D11VA_VLD) {
            ...
        }
#endif
#if CONFIG_DXVA2
        if (avctx->pix_fmt == AV_PIX_FMT_DXVA2_VLD) {
            ...
        }
#endif

So the same should probably be done here.
You could also simplify all this by replacing the DXVA_CONTEXT_SURFACE macro,
which already does a avctx->pix_fmt check, with the corresponding contexts
for D3D11 and DXVA2.

>              return i;
> +#endif
>  
>      assert(0);
>      return 0;
>