[FFmpeg-devel] [PATCH] libavutil/hwcontext_dxva2: Add check for possible errors from GetAdapterDisplayModeEx

[wm4]

On Wed,  3 Jan 2018 15:22:15 -0800
Humberto Ribeiro <mont3z.claro5 at gmail.com> wrote:

> This prevents a possible crash in CreateDeviceEx when using faulty response from GetAdapterDisplayModeEx and allows ffmpeg to fallback to classic d3d9.
> ---
>  libavutil/hwcontext_dxva2.c | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
> 
> diff --git a/libavutil/hwcontext_dxva2.c b/libavutil/hwcontext_dxva2.c
> index 2ddd4be..44ebdbc 100644
> --- a/libavutil/hwcontext_dxva2.c
> +++ b/libavutil/hwcontext_dxva2.c
> @@ -485,7 +485,11 @@ static int dxva2_device_create9ex(AVHWDeviceContext *ctx, UINT adapter)
>      if (FAILED(hr))
>          return AVERROR_UNKNOWN;
>  
> -    IDirect3D9Ex_GetAdapterDisplayModeEx(d3d9ex, adapter, &modeex, NULL);
> +    hr = IDirect3D9Ex_GetAdapterDisplayModeEx(d3d9ex, adapter, &modeex, NULL);
> +    if (FAILED(hr)) {
> +        IDirect3D9Ex_Release(d3d9ex);
> +        return AVERROR_UNKNOWN;
> +    }
>  
>      d3dpp.BackBufferFormat = modeex.Format;
>  

I'll apply this and this trivial fix in an hour or so (after testing
it):

--- a/libavutil/hwcontext_dxva2.c
+++ b/libavutil/hwcontext_dxva2.c
@@ -485,6 +485,7 @@ static int dxva2_device_create9ex(AVHWDeviceContext *ctx, UINT adapter)
     if (FAILED(hr))
         return AVERROR_UNKNOWN;
 
+    modeex.Size = sizeof(D3DDISPLAYMODEEX);
     hr = IDirect3D9Ex_GetAdapterDisplayModeEx(d3d9ex, adapter, &modeex, NULL);
     if (FAILED(hr)) {
         IDirect3D9Ex_Release(d3d9ex);