[FFmpeg-devel] [PATCH] Prevents crash from CreateDeviceEx and proper fallback to classic d3d9.

[Hendrik Leppkes]

On Wed, Jan 3, 2018 at 11:18 PM, wm4 <nfxjfg at googlemail.com> wrote:
> On Wed,  3 Jan 2018 13:59:34 -0800
> Humberto Ribeiro <mont3z.claro5 at gmail.com> wrote:
>
>> ---
>>  libavutil/hwcontext_dxva2.c | 6 +++++-
>>  1 file changed, 5 insertions(+), 1 deletion(-)
>>
>> diff --git a/libavutil/hwcontext_dxva2.c b/libavutil/hwcontext_dxva2.c
>> index 2ddd4be..44ebdbc 100644
>> --- a/libavutil/hwcontext_dxva2.c
>> +++ b/libavutil/hwcontext_dxva2.c
>> @@ -485,7 +485,11 @@ static int dxva2_device_create9ex(AVHWDeviceContext *ctx, UINT adapter)
>>      if (FAILED(hr))
>>          return AVERROR_UNKNOWN;
>>
>> -    IDirect3D9Ex_GetAdapterDisplayModeEx(d3d9ex, adapter, &modeex, NULL);
>> +    hr = IDirect3D9Ex_GetAdapterDisplayModeEx(d3d9ex, adapter, &modeex, NULL);
>> +    if (FAILED(hr)) {
>> +        IDirect3D9Ex_Release(d3d9ex);
>> +        return AVERROR_UNKNOWN;
>> +    }
>>
>>      d3dpp.BackBufferFormat = modeex.Format;
>>
>
> LGTM, but when does it happen? Why would creation of IDirect3D9Ex
> succeed, but this method fail? The strangest thing being that
> CreateDeviceEx apparently fails, even though modeex is
> zero-initialized and doesn't contain any pointers.
>

I noticed something about this some time ago - D3DDISPLAYMODEEX should
not be entirely zero-initialized before this call, instead its "Size"
member should be set to sizeof(D3DDISPLAYMODEEX) - see
https://msdn.microsoft.com/en-us/library/windows/desktop/bb172549(v=vs.85).aspx
Currently it always causes IDirect3D9Ex_GetAdapterDisplayModeEx to
fail due to that (at least on my hardware), but it didn't seem to have
any impact on the later create device call in my case.

- Hendrik