[FFmpeg-devel] [PATCH] Prevents crash from CreateDeviceEx and proper fallback to classic d3d9.

[Mont3z Claros]

Thx for this inside. Unfortunately my application still crashes even
after setting Size in modeex.
In my case, IDirect3D9Ex_GetAdapterDisplayModeEx always returns
D3DERR_INVALIDCALL.

Humberto

On Wed, Jan 3, 2018 at 4:17 PM, wm4 <nfxjfg at googlemail.com> wrote:
> On Thu, 4 Jan 2018 00:26:01 +0100
> Hendrik Leppkes <h.leppkes at gmail.com> wrote:
>
>> On Wed, Jan 3, 2018 at 11:18 PM, wm4 <nfxjfg at googlemail.com> wrote:
>> > On Wed,  3 Jan 2018 13:59:34 -0800
>> > Humberto Ribeiro <mont3z.claro5 at gmail.com> wrote:
>> >
>> >> ---
>> >>  libavutil/hwcontext_dxva2.c | 6 +++++-
>> >>  1 file changed, 5 insertions(+), 1 deletion(-)
>> >>
>> >> diff --git a/libavutil/hwcontext_dxva2.c b/libavutil/hwcontext_dxva2.c
>> >> index 2ddd4be..44ebdbc 100644
>> >> --- a/libavutil/hwcontext_dxva2.c
>> >> +++ b/libavutil/hwcontext_dxva2.c
>> >> @@ -485,7 +485,11 @@ static int dxva2_device_create9ex(AVHWDeviceContext *ctx, UINT adapter)
>> >>      if (FAILED(hr))
>> >>          return AVERROR_UNKNOWN;
>> >>
>> >> -    IDirect3D9Ex_GetAdapterDisplayModeEx(d3d9ex, adapter, &modeex, NULL);
>> >> +    hr = IDirect3D9Ex_GetAdapterDisplayModeEx(d3d9ex, adapter, &modeex, NULL);
>> >> +    if (FAILED(hr)) {
>> >> +        IDirect3D9Ex_Release(d3d9ex);
>> >> +        return AVERROR_UNKNOWN;
>> >> +    }
>> >>
>> >>      d3dpp.BackBufferFormat = modeex.Format;
>> >>
>> >
>> > LGTM, but when does it happen? Why would creation of IDirect3D9Ex
>> > succeed, but this method fail? The strangest thing being that
>> > CreateDeviceEx apparently fails, even though modeex is
>> > zero-initialized and doesn't contain any pointers.
>> >
>>
>> I noticed something about this some time ago - D3DDISPLAYMODEEX should
>> not be entirely zero-initialized before this call, instead its "Size"
>> member should be set to sizeof(D3DDISPLAYMODEEX) - see
>> https://msdn.microsoft.com/en-us/library/windows/desktop/bb172549(v=vs.85).aspx
>> Currently it always causes IDirect3D9Ex_GetAdapterDisplayModeEx to
>> fail due to that (at least on my hardware), but it didn't seem to have
>> any impact on the later create device call in my case.
>
> Oh, so possibly entirely our/my fault.
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel