[FFmpeg-devel] [PATCH v7 1/8] libavcodec/qsv: enabling d3d11va support, added mfxhdlpair

Thu Jul 15 12:32:28 EEST 2021

> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces at ffmpeg.org> On Behalf Of
> Xiang, Haihao
> Sent: Thursday, 15 July 2021 09:56
> To: ffmpeg-devel at ffmpeg.org
> Subject: Re: [FFmpeg-devel] [PATCH v7 1/8] libavcodec/qsv: enabling
> d3d11va support, added mfxhdlpair
> 
> On Thu, 2021-07-15 at 06:43 +0000, Soft Works wrote:
> > > -----Original Message-----
> > > From: ffmpeg-devel <ffmpeg-devel-bounces at ffmpeg.org> On Behalf Of
> > > Xiang, Haihao
> > > Sent: Thursday, 15 July 2021 07:43
> > > To: ffmpeg-devel at ffmpeg.org
> > > Subject: Re: [FFmpeg-devel] [PATCH v7 1/8] libavcodec/qsv: enabling
> > > d3d11va support, added mfxhdlpair
> > >
> > > On Thu, 2021-07-15 at 03:49 +0000, Soft Works wrote:
> > > > > -----Original Message-----
> > > > > From: ffmpeg-devel <ffmpeg-devel-bounces at ffmpeg.org> On
> Behalf
> > > > > Of James Almer
> > > > > Sent: Thursday, 15 July 2021 05:21
> > > > > To: ffmpeg-devel at ffmpeg.org
> > > > > Subject: Re: [FFmpeg-devel] [PATCH v7 1/8] libavcodec/qsv:
> > > > > enabling d3d11va support, added mfxhdlpair
> > > > >
> > > > > On 7/15/2021 12:10 AM, Soft Works wrote:
> > > > > >
> > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: ffmpeg-devel <ffmpeg-devel-bounces at ffmpeg.org> On
> > >
> > > Behalf
> > > > > > > Of Xiang, Haihao
> > > > > > > Sent: Thursday, 15 July 2021 04:35
> > > > > > > To: ffmpeg-devel at ffmpeg.org
> > > > > > > Subject: Re: [FFmpeg-devel] [PATCH v7 1/8] libavcodec/qsv:
> > > > > > > enabling d3d11va support, added mfxhdlpair
> > > > > > >
> > > > > > > On Tue, 2021-07-13 at 11:53 -0300, James Almer wrote:
> > > > > > > > On 11/14/2020 1:49 PM, Max Dmitrichenko wrote:
> > > > > > > > > On Tue, Nov 3, 2020 at 7:47 PM Artem Galin
> > > > > > > > > <artem.galin at gmail.com>
> > > > > > >
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Adding DX11 relevant device type checks and adjusting
> > > > > > > > > > callbacks with proper MediaSDK pair type support.
> > > > > > > > > >
> > > > > > > > > > Extending structure for proper MediaSDK pair type support.
> > > > > > > > > >
> > > > > > > > > > Signed-off-by: Artem Galin <artem.galin at intel.com> .....
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Patchset obviously closes the gap of DirectX 11 support
> > > > > > > > > and already checked with several apps that use FFMPEG.
> > > > > > > > >
> > > > > > > > > Any particular review comments should be yet expected?
> > > > > > > > >
> > > > > > > > > Changes seem to be straight forward and incorporate prev.
> > >
> > > comments.
> > > > > > > > >
> > > > > > > > > thank in advance
> > > > > > > > >
> > > > > > > > > regards
> > > > > > > > > Max
> > > > > > > >
> > > > > > > > There are some issues pointed out by Soft Works related to
> > > > > > > > switching the default to DX11 breaking existing command
> > > > > > > > lines with DX9, plus an OpenCL<->QSV interop regression
> > > > > > > > this would introduce that according to him should be easy to fix.
> > > > > > > >
> > > > > > > > If those are addressed, the set should be good.
> > > > > > >
> > > > > > > If we may apply
> > > > > > > http://ffmpeg.org/pipermail/ffmpeg-devel/2021-
> > > > > > > June/281778.html
> > > > > > > (qsvdec: add support for HW_DEVICE_CTX method) first, we
> > > > > > > should be able to use the common device stuff to initialize
> > > > > > > d3d11va device for QSV and needn't use child_device_type to
> > > > > > > specify child
> > >
> > > device.
> > > > > >
> > > > > > There's no doubt that the new method will be better, but the
> > > > > > point is not
> > > > >
> > > > > to break existing command lines. From my experience, ffmpeg
> > > > > usually avoids to break established command line patterns, which
> > > > > is a good thing
> > >
> > > imo.
> > > > > >
> > > > > > Anything more official than my perception? Please feel free to
> > > > > > chime in... ;-)
> > > > >
> > > > > That patch appears to attempt to maintain support for existing
> > > > > command lines for a deprecation period, something we've done
> > > > > before with other hwaccels like cuvid, so if it's really better,
> > > > > it's fine and
> > >
> > > acceptable.
> > > >
> > > > ^
> > > >
> > > > Hi James,
> > > >
> > > > "That patch" that preserves compatibility doesn't exist yet. We
> > > > need to make sure that command lines using the '--child_device'
> > > > parameter will continue to work, while Haihao's patch will be the
> "future way":
> > > >
> > > > > -init_hw_device d3d11va=d3d11va:xxx -init_hw_device
> qsv at d3d11va
> > > >
> > > > The new behaviour that is introduced by this patch is that an
> > > > initialized hw device can now be applied to qsv decoders
> > > > (currently only to filter graphs and qsv encoders).
> > >
> > > '-init_hw_device qsv=qsv:hw,child_device=xxx' still works,
> >
> > Sure - that needs to work anyway. What I meant is the -qsv_device
> > parameter, which is a global-scope param and takes a device path on
> > Linux or the D3D adapter ID on Windows.
> >
> > ffmpeg -hwaccel qsv -qsv_device /dev/dri/renderD128 -c:v h264_qsv -i
> > input.mp4 -c:v h264_qsv output.mp4
> 
> Yes, -qsv_device still works, I added opt_qsv_device() to handle qsv_device
> option.
> 
> >
> > For my implementation I had added an additional global parameter (
> > --qsv_use_dx11). When it is set, the -qsv_device parameter indicates a
> > DXGI adapter id rather than a D3D adapter. But let's forget about
> > that. The init_hw_device approach is much better, and -qsv_device
> > should just keep working like before.
> >
> > I'm currently rebasing all our patches to the latest ffmpeg; probably
> > tomorrow I'll check out your patch.
> 
> Thanks for checking out the patch.
> 
> >
> > > > Additionally, the default should remain to be D3D9 (if none of the
> > > > above is
> > > > specified) as laid out earlier - again for maintaining
> > > > compatibility but also for better robustness and performance.
> > >
> > > We may refine Artem's patch to make sure the default is D3D9 on
> > > Microsoft windows
> >
> > Sounds good.
> >
> > > when both dxva2 and d3d11va are enabled if applying my patch first.
> >
> > Not sure what you mean by that.
> 
> What I meant is we needn't use parameter child_device_type (which was
> added in Artem's patch) to specify the child device type after applying my
> patch, but we should make sure the default child device type is DXVA2 when
> both DXVA2 and D3D11VA are available however user doesn't specify the
> type. E.g. '- init_hw_device qsv=qsv:hw' should work with DXVA2, not
> D3D11VA if both DXVA2 and D3D11VA are availabe.

Ah, thanks for the clarification, but that doesn't work out. The parameter - child_device_type is essential:

1. The device numbers that you specify via qsv_device or child_device are totally different between D3D9 and D3D11/DXGI. In the latter case, you specify the index of a graphics adapter/hardware device, no matter how many displays you _can_ connect to it and no matter how many displays _are_ connected.
In case of D3D9, you specify the index in a collection of connected displays  which belong to the context of the active user session.
=> without specifying a device type, an index number is meaningless

2. It's important that the behavior is deterministic: when you specify D3D9, you need to be able to rely on getting D3D9 (or fail otherwise), same for D3D11. 

3. When you want to combine a DXVA2/D3D11VA decoder with a qsv encoder or filter context, you need to make a choice between these two, which also means that you need to be able to control D3D9/11 at the QSV side.

4. OpenCL interop support is different between D3D9 and D3D11 and you might need to adjust your command line accordingly. That's only possible when you can be sure up-front about which kind of context you'll get.

5. Drivers for Gen 3, 4 and 5 are no longer getting updates for QSV. There are still driver updates being published, but the QSV/MSDK part remains unchanged. These versions have issues that Artem's patch doesn't address

Summing up: a selection mechanism between D3D9 and D3D11 is  inevitable, but it must not be automatic.
Of course - it would be possible to create a DXVA2 or D3D11VA context first and derive from that one, but that's technically not the same and too complicated for the average user.

Kind Regards,
softworkz