[FFmpeg-devel] [PATCH v2 1/2] qsvdec: add support for HW_DEVICE_CTX method

Soft Works softworkz at hotmail.com
Mon Aug 2 13:27:56 EEST 2021



> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces at ffmpeg.org> On Behalf Of
> Xiang, Haihao
> Sent: Monday, 2 August 2021 10:25
> To: ffmpeg-devel at ffmpeg.org
> Subject: Re: [FFmpeg-devel] [PATCH v2 1/2] qsvdec: add support for
> HW_DEVICE_CTX method
> 
> On Mon, 2021-08-02 at 07:11 +0000, Soft Works wrote:
> > > -----Original Message-----
> > > From: ffmpeg-devel <ffmpeg-devel-bounces at ffmpeg.org> On Behalf Of
> > > Xiang, Haihao
> > > Sent: Monday, 2 August 2021 07:53
> > > To: ffmpeg-devel at ffmpeg.org
> > > Subject: Re: [FFmpeg-devel] [PATCH v2 1/2] qsvdec: add support for
> > > HW_DEVICE_CTX method
> > >
> > > On Fri, 2021-07-30 at 08:18 +0000, Soft Works wrote:
> > > > > -----Original Message-----
> > > > > From: ffmpeg-devel <ffmpeg-devel-bounces at ffmpeg.org> On
> Behalf
> > > > > Of Haihao Xiang
> > > > > Sent: Thursday, 29 July 2021 09:04
> > > > > To: ffmpeg-devel at ffmpeg.org
> > > > > Cc: Haihao Xiang <haihao.xiang at intel.com>
> > > > > Subject: [FFmpeg-devel] [PATCH v2 1/2] qsvdec: add support for
> > > > > HW_DEVICE_CTX method
> > > > >
> > > > > This allows user set hw_device_ctx instead of hw_frames_ctx for
> > > > > QSV decoders, hence we may remove the ad-hoc libmfx setup code
> > > > > from FFmpeg.
> > > > >
> > > > > "-hwaccel_output_format format" is applied to QSV decoders after
> > > > > removing the ad-hoc libmfx code. To keep compatibility with old
> > > > > commandlines, the default format is set to AV_PIX_FMT_QSV. User
> > > > > should set "- hwaccel_output_format qsv" explicitly if
> > > > > AV_PIX_FMT_QSV
> > >
> > > is expected.
> > > > >
> > > > > User may use the normal way to set device for QSV decoders now.
> > > > > "-qsv_device device" is deprecated and will be removed from
> FFmpeg.
> > > > >
> > > > > For example:
> > > > >
> > > > > $> ffmpeg -init_hw_device vaapi=va:/dev/dri/card0
> > > > > -init_hw_device qsv=hw at va -hwaccel qsv -c:v h264_qsv -i
> > > > > input.h264 -f null -
> > > >
> > > > Hi Haihao,
> > > >
> > > > Why so complicated with double device initialization?
> > > > The regular way is this:
> > > >
> > > > ffmpeg -init_hw_device
> "qsv=dev:hw_any,child_device=/dev/dri/card0"
> > > > -hwaccel qsv -c:v h264_qsv -i input.h264 -f null -
> > > >
> > > > Or for Windows:
> > > >
> > > > ffmpeg -init_hw_device "qsv=dev:hw_any,child_device=1" -hwaccel
> > > > qsv -c:v h264_qsv -i input.h264 -f null -
> > >
> > > After applying this patch, the above commands may work too, actually
> > > a child device is created internally in this way, then a QSV device
> > > is derived from the child device implicitly. However deriving a
> > > device explicitly creates a connection between source device and
> > > derived device. We may derive a new device of the source type from
> > > the derived device.
> >
> > "may work too" is not sufficient. The existing and documented method
> > must continue to work.
> 
> Could you please provide an example which is broken by this patch? I don't
> seeregression in my testing.

What I wanted to point out is that "may work too" is not a good attitude 
towards functionality used by millions of users (most of which without 
even knowing).

Besides compatibility and command line complexity, there's also a technical
problem with your approach.

The following command line does not work:

ffmpeg -init_hw_device dxva2=dx:1 -init_hw_device qsv=qd at dx -hwaccel qsv -c:v h264_qsv -i INPUT -filter_complex "testsrc2=s=256x256,hwupload=extra_hw_frames=8[subs];[0:0][subs]overlay_qsv" -c:v h264_qsv output.mp4

ERROR: A hardware device reference is required to upload frames to

It can be fixed by setting filter_hw_device:

ffmpeg -init_hw_device dxva2=dx:1 -init_hw_device qsv=qd at dx -hwaccel qsv -c:v h264_qsv -i INPUT -filter_hw_device qd -filter_complex "testsrc2=s=256x256,hwupload=extra_hw_frames=8[subs];[0:0][subs]overlay_qsv" -c:v h264_qsv

This is not required when using the regular QSV initialization:

ffmpeg -init_hw_device qsv=qd1:hw2,child_device=1 -hwaccel qsv -c:v h264_qsv -i INPUT -filter_complex "testsrc2=s=256x256,hwupload=extra_hw_frames=8[subs];[0:0][subs]overlay_qsv" -c:v h264_qsv output.mp4


Needing to add filter_hw_device seems to be easy - but only when you know that you need to do that. And that's the kind of point where many users would give up.


But now let's look at the classic initialization using qsv_device:
(btw: I don't think that this should be deprecated at all; it's a more convenient way - easier to remember and not that much to type)

ffmpeg -qsv_device 1 -hwaccel qsv -c:v h264_qsv -i INPUT -filter_complex "testsrc2=s=256x256,hwupload=extra_hw_frames=8[subs];[0:0][subs]overlay_qsv" -c:v h264_qsv output.mp4

This doesn't work with your patch either, and it's not possible to set the filter device that way.
Though, it will work when you change your internal initialization to use the regular single-shot initialization.

It's not exactly a regression because it didn't work before, but that's how it should work (finally) thanks to this patch.

Kind regards,
softworkz 






The issues here are:

1. Wasting (previous) video memory

The above command causes the creation of 



> 
> >
> > Besides that, it doesn't make sense to me to start advertising those
> > overcomplicated command lines with double device initialization as the
> > default way for using QSV.
> 
> This patch doesn't advertise that double device initialization is the default
> way for QSV. I just use it as an example in the commit log, I may add another
> example in the commit log if needed.
> 
> >
> > >
> > > e.g.
> > >
> > > The command below works
> > >
> > > $> ffmpeg -init_hw_device vaapi=intel:/dev/dri/renderD128
> > > -init_hw_device qsv=qsv at intel -hwaccel qsv -c:v h264_qsv -i
> > > input.h264 -vf hwmap=derive_device=vaapi,scale_vaapi -f null -
> > >
> > > but the command below doesn't work
> > >
> > > $> ffmpeg -init_hw_device
> > > qsv=dev:hw_any,child_device=/dev/dri/renderD128
> > > -hwaccel qsv -c:v h264_qsv -i ~/graphics/movies/720p.mp4 -vf
> > > hwmap=derive_device=vaapi,scale_vaapi -f null -
> > >
> > > (We may apply http://ffmpeg.org/pipermail/ffmpeg-devel/2021-
> > > February/276695.html
> > >  to create a connection between two devices too, but this change
> > > breaks 'make fate-hwdevice V=2').
> >
> > I had done a similar patch quite a while ago, just forgot about it, so
> > yes, that patch is crucial as well.
> 
> Let's fix it in another patch if the above command is expected to work too.
> 
> Thanks
> Haihao
> 
> 
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email ffmpeg-devel-request at ffmpeg.org
> with subject "unsubscribe".


More information about the ffmpeg-devel mailing list