[FFmpeg-devel] [PATCH v2 1/2] qsvdec: add support for HW_DEVICE_CTX method
Xiang, Haihao
haihao.xiang at intel.com
Mon Aug 2 08:52:36 EEST 2021
On Fri, 2021-07-30 at 08:18 +0000, Soft Works wrote:
> > -----Original Message-----
> > From: ffmpeg-devel <ffmpeg-devel-bounces at ffmpeg.org> On Behalf Of
> > Haihao Xiang
> > Sent: Thursday, 29 July 2021 09:04
> > To: ffmpeg-devel at ffmpeg.org
> > Cc: Haihao Xiang <haihao.xiang at intel.com>
> > Subject: [FFmpeg-devel] [PATCH v2 1/2] qsvdec: add support for
> > HW_DEVICE_CTX method
> >
> > This allows user set hw_device_ctx instead of hw_frames_ctx for QSV
> > decoders, hence we may remove the ad-hoc libmfx setup code from
> > FFmpeg.
> >
> > "-hwaccel_output_format format" is applied to QSV decoders after removing
> > the ad-hoc libmfx code. To keep compatibility with old commandlines, the
> > default format is set to AV_PIX_FMT_QSV. User should set "-
> > hwaccel_output_format qsv" explicitly if AV_PIX_FMT_QSV is expected.
> >
> > User may use the normal way to set device for QSV decoders now.
> > "-qsv_device device" is deprecated and will be removed from FFmpeg.
> >
> > For example:
> >
> > $> ffmpeg -init_hw_device vaapi=va:/dev/dri/card0 -init_hw_device
> > qsv=hw at va -hwaccel qsv -c:v h264_qsv -i input.h264 -f null -
>
> Hi Haihao,
>
> Why so complicated with double device initialization?
> The regular way is this:
>
> ffmpeg -init_hw_device "qsv=dev:hw_any,child_device=/dev/dri/card0" -hwaccel
> qsv -c:v h264_qsv -i input.h264 -f null -
>
> Or for Windows:
>
> ffmpeg -init_hw_device "qsv=dev:hw_any,child_device=1" -hwaccel qsv -c:v
> h264_qsv -i input.h264 -f null -
After applying this patch, the above commands may work too, actually a child
device is created internally in this way, then a QSV device is derived from the
child device implicitly. However deriving a device explicitly creates a
connection between source device and derived device. We may derive a new device
of the source type from the derived device.
e.g.
The command below works
$> ffmpeg -init_hw_device vaapi=intel:/dev/dri/renderD128 -init_hw_device
qsv=qsv at intel -hwaccel qsv -c:v h264_qsv -i input.h264 -vf
hwmap=derive_device=vaapi,scale_vaapi -f null -
but the command below doesn't work
$> ffmpeg -init_hw_device qsv=dev:hw_any,child_device=/dev/dri/renderD128
-hwaccel qsv -c:v h264_qsv -i ~/graphics/movies/720p.mp4 -vf
hwmap=derive_device=vaapi,scale_vaapi -f null -
(We may apply http://ffmpeg.org/pipermail/ffmpeg-devel/2021-February/276695.html
to create a connection between two devices too, but this change breaks 'make
fate-hwdevice V=2').
>
> >
> > /dev/dri/renderD128 is actually open for h264_qsv decoder without this
> > patch. After applying this patch, /dev/dri/card0 is used.
> > ---
> > v2: Rebase the patchset against the latest master branch and resolve a
> > conflict.
> >
> > fftools/Makefile | 1 -
> > fftools/ffmpeg.h | 1 -
> > fftools/ffmpeg_hw.c | 12 +++++
> > fftools/ffmpeg_opt.c | 54 +++++++++++++++++++-- fftools/ffmpeg_qsv.c
> > > 109 -------------------------------------------
> >
> > libavcodec/qsvdec.c | 31 +++++++++++-
> > 6 files changed, 92 insertions(+), 116 deletions(-) delete mode 100644
> > fftools/ffmpeg_qsv.c
> >
> > diff --git a/fftools/Makefile b/fftools/Makefile index
> > 5affaa3f56..5234932ab0
> > 100644
> > --- a/fftools/Makefile
> > +++ b/fftools/Makefile
> > @@ -10,7 +10,6 @@ ALLAVPROGS =
> > $(AVBASENAMES:%=%$(PROGSSUF)$(EXESUF))
> > ALLAVPROGS_G = $(AVBASENAMES:%=%$(PROGSSUF)_g$(EXESUF))
> >
> > OBJS-ffmpeg += fftools/ffmpeg_opt.o
> > fftools/ffmpeg_filter.o
> > fftools/ffmpeg_hw.o
> > -OBJS-ffmpeg-$(CONFIG_LIBMFX) += fftools/ffmpeg_qsv.o
> > ifndef CONFIG_VIDEOTOOLBOX
> > OBJS-ffmpeg-$(CONFIG_VDA) += fftools/ffmpeg_videotoolbox.o
> > endif
> > diff --git a/fftools/ffmpeg.h b/fftools/ffmpeg.h index
> > d9c0628657..d2dd7ca092 100644
> > --- a/fftools/ffmpeg.h
> > +++ b/fftools/ffmpeg.h
> > @@ -61,7 +61,6 @@ enum HWAccelID {
> > HWACCEL_AUTO,
> > HWACCEL_GENERIC,
> > HWACCEL_VIDEOTOOLBOX,
> > - HWACCEL_QSV,
> > };
> >
> > typedef struct HWAccel {
> > diff --git a/fftools/ffmpeg_hw.c b/fftools/ffmpeg_hw.c index
> > fc4a5d31d6..043e4c5863 100644
> > --- a/fftools/ffmpeg_hw.c
> > +++ b/fftools/ffmpeg_hw.c
> > @@ -339,6 +339,18 @@ int hw_device_setup_for_decode(InputStream *ist)
> > } else if (ist->hwaccel_id == HWACCEL_GENERIC) {
> > type = ist->hwaccel_device_type;
> > dev = hw_device_get_by_type(type);
> > +
> > + // When "-qsv_device device" is used, an internal QSV device
> > named
> > + // as "__qsv_device" is created and another QSV device is
> > created
> > + // if "-init_hw_device qsv=name at name" is used. There are 2 QSV
> > devices
> > + // if both "-qsv_device device" and "-init_hw_device
> > qsv=name at name"
> > + // are used, hw_device_get_by_type(AV_HWDEVICE_TYPE_QSV)
> > returns NULL.
> > + // To keep back-compatibility with the removed ad-hoc libmfx
> > setup
> > code,
> > + // call hw_device_get_by_name("__qsv_device") to select the
> > internal QSV
> > + // device. This will be removed once -qsv_device is no longer
> > supported.
> > + if (!dev && type == AV_HWDEVICE_TYPE_QSV)
> > + dev = hw_device_get_by_name("__qsv_device");
> > +
> > if (!dev)
> > err = hw_device_init_from_type(type, NULL, &dev);
> > } else {
> > diff --git a/fftools/ffmpeg_opt.c b/fftools/ffmpeg_opt.c index
> > 1b43bab9fc..4ddc6c939b 100644
> > --- a/fftools/ffmpeg_opt.c
> > +++ b/fftools/ffmpeg_opt.c
> > @@ -137,9 +137,6 @@ static const char *const opt_name_enc_time_bases[]
> > = {"enc_time_base"
> > const HWAccel hwaccels[] = {
> > #if CONFIG_VIDEOTOOLBOX
> > { "videotoolbox", videotoolbox_init, HWACCEL_VIDEOTOOLBOX,
> > AV_PIX_FMT_VIDEOTOOLBOX }, -#endif -#if CONFIG_LIBMFX
> > - { "qsv", qsv_init, HWACCEL_QSV, AV_PIX_FMT_QSV },
> > #endif
> > { 0 },
> > };
> > @@ -571,6 +568,49 @@ static int opt_vaapi_device(void *optctx, const char
> > *opt, const char *arg) } #endif
> >
> > +#if CONFIG_QSV
> > +static int opt_qsv_device(void *optctx, const char *opt, const char
> > +*arg) { #if CONFIG_VAAPI
> > + const char *prefix = "vaapi=__qsv_child_device:"; #elif
> > +CONFIG_DXVA2
> > + const char *prefix = "dxva2=__qsv_child_device:"; #else
> > + const char *prefix = NULL;
> > +#endif
>
> Same like for the command line. You can directly initialize a QSV context
> with a single operation.
According to the above explanation, I prefer to derive device explicitly.
>
> [...]
>
> > + if (avctx->hw_device_ctx && !avctx->hw_frames_ctx && ret ==
> > AV_PIX_FMT_QSV) {
> > + AVHWFramesContext *hwframes_ctx;
> > + AVQSVFramesContext *frames_hwctx;
> > +
> > + avctx->hw_frames_ctx =
> > + av_hwframe_ctx_alloc(avctx->hw_device_ctx);
> > +
> > + if (!avctx->hw_frames_ctx) {
> > + av_log(avctx, AV_LOG_ERROR, "av_hwframe_ctx_alloc failed\n");
> > + return AVERROR(ENOMEM);
> > + }
> > +
> > + hwframes_ctx = (AVHWFramesContext*)avctx->hw_frames_ctx-
> > > data;
> >
> > + frames_hwctx = hwframes_ctx->hwctx;
> > + hwframes_ctx->width = FFALIGN(avctx->coded_width, 32);
> > + hwframes_ctx->height = FFALIGN(avctx->coded_height, 32);
> > + hwframes_ctx->format = AV_PIX_FMT_QSV;
> > + hwframes_ctx->sw_format = avctx->sw_pix_fmt;
> > + hwframes_ctx->initial_pool_size = 64 + avctx->extra_hw_frames;
>
>
> I'm not sure whether 64 is a good default choice. I had to reduce this to 32
> when processing 4k video with OpenCL interop (memory sharing), otherwise I got
> OOM errors. (with video mem set to the maximum of 1G in the BIOS)
>
64 is used in ffmpeg_qsv.c, we have another patchset to optimize the memory
usage in qsv path and will submit it once this patchset is merged.
>
> Besides the above comments, this looks good to me in general. It is the right
> step to go forward IMO.
>
> Just the double device initialization should not be required (it isn't now)
> when it comes to the selection between D3D9 and D3D11. I am simply using a
> device parameter for this, and a command line looks like this:
>
> ffmpeg -init_hw_device "qsv=dev:hw_any,child_device=1,dx11=1" -hwaccel qsv
> -c:v h264_qsv -i input.h264 -f null -
>
According to the above explanation, you can still use this way for the selection
between D3D9 and D3D11.
Thanks
Haihao
>
> But let's get this done done first..
> softworkz
>
>
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
More information about the ffmpeg-devel
mailing list