[FFmpeg-devel] [PATCH] Added QSV based VPP filter - second try
wm4
nfxjfg at googlemail.com
Thu Nov 5 12:19:08 CET 2015
On Thu, 5 Nov 2015 11:18:38 +0100
"Sven Dueking" <sven at nablet.com> wrote:
> From 0fe140c8204b4e09f577358b02295b76c2b5f3be Mon Sep 17 00:00:00 2001
> From: Sven Dueking <sven at nablet.com>
> Date: Thu, 5 Nov 2015 09:26:43 +0000
> Subject: [PATCH] added QSV VPP filter
>
> ---
> configure | 1 +
> doc/filters.texi | 174 ++++++++++++
> libavfilter/Makefile | 1 +
> libavfilter/allfilters.c | 1 +
> libavfilter/vf_qsv_vpp.c | 722 +++++++++++++++++++++++++++++++++++++++++++++++
> 5 files changed, 899 insertions(+)
> create mode 100644 libavfilter/vf_qsv_vpp.c
>
> diff --git a/configure b/configure
> index f770534..65cf013 100755
> --- a/configure
> +++ b/configure
> @@ -2843,6 +2843,7 @@ super2xsai_filter_deps="gpl"
> tinterlace_filter_deps="gpl"
> vidstabdetect_filter_deps="libvidstab"
> vidstabtransform_filter_deps="libvidstab"
> +vppqsv_filter_deps="libmfx"
> pixfmts_super2xsai_test_deps="super2xsai_filter"
> tinterlace_merge_test_deps="tinterlace_filter"
> tinterlace_pad_test_deps="tinterlace_filter"
> diff --git a/doc/filters.texi b/doc/filters.texi
> index 15ea77a..a3bf66c 100644
> --- a/doc/filters.texi
> +++ b/doc/filters.texi
> @@ -9089,6 +9089,180 @@ qp=2+2*sin(PI*qp)
> @end example
> @end itemize
>
> + at section vppqsv
> +
> +The VPP library is part of the Intel® Media SDK and exposes the media acceleration capabilities of Intel® platforms.
> +
> +Specifically, this library performs the following conversions:
> +
> + at itemize
> + at item
> + at emph{Color Conversion}: is the process of changing one type of color-encoded signal into another.
> +VPP supports several input color formats and NV12 as output format.
> +
> + at verbatim
> +Format Type |
> +-------------------------------------------------------------------------
> +Input (uncompressed) | YV12, NV12, YUY2, RGB4 (RGB 32-bit)
> +-------------------------------------------------------------------------
> +Output (uncompressed) | NV12
> + at end verbatim
> +
> + at item
> + at emph{Scaling}: is the process of resizing an image.
> +An image size can be changed in several ways,
> +VPP uses a separable 8-tap poly-phase scaling filter with adaptive filter ringing suppression.
> +
> + at item
> + at emph{Deinterlacing}: is the process of converting interlaced video into a non-interlaced form.
> +VPP uses a motion adaptive based deinterlacing algorithm.
> +
> + at verbatim
> +Input Frame Rate (FPS) | Output Frame Rate (FPS)
> + Interlaced | Progressive
> + | 23.976 | 25 | 29.97 | 30 | 50 | 59.94 | 60
> +-------------------------------------------------------------------------------------
> + 29.97 | Inverse | | | | | |
> + | Telecine | | x | | | |
> +-------------------------------------------------------------------------------------
> + 50 | | x | | | x | |
> +-------------------------------------------------------------------------------------
> + 59.94 | | | x | | | x |
> +-------------------------------------------------------------------------------------
> + 60 | | | | x | | | x
> +
> +x indicates a supported function
> + at end verbatim
> +
> + at item
> + at emph{Denoising}: is the process of removing noise from a video signal.
> +VPP uses a motion detection based temporal (adaptive to noise
> +history) and spatial (edge/ texture preserved) denoiser.
> +The spatial video denoiser is applied to each frame individually
> +and the temporal video denoiser to reduce noise between frames.
> +
> + at item
> + at emph{Frame Rate Conversion}: is the process of converting the frame rate of a video stream,
> +VPP supports frame drop/repetition frame rate conversion algorithm or frame interpolation
> +which means it will interpolate the new frames in case of increasing the frame rate.
> +
> + at item
> + at emph{Detail Enhancement}: is the process of enhancing the edge contrast to improve its sharpness.
> +VPP uses an adaptive detail enhancement algorithm to increase the edge sharpness.
> +
> + at end itemize
> +
> +The filter accepts the following option:
> +
> + at table @option
> +
> + at item deinterlace
> +Sets the deinterlacing mode.
> +
> +It accepts the following values:
> + at table @option
> + at item 0
> +No deinterlacing (default).
> + at item 1
> +BOB deinterlacing mode
> + at item 2
> +Advanced deinterlacing.
> +
> + at emph{Note}: Advanced deinterlacing uses reference frames and has better quality.
> +BOB is faster than advanced deinterlacing.
> +So user can select between speed and quality
> + at end table
> +
> + at item denoise
> +Enables denoise algorithm.
> +
> + at table @option
> + at item Value of 0-100 (inclusive) indicates the level of noise to remove. Default value is 0 (disabled).
> + at end table
> +
> + at item detail
> +Enables detail enhancement algorithm.
> +
> + at table @option
> + at item Value of 0-100 (inclusive) indicates the level of details to be enhanced. Default value is 0 (disabled).
> + at end table
> +
> + at item w
> + at item width
> +
> + at table @option
> + at item Output video width (Range [32,4096]).
> + at end table
> +
> + at item h
> + at item height
> +
> + at table @option
> + at item Output video height (Range [32,2304]).
> + at end table
> +
> + at item dpic
> +Sets output picture structure.
> +
> +It accepts the following values:
> + at table @option
> + at item 0
> +Interlaced top field first.
> + at item 1
> +Progressive (default).
> + at item 2
> +Interlaced bottom field first.
> + at end table
> +
> + at item fps
> +Sets output frame rate (frames/ seconds (fps)).
> +
> +It accepts the following values:
> + at table @option
> + at item 0
> +Input frame rate is used
> + at item else
> +Value is used.
> + at end table
> +
> + at item async_depth
> +
> + at table @option
> +
> + at item Specifies how many asynchronous operations VPP performs before the results are explicitly synchronized (Default Value is 4).
> +
> + at end table
> +
> + at item max_b_frames
> +
> + at table @option
> +
> + at item Specifies how many B frames are used, this value sets the number of extra surfaces used for re-ordering (Default Value is 3).
> +
> + at end table
> +
> + at emph{Warning}: The number of allocated frame surfaces depends on the number of async_depth and max_b_frames.
> +Wrong configuration could result in lost frames, since the VPP works asynchronous and could request multiple surfaces.
> +
> + at end table
> +
> +Limitations and known issues:
> +
> + at itemize
> + at item
> + at emph{Maximum supported resolution}: 4096x2304
> + at item
> + at emph{Minimum supported resolution}: 32x32
> + at item
> + at emph{Frame size}: frame width must be
> +a multiple of 16, frame height must be a multiple of 16 for progressive frames and a multiple
> +of 32 otherwise.
> +
> + at emph{NOTE}: VPP checks feature availability on any given machine at
> +runtime. Availability of features depends on hardware capabilities as well as driver version.
> +
> + at end itemize
> +
> @section random
>
> Flush video frames from internal cache of frames into a random order.
> diff --git a/libavfilter/Makefile b/libavfilter/Makefile
> index 1b23085..7a32cae 100644
> --- a/libavfilter/Makefile
> +++ b/libavfilter/Makefile
> @@ -241,6 +241,7 @@ OBJS-$(CONFIG_VFLIP_FILTER) += vf_vflip.o
> OBJS-$(CONFIG_VIDSTABDETECT_FILTER) += vidstabutils.o vf_vidstabdetect.o
> OBJS-$(CONFIG_VIDSTABTRANSFORM_FILTER) += vidstabutils.o vf_vidstabtransform.o
> OBJS-$(CONFIG_VIGNETTE_FILTER) += vf_vignette.o
> +OBJS-$(CONFIG_VPPQSV_FILTER) += vf_qsv_vpp.o
> OBJS-$(CONFIG_VSTACK_FILTER) += vf_stack.o framesync.o
> OBJS-$(CONFIG_W3FDIF_FILTER) += vf_w3fdif.o
> OBJS-$(CONFIG_WAVEFORM_FILTER) += vf_waveform.o
> diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c
> index a538b81..8e8f379 100644
> --- a/libavfilter/allfilters.c
> +++ b/libavfilter/allfilters.c
> @@ -262,6 +262,7 @@ void avfilter_register_all(void)
> REGISTER_FILTER(VIDSTABDETECT, vidstabdetect, vf);
> REGISTER_FILTER(VIDSTABTRANSFORM, vidstabtransform, vf);
> REGISTER_FILTER(VIGNETTE, vignette, vf);
> + REGISTER_FILTER(VPPQSV, vppqsv, vf);
> REGISTER_FILTER(VSTACK, vstack, vf);
> REGISTER_FILTER(W3FDIF, w3fdif, vf);
> REGISTER_FILTER(WAVEFORM, waveform, vf);
> diff --git a/libavfilter/vf_qsv_vpp.c b/libavfilter/vf_qsv_vpp.c
> new file mode 100644
> index 0000000..ff305e2
> --- /dev/null
> +++ b/libavfilter/vf_qsv_vpp.c
> @@ -0,0 +1,722 @@
> +/*
> + * Intel MediaSDK Quick Sync Video VPP filter
> + *
> + * copyright (c) 2015 Sven Dueking
> + *
> + * This file is part of FFmpeg.
> + *
> + * Libav is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU Lesser General Public
> + * License as published by the Free Software Foundation; either
> + * version 2.1 of the License, or (at your option) any later version.
> + *
> + * Libav is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> + * Lesser General Public License for more details.
> + *
> + * You should have received a copy of the GNU Lesser General Public
> + * License along with Libav; if not, write to the Free Software
> + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
> + */
> +
> +#include <mfx/mfxvideo.h>
> +#include <mfx/mfxplugin.h>
> +
> +#include "avfilter.h"
> +#include "internal.h"
> +#include "formats.h"
> +
> +#include "libavutil/avstring.h"
> +#include "libavutil/error.h"
> +#include "libavutil/opt.h"
> +#include "libavutil/pixfmt.h"
> +#include "libavutil/time.h"
> +#include "libavcodec/avcodec.h"
> +#include "libavcodec/qsv_internal.h"
> +
> +
> +/**
> + * ToDo :
> + *
> + * - cropping
> + */
I'd remove this, before it stays there for years.
> +
> +#define VPP_ZERO_MEMORY(VAR) { memset(&VAR, 0, sizeof(VAR)); }
This macro seems very silly. Better do:
struct type var = {0};
....
var = (struct type){0};
(Doesn't work for arrays.)
> +
> +// number of video enhancement filters (denoise, procamp, detail, video_analysis, image stab)
> +#define ENH_FILTERS_COUNT 5
> +
> +typedef struct {
> + const AVClass *class;
> +
> + AVFilterContext *ctx;
> +
> + mfxSession session;
> + QSVSession internal_qs;
> +
> + AVRational framerate; // target framerate
> +
> + mfxFrameSurface1 **in_surface;
> + mfxFrameSurface1 **out_surface;
> +
> + mfxFrameAllocRequest req[2]; // [0] - in, [1] - out
> +
> + int num_surfaces_in; // input surfaces
> + int num_surfaces_out; // output surfaces
> +
> + unsigned char * surface_buffers_out; // output surface buffer
> +
> + char *load_plugins;
> +
> + mfxVideoParam* pVppParam;
> +
> + int cur_out_idx;
> +
> + /* VPP extension */
> + mfxExtBuffer* pExtBuf[1+ENH_FILTERS_COUNT];
> + mfxExtVppAuxData extVPPAuxData;
> +
> + /* Video Enhancement Algorithms */
> + mfxExtVPPDeinterlacing deinterlace_conf;
> + mfxExtVPPFrameRateConversion frc_conf;
> + mfxExtVPPDenoise denoise_conf;
> + mfxExtVPPDetail detail_conf;
> +
> + int out_width;
> + int out_height;
> +
> + int dpic; // destination picture structure
> + // -1 = unknown
> + // 0 = interlaced top field first
> + // 1 = progressive
> + // 2 = interlaced bottom field first
> +
> + int deinterlace; // deinterlace mode : 0=off, 1=bob, 2=advanced
> + int denoise; // enable denoise algorithm. Level is the optional value from the interval [0; 100]
> + int detail; // enable detail enhancement algorithm.
> + // level is the optional value from the interval [0; 100]
> + int async_depth; // async dept used by encoder
> + int max_b_frames; // maxiumum number of b frames used by encoder
> +
> + int frame_number;
> +
> + int use_frc; // use framerate conversion
> +
> + int num_of_out_surfaces;
> +
> +} VPPContext;
> +
> +#define OFFSET(x) offsetof(VPPContext, x)
> +#define FLAGS AV_OPT_FLAG_VIDEO_PARAM|AV_OPT_FLAG_FILTERING_PARAM
> +
> +static const AVOption vpp_options[] = {
> + { "deinterlace", "deinterlace mode: 0=off, 1=bob, 2=advanced", OFFSET(deinterlace), AV_OPT_TYPE_INT, {.i64=0}, 0, MFX_DEINTERLACING_ADVANCED, .flags = FLAGS },
> + { "denoise", "denoise level [0, 100]", OFFSET(denoise), AV_OPT_TYPE_INT, {.i64=0}, 0, 100, .flags = FLAGS },
> + { "detail", "detail enhancement level [0, 100]", OFFSET(detail), AV_OPT_TYPE_INT, {.i64=0}, 0, 100, .flags = FLAGS },
> + { "w", "Output video width", OFFSET(out_width), AV_OPT_TYPE_INT, {.i64=0}, 0, 4096, .flags = FLAGS },
> + { "width", "Output video width", OFFSET(out_width), AV_OPT_TYPE_INT, {.i64=0}, 0, 4096, .flags = FLAGS },
> + { "h", "Output video height", OFFSET(out_height), AV_OPT_TYPE_INT, {.i64=0}, 0, 2304, .flags = FLAGS },
> + { "height", "Output video height : ", OFFSET(out_height), AV_OPT_TYPE_INT, {.i64=0}, 0, 2304, .flags = FLAGS },
Might make sense to move them to the start, so that positional filter
arguments work? (I don't know.)
> + { "dpic", "dest pic struct: 0=tff, 1=progressive [default], 2=bff", OFFSET(dpic), AV_OPT_TYPE_INT, {.i64 = 1 }, 0, 2, .flags = FLAGS },
> + { "fps", "A string describing desired output framerate", OFFSET(framerate), AV_OPT_TYPE_VIDEO_RATE, { .str = "0" }, .flags = FLAGS },
> + { "async_depth", "Maximum processing parallelism [default = 4]", OFFSET(async_depth), AV_OPT_TYPE_INT, { .i64 = ASYNC_DEPTH_DEFAULT }, 0, INT_MAX, .flags = FLAGS },
> + { "max_b_frames","Maximum number of b frames [default = 3]", OFFSET(max_b_frames), AV_OPT_TYPE_INT, { .i64 = 3 }, 0, INT_MAX, .flags = FLAGS },
Uh, what do b_frames do outside of the decoder? Filters always get
frames in display order. This makes no sense to me.
> + { NULL }
> +};
> +
> +AVFILTER_DEFINE_CLASS(vpp);
> +
> +static int get_bpp(unsigned int fourcc)
> +{
> + switch (fourcc) {
> + case MFX_FOURCC_YUY2:
> + return 16;
> + case MFX_FOURCC_RGB4:
> + return 32;
> + }
> + return 12;
> +}
> +
> +static int option_id_to_mfx_pic_struct(int id)
> +{
> + switch (id) {
> + case 0:
> + return MFX_PICSTRUCT_FIELD_TFF;
> + case 1:
> + return MFX_PICSTRUCT_PROGRESSIVE;
> + case 2:
> + return MFX_PICSTRUCT_FIELD_BFF;
> + }
> + return MFX_PICSTRUCT_UNKNOWN;
> +}
> +
> +static int get_chroma_fourcc(unsigned int fourcc)
> +{
> + switch (fourcc) {
> + case MFX_FOURCC_YUY2:
> + return MFX_CHROMAFORMAT_YUV422;
> + case MFX_FOURCC_RGB4:
> + return MFX_CHROMAFORMAT_YUV444;
> + }
> + return MFX_CHROMAFORMAT_YUV420;
> +}
> +
> +static int avframe_id_to_mfx_pic_struct(AVFrame * pic)
> +{
> + if (!pic->interlaced_frame)
> + return MFX_PICSTRUCT_PROGRESSIVE;
> +
> + if (pic->top_field_first == 1)
> + return MFX_PICSTRUCT_FIELD_TFF;
> +
> + return MFX_PICSTRUCT_FIELD_BFF;
> +}
> +
> +static int query_formats(AVFilterContext *ctx)
> +{
> + static const enum AVPixelFormat pix_fmts[] = {
> + AV_PIX_FMT_YUV420P,
> + AV_PIX_FMT_NV12,
> + AV_PIX_FMT_YUV422P,
> + AV_PIX_FMT_RGB32,
> + AV_PIX_FMT_NONE
> + };
> +
> + return ff_set_common_formats(ctx, ff_make_format_list(pix_fmts));
> +}
> +
> +static av_cold int init_vpp_param(AVFilterLink *inlink , AVFrame * pic)
> +{
> + AVFilterContext *ctx = inlink->dst;
> + VPPContext *vpp= ctx->priv;
> +
> + // input data
> + if (inlink->format == AV_PIX_FMT_YUV420P)
> + vpp->pVppParam->vpp.In.FourCC = MFX_FOURCC_YV12;
> + else if (inlink->format == AV_PIX_FMT_YUV422P)
> + vpp->pVppParam->vpp.In.FourCC = MFX_FOURCC_YUY2;
> + else if (inlink->format == AV_PIX_FMT_NV12)
> + vpp->pVppParam->vpp.In.FourCC = MFX_FOURCC_NV12;
> + else
> + vpp->pVppParam->vpp.In.FourCC = MFX_FOURCC_RGB4;
> +
> + vpp->pVppParam->vpp.In.ChromaFormat = get_chroma_fourcc(vpp->pVppParam->vpp.In.FourCC);
> + vpp->pVppParam->vpp.In.CropX = 0;
> + vpp->pVppParam->vpp.In.CropY = 0;
> + vpp->pVppParam->vpp.In.CropW = inlink->w;
> + vpp->pVppParam->vpp.In.CropH = inlink->h;
> + vpp->pVppParam->vpp.In.PicStruct = avframe_id_to_mfx_pic_struct(pic);
> + vpp->pVppParam->vpp.In.FrameRateExtN = inlink->frame_rate.num;
> + vpp->pVppParam->vpp.In.FrameRateExtD = inlink->frame_rate.den;
> + vpp->pVppParam->vpp.In.BitDepthLuma = 8;
> + vpp->pVppParam->vpp.In.BitDepthChroma = 8;
> +
> + // width must be a multiple of 16
> + // height must be a multiple of 16 in case of frame picture and a multiple of 32 in case of field picture
> + vpp->pVppParam->vpp.In.Width = FFALIGN(inlink->w, 16);
> + vpp->pVppParam->vpp.In.Height =
> + (MFX_PICSTRUCT_PROGRESSIVE == vpp->pVppParam->vpp.In.PicStruct) ?
> + FFALIGN(inlink->h, 16) :
> + FFALIGN(inlink->h, 32);
Well, I hope this doesn't over-read the input AVFrames if the AVFrames
are not aligned. Some AVFrames might still be padded (if they're output
by a decoder which does so), but you can't rely on this.
> +
> + // output data
> + vpp->pVppParam->vpp.Out.FourCC = MFX_FOURCC_NV12;
> + vpp->pVppParam->vpp.Out.ChromaFormat = MFX_CHROMAFORMAT_YUV420;
> + vpp->pVppParam->vpp.Out.CropX = 0;
> + vpp->pVppParam->vpp.Out.CropY = 0;
> + vpp->pVppParam->vpp.Out.CropW = !vpp->out_width ? inlink->w : vpp->out_width;
> + vpp->pVppParam->vpp.Out.CropH = !vpp->out_height ? inlink->h : vpp->out_height;
> + vpp->pVppParam->vpp.Out.PicStruct = option_id_to_mfx_pic_struct(vpp->dpic);
> + vpp->pVppParam->vpp.Out.FrameRateExtN = vpp->framerate.num;
> + vpp->pVppParam->vpp.Out.FrameRateExtD = vpp->framerate.den;
> + vpp->pVppParam->vpp.Out.BitDepthLuma = 8;
> + vpp->pVppParam->vpp.Out.BitDepthChroma = 8;
> +
> + if ((vpp->pVppParam->vpp.In.FrameRateExtN / vpp->pVppParam->vpp.In.FrameRateExtD) !=
> + (vpp->pVppParam->vpp.Out.FrameRateExtN / vpp->pVppParam->vpp.Out.FrameRateExtD))
> + vpp->use_frc = 1;
> + else
> + vpp->use_frc = 0;
> +
> + // width must be a multiple of 16
> + // height must be a multiple of 16 in case of frame picture and a multiple of 32 in case of field picture
> + vpp->pVppParam->vpp.Out.Width = FFALIGN(vpp->pVppParam->vpp.Out.CropW, 16);
> + vpp->pVppParam->vpp.Out.Height =
> + (MFX_PICSTRUCT_PROGRESSIVE == vpp->pVppParam->vpp.Out.PicStruct) ?
> + FFALIGN(vpp->pVppParam->vpp.Out.CropH, 16) :
> + FFALIGN(vpp->pVppParam->vpp.Out.CropH, 32);
> +
> + vpp->pVppParam->IOPattern =
> + MFX_IOPATTERN_IN_SYSTEM_MEMORY | MFX_IOPATTERN_OUT_SYSTEM_MEMORY;
> +
> + av_log(ctx, AV_LOG_INFO, "In %dx%d %d fps\t Out %dx%d %d fps\n",
> + FFALIGN(vpp->pVppParam->vpp.In.Width, 32),
> + FFALIGN(vpp->pVppParam->vpp.In.Height, 32),
> + vpp->pVppParam->vpp.In.FrameRateExtN / vpp->pVppParam->vpp.In.FrameRateExtD,
> + FFALIGN(vpp->pVppParam->vpp.Out.Width, 32),
> + FFALIGN(vpp->pVppParam->vpp.Out.Height, 32),
> + vpp->pVppParam->vpp.Out.FrameRateExtN / vpp->pVppParam->vpp.Out.FrameRateExtD);
> +
> + if (vpp->use_frc == 1)
> + av_log(ctx, AV_LOG_INFO, "Framerate conversion enabled\n");
> +
> + return 0;
> +}
> +
> +static av_cold int init_surface(AVFilterLink *inlink)
> +{
> + AVFilterContext *ctx = inlink->dst;
> + VPPContext *vpp= ctx->priv;
> +
> + int i,j;
> + unsigned int width = 0;
> + unsigned int height = 0;
> + unsigned int surface_size = 0;
> +
> + vpp->num_surfaces_in = FFMAX(vpp->req[0].NumFrameSuggested, vpp->async_depth + vpp->max_b_frames + 1);
> + vpp->num_surfaces_out = FFMAX(vpp->req[1].NumFrameSuggested, vpp->num_of_out_surfaces);
> +
> + width = (unsigned int) FFALIGN(vpp->pVppParam->vpp.In.Width, 32);
> + height = (unsigned int) FFALIGN(vpp->pVppParam->vpp.In.Height, 32);
> + surface_size = width * height * get_bpp(vpp->pVppParam->vpp.In.FourCC) / 8;
> +
> + vpp->in_surface = av_mallocz(sizeof(char*) * vpp->num_surfaces_in);
> +
> + if (!vpp->in_surface) {
> + vpp->num_surfaces_out = 0;
> + vpp->num_surfaces_in = 0;
> + return ENOMEM;
Generally, all of these occurrences should be replaced with proper
ffmpeg-style error codes and error checks. Use AVERROR(ENOMEM), and use
"err < 0" as check whether a function failed. Make it consistent,
because the current state is confusing.
> + }
> +
> + for (i = 0; i < vpp->num_surfaces_in; i++) {
> + vpp->in_surface[i] = av_mallocz(sizeof(mfxFrameSurface1));
> +
> + if (!vpp->in_surface[i]) {
> + for (j = 0; j < i; j++) {
> + av_freep(&vpp->in_surface[j]);
> + }
> +
> + vpp->num_surfaces_out = 0;
> + vpp->num_surfaces_in = 0;
> + return ENOMEM;
> + }
> +
> + memset(vpp->in_surface[i], 0, sizeof(mfxFrameSurface1));
Pointless memset. av_mallocz does this already.
> + memcpy(&(vpp->in_surface[i]->Info), &(vpp->pVppParam->vpp.In), sizeof(mfxFrameInfo));
Uh, use an assignment?
> + }
> +
> + width = FFALIGN(vpp->pVppParam->vpp.Out.Width, 32);
> + height = FFALIGN(vpp->pVppParam->vpp.Out.Height, 32);
> + surface_size = width * height * 12 / 8;
> + vpp->surface_buffers_out = av_mallocz(surface_size * vpp->num_surfaces_out);
> +
> + if (!vpp->surface_buffers_out) {
> + vpp->num_surfaces_out = 0;
> + return ENOMEM;
> + }
> +
> + vpp->out_surface = av_mallocz(sizeof(char*) * vpp->num_surfaces_out);
> +
> + if (!vpp->out_surface) {
> + vpp->num_surfaces_out = 0;
> + return ENOMEM;
> + }
> +
> + for (i = 0; i < vpp->num_surfaces_out; i++) {
> + vpp->out_surface[i] = av_mallocz(sizeof(mfxFrameSurface1));
> +
> + if (!vpp->out_surface[i]) {
> + for (j = 0; j < i; j++) {
> + if (vpp->out_surface[j])
> + av_freep(&vpp->out_surface[j]);
> + }
> +
> + vpp->num_surfaces_out = 0;
> + return ENOMEM;
> + }
> +
> + memset(vpp->out_surface[i], 0, sizeof(mfxFrameSurface1));
> + memcpy(&(vpp->out_surface[i]->Info), &(vpp->pVppParam->vpp.Out), sizeof(mfxFrameInfo));
Same mistakes repeated.
> + vpp->out_surface[i]->Data.Y = &vpp->surface_buffers_out[surface_size * i];
> + vpp->out_surface[i]->Data.U = vpp->out_surface[i]->Data.Y + width * height;
> + vpp->out_surface[i]->Data.V = vpp->out_surface[i]->Data.U + 1;
> + vpp->out_surface[i]->Data.PitchLow = width;
> + }
> +
> + return 0;
> +}
> +
> +static av_cold int config_vpp(AVFilterLink *inlink, AVFrame * pic)
> +{
> + AVFilterContext *ctx = inlink->dst;
> + VPPContext *vpp= ctx->priv;
> +
> + int ret = 0;
> +
> + mfxVideoParam mfxParamsVideo;
> +
> + VPP_ZERO_MEMORY(mfxParamsVideo);
> + vpp->pVppParam = &mfxParamsVideo;
> +
> + init_vpp_param(inlink, pic);
> +
> + vpp->pVppParam->NumExtParam = 0;
> +
> + vpp->pVppParam->ExtParam = (mfxExtBuffer**)vpp->pExtBuf;
> +
> + if (vpp->deinterlace) {
> + memset(&vpp->deinterlace_conf, 0, sizeof(mfxExtVPPDeinterlacing));
> + vpp->deinterlace_conf.Header.BufferId = MFX_EXTBUFF_VPP_DEINTERLACING;
> + vpp->deinterlace_conf.Header.BufferSz = sizeof(mfxExtVPPDeinterlacing);
> + vpp->deinterlace_conf.Mode = vpp->deinterlace == 1 ? MFX_DEINTERLACING_BOB : MFX_DEINTERLACING_ADVANCED;
> +
> + vpp->pVppParam->ExtParam[vpp->pVppParam->NumExtParam++] = (mfxExtBuffer*)&(vpp->deinterlace_conf);
> + }
> +
> + if (vpp->denoise) {
> + memset(&vpp->denoise_conf, 0, sizeof(mfxExtVPPDenoise));
> + vpp->denoise_conf.Header.BufferId = MFX_EXTBUFF_VPP_DENOISE;
> + vpp->denoise_conf.Header.BufferSz = sizeof(mfxExtVPPDenoise);
> + vpp->denoise_conf.DenoiseFactor = vpp->denoise;
> +
> + vpp->pVppParam->ExtParam[vpp->pVppParam->NumExtParam++] = (mfxExtBuffer*)&(vpp->denoise_conf);
> + }
> +
> + if (vpp->detail) {
> + memset(&vpp->detail_conf, 0, sizeof(mfxExtVPPDetail));
> + vpp->detail_conf.Header.BufferId = MFX_EXTBUFF_VPP_DETAIL;
> + vpp->detail_conf.Header.BufferSz = sizeof(mfxExtVPPDetail);
> + vpp->detail_conf.DetailFactor = vpp->detail;
> +
> + vpp->pVppParam->ExtParam[vpp->pVppParam->NumExtParam++] = (mfxExtBuffer*)&(vpp->detail_conf);
> + }
> +
> + if (vpp->use_frc || !vpp->pVppParam->NumExtParam) {
> + memset(&vpp->frc_conf, 0, sizeof(mfxExtVPPFrameRateConversion));
> + vpp->frc_conf.Header.BufferId = MFX_EXTBUFF_VPP_FRAME_RATE_CONVERSION;
> + vpp->frc_conf.Header.BufferSz = sizeof(mfxExtVPPFrameRateConversion);
> + vpp->frc_conf.Algorithm = MFX_FRCALGM_PRESERVE_TIMESTAMP; // make optional
> +
> + vpp->pVppParam->ExtParam[vpp->pVppParam->NumExtParam++] = (mfxExtBuffer*)&(vpp->frc_conf);
> + }
> +
> + ret = MFXVideoVPP_Query(vpp->session, vpp->pVppParam, vpp->pVppParam);
> +
> + if (ret >= MFX_ERR_NONE) {
> + av_log(ctx, AV_LOG_INFO, "MFXVideoVPP_Query returned %d \n", ret);
> + } else {
> + av_log(ctx, AV_LOG_ERROR, "Error %d querying the VPP parameters\n", ret);
> + return ff_qsv_error(ret);
> + }
> +
> + memset(&vpp->req, 0, sizeof(mfxFrameAllocRequest) * 2);
sizeof(vpp->req)?
> + ret = MFXVideoVPP_QueryIOSurf(vpp->session, vpp->pVppParam, &vpp->req[0]);
> +
> + if (ret < 0) {
> + av_log(ctx, AV_LOG_ERROR, "Error querying the VPP IO surface\n");
> + return ff_qsv_error(ret);
> + }
> +
> + ret = init_surface(inlink);
> +
> + if (ret == ENOMEM) {
> + av_log(ctx, AV_LOG_ERROR, "Error %d initialize VPP surfaces\n", ret);
> + return ret;
> + }
> +
> + ret = MFXVideoVPP_Init(vpp->session, vpp->pVppParam);
> +
> + if (MFX_WRN_PARTIAL_ACCELERATION == ret) {
> + av_log(ctx, AV_LOG_WARNING, "VPP will work with partial HW acceleration\n");
> + } else if (ret < 0) {
> + av_log(ctx, AV_LOG_ERROR, "Error initializing VPP\n");
> + return ff_qsv_error(ret);
> + }
> +
> + return 0;
> +}
> +
> +static av_cold void free_surface(AVFilterLink *inlink)
> +{
> + AVFilterContext *ctx = inlink->dst;
> + VPPContext *vpp= ctx->priv;
> + unsigned int i;
> +
> + for (i = 0; i < vpp->num_surfaces_in; i++) {
> + if (vpp->in_surface[i])
> + av_freep(&vpp->in_surface[i]);
> + }
> +
> + if (vpp->in_surface)
> + av_freep(&vpp->in_surface);
Redundant ifs. (And more of them below.)
> +
> + for (i = 0; i < vpp->num_surfaces_out; i++) {
> + if (vpp->out_surface[i])
> + av_freep(&vpp->out_surface[i]);
> + }
> +
> + if (vpp->out_surface)
> + av_freep(&vpp->out_surface);
> +
> + if (vpp->surface_buffers_out)
> + av_free(vpp->surface_buffers_out);
> +
> + vpp->num_surfaces_in = 0;
> + vpp->num_surfaces_out = 0;
> +}
> +
> +static av_cold int config_input(AVFilterLink *inlink)
> +{
> + AVFilterLink *outlink = inlink->dst->outputs[0];
> + AVFilterContext *ctx = inlink->dst;
> + VPPContext *vpp= ctx->priv;
> +
> + vpp->out_width = FFALIGN(vpp->out_width, 16);
> + vpp->out_height = (vpp->dpic == 2) ?
> + FFALIGN(vpp->out_height, 16) :
> + FFALIGN(vpp->out_height, 32); // force 32 if unknown
> +
> + outlink->w = vpp->out_width;
> + outlink->h = vpp->out_height;
> +
> + if (vpp->framerate.den && vpp->framerate.num) {
> + outlink->frame_rate = vpp->framerate;
> + outlink->time_base = av_inv_q(vpp->framerate);
> + outlink->time_base.den = outlink->time_base.den * 1000;
> + } else {
> + vpp->framerate = inlink->frame_rate;
> + outlink->frame_rate = vpp->framerate;
> + outlink->time_base = av_inv_q(vpp->framerate);
> + outlink->time_base.den = outlink->time_base.den * 1000;
> + }
> +
> + outlink->format = AV_PIX_FMT_NV12;
> +
> + return 0;
> +}
> +
> +static int get_free_surface_index_in(AVFilterContext *ctx, mfxFrameSurface1 ** surface_pool, int pool_size)
> +{
> + if (surface_pool) {
> + for (mfxU16 i = 0; i < pool_size; i++) {
> + if (!surface_pool[i]->Data.Locked)
> + return i;
> + }
> + }
> +
> + av_log(ctx, AV_LOG_ERROR, "Error getting a free surface, pool size is %d\n", pool_size);
> + return MFX_ERR_NOT_FOUND;
> +}
> +
> +static int get_free_surface_index_out(AVFilterContext *ctx, mfxFrameSurface1 ** surface_pool, int pool_size)
> +{
> + VPPContext *vpp = ctx->priv;
> +
> + if (surface_pool) {
> + for (mfxU16 i = 0; i < pool_size; i++) {
> + if (!surface_pool[i]->Data.Locked) {
> + if (i == vpp->cur_out_idx) {
> + if (vpp->cur_out_idx == pool_size - 1)
> + vpp->cur_out_idx = 0;
> + else
> + vpp->cur_out_idx ++;
> +
> + return i;
> + }
> + }
> + }
> + }
> +
> + av_log(ctx, AV_LOG_ERROR, "Error getting a free surface, pool size is %d\n", pool_size);
> + return MFX_ERR_NOT_FOUND;
> +}
> +
> +static int filter_frame(AVFilterLink *inlink, AVFrame *picref)
> +{
> + AVFilterContext *ctx = inlink->dst;
> + VPPContext *vpp = ctx->priv;
> +
> + mfxSyncPoint sync = NULL;
> + int ret = 0;
> + int filter_frame_ret = 0;
> +
> + AVFilterLink *outlink = inlink->dst->outputs[0];
> +
> + AVFrame *out;
> +
> + int in_idx = 0;
> + int out_idx = 0;
> +
> + if (!vpp->frame_number) {
> + ret = config_vpp(inlink, picref);
> +
> + if (ret)
> + return AVERROR(ENOMEM);
> + }
> +
> + do {
> + in_idx = get_free_surface_index_in(ctx, vpp->in_surface, vpp->num_surfaces_in);
> + out_idx = get_free_surface_index_out(ctx, vpp->out_surface, vpp->num_surfaces_out);
> +
> + if (in_idx == MFX_ERR_NOT_FOUND || out_idx == MFX_ERR_NOT_FOUND) {
> + av_frame_free(&picref);
> + return AVERROR(ENOMEM);
> + }
> +
> + out = ff_get_video_buffer(outlink,
> + vpp->out_surface[out_idx]->Info.Width,
> + vpp->out_surface[out_idx]->Info.Height);
> + if (!out) {
> + av_frame_free(&picref);
> + return AVERROR(ENOMEM);
> + }
> +
> + av_frame_copy_props(out, picref);
> +
> + // init in surface
> + if (inlink->format == AV_PIX_FMT_NV12) {
> + vpp->in_surface[in_idx]->Data.Y = picref->data[0];
> + vpp->in_surface[in_idx]->Data.VU = picref->data[1];
> + vpp->in_surface[in_idx]->Data.PitchLow = picref->linesize[0];
> + } else if (inlink->format == AV_PIX_FMT_YUV420P) {
> + vpp->in_surface[in_idx]->Data.Y = picref->data[0];
> + vpp->in_surface[in_idx]->Data.U = picref->data[1];
> + vpp->in_surface[in_idx]->Data.V = picref->data[2];
> + vpp->in_surface[in_idx]->Data.PitchLow = picref->linesize[0];
> + } else if (inlink->format == AV_PIX_FMT_YUV422P ) {
> + vpp->in_surface[in_idx]->Data.Y = picref->data[0];
> + vpp->in_surface[in_idx]->Data.U = picref->data[0] + 1;
> + vpp->in_surface[in_idx]->Data.V = picref->data[0] + 3;
> + vpp->in_surface[in_idx]->Data.PitchLow = picref->linesize[0];
> + } else if (inlink->format == AV_PIX_FMT_ARGB) {
> + vpp->in_surface[in_idx]->Data.B = picref->data[0];
> + vpp->in_surface[in_idx]->Data.G = picref->data[0] + 1;
> + vpp->in_surface[in_idx]->Data.R = picref->data[0] + 2;
> + vpp->in_surface[in_idx]->Data.A = picref->data[0] + 3;
> + vpp->in_surface[in_idx]->Data.PitchLow = picref->linesize[0];
> + }
> +
> + do {
> + ret = MFXVideoVPP_RunFrameVPPAsync(vpp->session, vpp->in_surface[in_idx], vpp->out_surface[out_idx], NULL, &sync);
> +
> + if (ret == MFX_ERR_MORE_SURFACE) {
> + break;
> + } else if (ret == MFX_WRN_DEVICE_BUSY) {
> + av_usleep(500);
What. Use proper event-based waiting.
> + continue;
> + } else if (ret == MFX_ERR_MORE_DATA) {
> + continue;
> + }
> + break;
> + } while ( 1 );
> +
> +
> + if (ret < 0 && ret != MFX_ERR_MORE_SURFACE) {
> + if (ret == MFX_ERR_MORE_DATA)
> + return 0;
> + av_log(ctx, AV_LOG_ERROR, "RunFrameVPPAsync %d\n", ret);
> + return ff_qsv_error(ret);
> + }
> +
> + if (ret == MFX_WRN_INCOMPATIBLE_VIDEO_PARAM) {
> + av_log(ctx, AV_LOG_WARNING,
> + "EncodeFrameAsync returned 'incompatible param' code\n");
> + }
> +
> + MFXVideoCORE_SyncOperation(vpp->session, sync, 60000);
> +
> + out->interlaced_frame = vpp->dpic == 1 || vpp->dpic == 3;
> + out->top_field_first = vpp->dpic == 1;
> +
> + out->data[0] = vpp->out_surface[out_idx]->Data.Y;
> + out->data[1] = vpp->out_surface[out_idx]->Data.VU;
> + out->linesize[0] = vpp->out_surface[out_idx]->Info.Width;
> + out->linesize[1] = vpp->out_surface[out_idx]->Info.Width;
> +
> + out->pts = ((av_rescale_q(0, inlink->time_base,
> + outlink->time_base) + vpp->frame_number) * 1000);
> +
> + filter_frame_ret = ff_filter_frame(inlink->dst->outputs[0], out);
> +
> + if (filter_frame_ret < 0) {
> + av_log(ctx, AV_LOG_ERROR, "ff_filter_frame %d\n", filter_frame_ret);
> + return filter_frame_ret;
> + }
> +
> + vpp->frame_number++;
> +
> + } while (ret == MFX_ERR_MORE_SURFACE);
> +
> + av_frame_free(&picref);
> +
> + return 0;
> +}
> +
> +static av_cold int vpp_init(AVFilterContext *ctx)
> +{
> + VPPContext *vpp= ctx->priv;
> +
> + AVCodecContext *avctx = (AVCodecContext *)ctx;
> +
> + if (!vpp->session) {
> + int ret = ff_qsv_init_internal_session(avctx, &vpp->internal_qs,
> + vpp->load_plugins);
> +
> + if (ret < 0)
> + return ret;
> +
> + vpp->session = vpp->internal_qs.session;
> + }
> +
> + vpp->frame_number = 0;
> + vpp->cur_out_idx = 0;
> +
> + vpp->num_of_out_surfaces = 1;
> +
> + vpp->in_surface = NULL;
> + vpp->out_surface = NULL;
> + vpp->surface_buffers_out = NULL;
> +
> + return 0;
> +}
> +
> +static av_cold void vpp_uninit(AVFilterContext *ctx)
> +{
> + VPPContext *vpp= ctx->priv;
> +
> + free_surface(ctx->inputs[0]);
> +
> + ff_qsv_close_internal_session(&vpp->internal_qs);
> +}
> +
> +static const AVFilterPad vpp_inputs[] = {
> + {
> + .name = "default",
> + .type = AVMEDIA_TYPE_VIDEO,
> + .config_props = config_input,
> + .filter_frame = filter_frame,
> +
> + },
> + { NULL }
> +};
> +
> +static const AVFilterPad vpp_outputs[] = {
> + {
> + .name = "default",
> + .type = AVMEDIA_TYPE_VIDEO,
> + },
> + { NULL }
> +};
> +
> +AVFilter ff_vf_vppqsv = {
> + .name = "vppqsv",
> + .description = NULL_IF_CONFIG_SMALL("Quick Sync Video VPP."),
> + .priv_size = sizeof(VPPContext),
> + .query_formats = query_formats,
> + .init = vpp_init,
> + .uninit = vpp_uninit,
> + .inputs = vpp_inputs,
> + .outputs = vpp_outputs,
> + .priv_class = &vpp_class,
> +};
This is just a partial review. I probably missed a bunch of things, and
I don't know the MFX API at all.
More information about the ffmpeg-devel
mailing list