[FFmpeg-devel] [PATCH v2 1/4] lavc/vaapi_encode_h265: Add P frame to GPB frame support for hevc_vaapi
Wang, Fei W
fei.w.wang at intel.com
Wed Feb 23 15:33:23 EET 2022
> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces at ffmpeg.org> On Behalf Of Wang,
> Fei W
> Sent: Tuesday, February 22, 2022 4:49 PM
> To: ffmpeg-devel at ffmpeg.org
> Subject: Re: [FFmpeg-devel] [PATCH v2 1/4] lavc/vaapi_encode_h265: Add P
> frame to GPB frame support for hevc_vaapi
>
> On Tue, 2022-02-22 at 05:46 +0000, Xiang, Haihao wrote:
> > On Mon, 2022-02-21 at 12:06 +0000, Mark Thompson wrote:
> > > On 21/02/2022 02:13, Fei Wang wrote:
> > > > From: Linjie Fu <linjie.fu at intel.com>
> > > >
> > > > Use GPB frames to replace regular P frames if backend driver does
> > > > not support it.
> > > >
> > > > - GPB:
> > > > Generalized P and B picture. P frames replaced by B frames
> > > > with
> > > > forward-predict only, L0 == L1. Normal B frames still have 2
> > > > different ref_lists and allow bi-prediction
> > > >
> > > > Signed-off-by: Linjie Fu <linjie.fu at intel.com>
> > > > Signed-off-by: Fei Wang <fei.w.wang at intel.com>
> > > > ---
> > > > libavcodec/vaapi_encode.c | 33
> > > > +++++++++++++++++++++++++++++++--
> > > > libavcodec/vaapi_encode.h | 1 +
> > > > libavcodec/vaapi_encode_h265.c | 15 +++++++++++++++
> > > > 3 files changed, 47 insertions(+), 2 deletions(-)
> > >
> > > This always fails immediately on current iHD
> > > (7e357b4bea76b2fe2522e6af41ae02ea69cec49e):
> > >
> > > $ ./ffmpeg_g -v 44 -y -hwaccel vaapi -hwaccel_output_format vaapi -i
> > > in.mp4 -an -c:v hevc_vaapi -low_power 1 out.mp4 ...
> > > [hevc_vaapi @ 0x560e81d45e80] Using input frames context (format
> > > vaapi) with
> > > hevc_vaapi encoder.
> > > [hevc_vaapi @ 0x560e81d45e80] Input surface format is nv12.
> > > [hevc_vaapi @ 0x560e81d45e80] Using VAAPI profile VAProfileHEVCMain
> > > (17).
> > > [hevc_vaapi @ 0x560e81d45e80] Using VAAPI entrypoint
> > > VAEntrypointEncSliceLP (8).
> > > [hevc_vaapi @ 0x560e81d45e80] Using VAAPI render target format
> > > YUV420 (0x1).
> > > [hevc_vaapi @ 0x560e81d45e80] Using CTU size 64x64, min CB size 8x8.
> > > [hevc_vaapi @ 0x560e81d45e80] No quality level set; using default
> > > (25).
> > > [hevc_vaapi @ 0x560e81d45e80] RC mode: CQP.
> > > [hevc_vaapi @ 0x560e81d45e80] RC quality: 25.
> > > [hevc_vaapi @ 0x560e81d45e80] RC framerate: 30000/1001 (29.97 fps).
> > > [hevc_vaapi @ 0x560e81d45e80] Use GPB B frames to replace regular P
> > > frames.
> > > [hevc_vaapi @ 0x560e81d45e80] Using intra, GPB-B-frames and B-
> > > frames (supported references: 3 / 3).
> > > [hevc_vaapi @ 0x560e81d45e80] All wanted packed headers available
> > > (wanted 0xd, found 0x1f).
> > > [hevc_vaapi @ 0x560e81d45e80] Using level 4.
> > > ...
> > > [hevc_vaapi @ 0x560e81d45e80] Failed to end picture encode issue:
> > > 24 (internal
> > > encoding error).
> > > [hevc_vaapi @ 0x560e81d45e80] Encode failed: -5.
> > > Video encoding failed
> > > ...
> > > $ cat /proc/cpuinfo | grep 'model name' | head -1
> > > model name : Intel(R) Core(TM) i7-1065G7 CPU @ 1.30GHz
> > > $ uname -v
> > > #1 SMP PREEMPT Debian 5.16.7-2 (2022-02-09)
> > >
> > > Do you get this too, or is your setup different somehow?
> >
> > Hi Mark,
> >
> > I tested this patchset with iHD
> > (7e357b4bea76b2fe2522e6af41ae02ea69cec49e) on CFL (low_power=0), RKL
> > and DG1, i965 on SKL, and didn't see this issue before.
> > This day I reproduced this issue on ICL. It seems iHD driver doesn't
> > return right values for ICL.
>
> Thanks Mark to report this issue on ICL.
>
> I tested on TGL and CFL before submitted this patch set, all looks good. I will
> check why this fail on ICL.
>
> And will fix your other comments together in next version.
>
> Fei
> Thanks
>
>
> >
> > > > diff --git a/libavcodec/vaapi_encode.c b/libavcodec/vaapi_encode.c
> > > > index 3bf379b1a0..95eca7c288 100644
> > > > --- a/libavcodec/vaapi_encode.c
> > > > +++ b/libavcodec/vaapi_encode.c
> > > > @@ -1845,6 +1845,30 @@ static av_cold int
> > > > vaapi_encode_init_gop_structure(AVCodecContext *avctx)
> > > > ref_l1 = attr.value >> 16 & 0xffff;
> > > > }
> > > >
> > > > + ctx->p_to_gpb = 0;
> > > > +
> > > > +#if VA_CHECK_VERSION(1, 9, 0)
> > > > + attr = (VAConfigAttrib) { VAConfigAttribPredictionDirection
> > > > };
> > > > + vas = vaGetConfigAttributes(ctx->hwctx->display,
> > > > + ctx->va_profile,
> > > > + ctx->va_entrypoint,
> > > > + &attr, 1);
> > >
> > > This probably shouldn't be done at all if the user has selected a
> > > codec without B-frames or a configuration which is intra-only,
> > > because the log message is confusing:
> > >
> > > [mjpeg_vaapi @ 0x55b90d72ee00] Driver does not report whether
> > > support GPB, use regular P frames.
> > > [mjpeg_vaapi @ 0x55b90d72ee00] Using intra frames only.
> > >
> > > > + if (vas != VA_STATUS_SUCCESS) {
> > > > + av_log(avctx, AV_LOG_WARNING, "Failed to query
> > > > prediction direction
> > > > "
> > > > + "attribute: %d (%s).\n", vas, vaErrorStr(vas));
> > >
> > > And fail?
> >
> > 4/4 also ignores the error. It would be better to handle the error in
> > the same way and update 4/4 too.
> >
> > > > + } else if (attr.value == VA_ATTRIB_NOT_SUPPORTED) {
> > > > + av_log(avctx, AV_LOG_VERBOSE, "Driver does not report
> > > > whether "
> > > > + "support GPB, use regular P frames.\n");
> > >
> > > "support GPB" is a strange thing to say. It's a constraint - any
> > > driver which supports B-frames will let you have the same thing in
> > > both RefPicLists, but some require it because they don't support
> > > P-frames.
> > >
> > > So maybe something like "Driver does not report any additional
> > > prediction constraints, using P-frames."?
> > >
> > > > + } else {
> > > > + if (attr.value & VA_PREDICTION_DIRECTION_BI_NOT_EMPTY) {
> > > > + ctx->p_to_gpb = 1;
> > > > + av_log(avctx, AV_LOG_VERBOSE, "Use GPB B frames to
> > > > replace "
> > > > + "regular P frames.\n");
> > >
> > > Maybe "Driver does not support P-frames, replacing them with B-
> > > frames."?
> > >
> > > > + } else
> > > > + av_log(avctx, AV_LOG_VERBOSE, "Use regular P
> > > > frames.\n");
> > > > + }
> > > > +#endif
> > > > +
> > > > if (ctx->codec->flags & FLAG_INTRA_ONLY ||
> > > > avctx->gop_size <= 1) {
> > > > av_log(avctx, AV_LOG_VERBOSE, "Using intra frames
> > > > only.\n"); @@ -1861,8 +1885,13 @@ static av_cold int
> > > > vaapi_encode_init_gop_structure(AVCodecContext *avctx)
> > > > ctx->p_per_i = INT_MAX;
> > > > ctx->b_per_p = 0;
> > > > } else {
> > > > - av_log(avctx, AV_LOG_VERBOSE, "Using intra, P- and B-
> > > > frames "
> > > > - "(supported references: %d / %d).\n", ref_l0,
> > > > ref_l1);
> > > > + if (ctx->p_to_gpb)
> > > > + av_log(avctx, AV_LOG_VERBOSE, "Using intra, GPB-B-
> > > > frames and "
> > > > + "B-frames (supported references: %d / %d).\n",
> > > > + ref_l0, ref_l1);
> > >
> > > Seems easier to just say intra and B-frames (though this isn't
> > > really adding anything to the message above - it's still effectively
> > > a P-frame referring to one previous frame, just using B-slices to do
> > > it in a strange way).
> > >
> > > > + else
> > > > + av_log(avctx, AV_LOG_VERBOSE, "Using intra, P- and B-
> > > > frames "
> > > > + "(supported references: %d / %d).\n", ref_l0,
> > > > ref_l1);
> > > > ctx->gop_size = avctx->gop_size;
> > > > ctx->p_per_i = INT_MAX;
> > > > ctx->b_per_p = avctx->max_b_frames; diff --git
> > > > a/libavcodec/vaapi_encode.h b/libavcodec/vaapi_encode.h index
> > > > b41604a883..61c5615eb8 100644
> > > > --- a/libavcodec/vaapi_encode.h
> > > > +++ b/libavcodec/vaapi_encode.h
> > > > @@ -328,6 +328,7 @@ typedef struct VAAPIEncodeContext {
> > > > int idr_counter;
> > > > int gop_counter;
> > > > int end_of_stream;
> > > > + int p_to_gpb;
> > > >
> > > > // Whether the driver supports ROI at all.
> > > > int roi_allowed;
> > > > diff --git a/libavcodec/vaapi_encode_h265.c
> > > > b/libavcodec/vaapi_encode_h265.c index e1dc53dfa9..ea45893508
> > > > 100644
> > > > --- a/libavcodec/vaapi_encode_h265.c
> > > > +++ b/libavcodec/vaapi_encode_h265.c
> > > > @@ -886,6 +886,7 @@ static int
> > > > vaapi_encode_h265_init_slice_params(AVCodecContext *avctx,
> > > > VAAPIEncodePictu
> > > > re *pic,
> > > > VAAPIEncodeSlice
> > > > *slice)
> > > > {
> > > > + VAAPIEncodeContext *ctx = avctx->priv_data;
> > > > VAAPIEncodeH265Context *priv = avctx->priv_data;
> > > > VAAPIEncodeH265Picture *hpic = pic->priv_data;
> > > > const H265RawSPS *sps = &priv->raw_sps;
> > > > @@ -908,6 +909,9 @@ static int
> > > > vaapi_encode_h265_init_slice_params(AVCodecContext *avctx,
> > > >
> > > > sh->slice_type = hpic->slice_type;
> > > >
> > > > + if (sh->slice_type == HEVC_SLICE_P && ctx->p_to_gpb)
> > > > + sh->slice_type = HEVC_SLICE_B;
> > > > +
> > > > sh->slice_pic_order_cnt_lsb = hpic->pic_order_cnt &
> > > > (1 << (sps->log2_max_pic_order_cnt_lsb_minus4 + 4)) - 1;
> > > >
> > > > @@ -1066,6 +1070,9 @@ static int
> > > > vaapi_encode_h265_init_slice_params(AVCodecContext *avctx,
> > > > av_assert0(pic->type == PICTURE_TYPE_P ||
> > > > pic->type == PICTURE_TYPE_B);
> > > > vslice->ref_pic_list0[0] = vpic->reference_frames[0];
> > > > + if (ctx->p_to_gpb && pic->type == PICTURE_TYPE_P)
> > > > + // Reference for GPB B-frame, L0 == L1
> > > > + vslice->ref_pic_list1[0] = vpic-
> > > > >reference_frames[0];
> > > > }
> > > > if (pic->nb_refs >= 2) {
> > > > // Forward reference for B-frame.
> > > > @@ -1073,6 +1080,14 @@ static int
> > > > vaapi_encode_h265_init_slice_params(AVCodecContext *avctx,
> > > > vslice->ref_pic_list1[0] = vpic->reference_frames[1];
> > > > }
> > > >
> > > > + if (pic->type == PICTURE_TYPE_P && ctx->p_to_gpb) {
> > > > + vslice->slice_type = HEVC_SLICE_B;
> > > > + for (i = 0; i < FF_ARRAY_ELEMS(vslice->ref_pic_list0);
> > > > i++) {
> > > > + vslice->ref_pic_list1[i].picture_id = vslice-
> > > > > ref_pic_list0[i].picture_id;
> > > > + vslice->ref_pic_list1[i].flags = vslice-
> > > > > ref_pic_list0[i].flags;
> > > > + }
> > > > + }
> > >
> > > Is it sensible to implement this properly in the generic code rather
> > > than having special ad-hoc code here?
> > >
> > > (Is there any chance this will also be a thing for e.g. H.264 or
> > > H.266 in
> > > future?)
> >
> > Considering that VAConfigAttribPredictionDirection is not a specific
> > attribute for hevc in VA-API, I think it is acceptable to implement
> > this in the generic code.
I tried to add the same reference twice into VAAPIEncodePicture.refs, and h265_vaapi
can get the correct L0/L1 and picture type without any change, test code:
https://github.com/intel-media-ci/ffmpeg/pull/485/commits/f98fa04fe26333f09cdc9a1a11b0bc65783f9ae4
But this change breaks slice header's information get by VAAPIEncodePicture.refs/nb_refs:
https://github.com/intel-media-ci/ffmpeg/blob/b14321e307fcd662f8d9aaa17c7f297cba2cbe4a/libavcodec/vaapi_encode_h265.c#L927
A tricky way is to reconstruct pic->refs/nb_refs here and remove the duplicated reference by using
pic->refs[i]->priv_data. pic_order_cnt. Seems this way looks even worse compare to the current implementation
in this patch. Not sure if you have any better idea?
Thanks
Fei
> >
> > Thanks
> > Haihao
> >
> > > > +
> > > > return 0;
> > > > }
> > > >
> > >
> > > - Mark
> > > _______________________________________________
> > > ffmpeg-devel mailing list
> > > ffmpeg-devel at ffmpeg.org
> > > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> > >
> > > To unsubscribe, visit link above, or email
> > > ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
> > _______________________________________________
> > ffmpeg-devel mailing list
> > ffmpeg-devel at ffmpeg.org
> > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> >
> > To unsubscribe, visit link above, or email
> > ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email ffmpeg-devel-request at ffmpeg.org
> with subject "unsubscribe".
More information about the ffmpeg-devel
mailing list