[FFmpeg-devel] ffmpeg-qsv: Is opaque memory allocation path used in ffmpeg-qsv?

[Max Dmitrichenko]

usage as
format            = AV_PIX_FMT_QSV;
frame_type      = MFX_MEMTYPE_OPAQUE_FRAME;
enforces usage of opaque memory from any 3rd_party application.

you can offer patch to remove opaque implementation
where it will be not reasonable to apply it unless deprecated and dropped
from MediaSDK definition
https://github.com/Intel-Media-SDK/MediaSDK/blob/518334ea3cd34764ff2c941d32591c8eb1cadee5/api/include/mfxstructures.h#L383


> it might not be actually functional
help in validation and fixes are welcome

regards
Max


On Sat, Oct 26, 2019 at 12:13 AM Rogozhkin, Dmitry V <
dmitry.v.rogozhkin at intel.com> wrote:

> On Fri, 2019-10-25 at 02:17 +0000, Fu, Linjie wrote:
> > > -----Original Message-----
> > > From: ffmpeg-devel <ffmpeg-devel-bounces at ffmpeg.org> On Behalf Of
> > > Rogozhkin, Dmitry V
> > > Sent: Friday, October 25, 2019 05:17
> > > To: ffmpeg-devel at ffmpeg.org
> > > Subject: Re: [FFmpeg-devel] ffmpeg-qsv: Is opaque memory allocation
> > > path
> > > used in ffmpeg-qsv?
> > >
> > > On Thu, 2019-10-24 at 17:12 +0200, Max Dmitrichenko wrote:
> > > > Hi,
> > > >
> > > > Just have a look at any usage as: ffmpeg -hwaccel qsv .......
> > >
> > > I looked into these 3 command lines on Linux:
> > >
> > > ffmpeg -hwaccel qsv -c:v h264_qsv -i AUD_MW_E.264  \
> > >        -vf hwdownload,format=nv12 -pix_fmt yuv420p \
> > >        AUD_MW.yuv
> > >
> > > ffmpeg -hwaccel qsv -c:v h264_qsv -i AUD_MW_E.264 \
> > >        -c:v hevc_qsv -b:v 5M AUD_MW_E.hevc
> > >
> > > ffmpeg -init_hw_device qsv=hw     \
> > >        -filter_hw_device hw -f rawvideo -pix_fmt  \
> > >        yuv420p -s:v 176x144 -i AUD_MW.yuv -vf     \
> > >        hwupload=extra_hw_frames=64,format=qsv     \
> > >        -c:v h264_qsv -b:v 5M -frames:v 10         \
> > >        -y ./AUD_MW_E.h264
> > >
> > > For what I see, none of the them is using QSV opaque memory. For
> > > the
> > > first 2 command lines ffmpeg explicitly prints (w/ -loglevel
> > > debug):
> > >
> > > [h264_qsv @ 0x34f6980] Decoder: output is video memory surface
> > >
> > > Besides, I checked w/ manual printfs from the code - opaque is not
> > > used.
> > >
> > > Hence, this question: is actually qsv opaque memory path used
> > > anywhere
> > > in ffmpeg.exe sample or in other libavcodec/libavfilter based
> > > applications? Is there Linux/Windows difference which I don't
> > > notice in
> > > code review?
> > >
> >
> > I didn't see MFX_MEMTYPE_OPAQUE_FRAME set for frames_hwctx-
> > >frame_type
> > in mainline ffmpeg application.
>
> Right. I also don't see it. I think that none of upstream ffmpeg
> applications actually exercise this path. This raises a concern why we
> have this in sources and complicate them if we don't use them? I guess
> most testing is done via mainline ffmpeg.exe and opaque path test
> coverage is likely missing meaning it might not be actually functional.
>
> If we will look from the end-user perspective, there is also
> opportunity to write their own applications using libavcodec/etc.
> libraries. In this case he will work with these 2 header files exposing
> QSV capabilities:
>
> ./libavcodec/qsv.h
> ./libavutil/hwcontext_qsv.h
>
> On this path he will be able to use the following:
> AVQSVFramesContext::frame_type
> AVQSVContext::opaque_alloc
> to configure QSV.
>
> Questions:
> 1. Is opaque memory usage limited to internal ffmpeg component
> connections? or it is possible to have external non-ffmpeg component
> connected to ffmpeg one via qsv opaque memory?
> 2. Does anyone actually uses the above to configure qsv opaque memory?
> 3. Regardless of the above: is that possible to deprecate and drop QSV
> opaque memory support in ffmpeg-qsv path?
>
> >
> > BTW, in the frame_allocator used for
> > MFXVideoCORE_SetFrameAllocator(),
> >
> https://github.com/FFmpeg/FFmpeg/blob/6f2a3958cfac135c60b509a61a4fd39432d8f9a9/libavcodec/qsv.c#L562
> >
> > only MFX_MEMTYPE_INTERNAL_FRAME and MFX_MEMTYPE_EXTERNAL_FRAME is
> > supported as an allocation request since commit:
> > 4ab61cd983b539749bd621ea271624ddb5196a8e
> >
> > (For VPP,  MFX_MEMTYPE_FROM_VPPIN is supported)
> > - linjie
> > _______________________________________________
> > ffmpeg-devel mailing list
> > ffmpeg-devel at ffmpeg.org
> > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> >
> > To unsubscribe, visit link above, or email
> > ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".