[FFmpeg-devel] [PATCH v4 0/6] Implement SEI parsing for QSV decoders

[ffmpegagent]

Missing SEI information has always been a major drawback when using the QSV
decoders. I used to think that there's no chance to get at the data without
explicit implementation from the MSDK side (or doing something weird like
parsing in parallel). It turned out that there's a hardly known api method
that provides access to all SEI (h264/hevc) or user data (mpeg2video).

This allows to get things like closed captions, frame packing, display
orientation, HDR data (mastering display, content light level, etc.) without
having to rely on those data being provided by the MSDK as extended buffers.

The commit "Implement SEI parsing for QSV decoders" includes some hard-coded
workarounds for MSDK bugs which I reported:
https://github.com/Intel-Media-SDK/MediaSDK/issues/2597#issuecomment-1072795311

But that doesn't help. Those bugs exist and I'm sharing my workarounds,
which are empirically determined by testing a range of files. If someone is
interested, I can provide private access to a repository where we have been
testing this. Alternatively, I could also leave those workarounds out, and
just skip those SEI types.

In a previous version of this patchset, there was a concern that payload
data might need to be re-ordered. Meanwhile I have researched this carefully
and the conclusion is that this is not required.

My detailed analysis can be found here:
https://gist.github.com/softworkz/36c49586a8610813a32270ee3947a932

v3

 * frame.h: clarify doc text for av_frame_copy_side_data()

v2

 * qsvdec: make error handling consistent and clear
 * qsvdec: remove AV_CODEC_ID_MPEG1VIDEO constants
 * hevcdec: rename function to ff_hevc_set_side_data(), add doc text

v3

 * qsvdec: fix c/p error

softworkz (6):
  avutil/frame: Add av_frame_copy_side_data() and
    av_frame_remove_all_side_data()
  avcodec/vpp_qsv: Copy side data from input to output frame
  avcodec/mpeg12dec: make mpeg_decode_user_data() accessible
  avcodec/hevcdec: make set_side_data() accessible
  avcodec/h264dec: make h264_export_frame_props() accessible
  avcodec/qsvdec: Implement SEI parsing for QSV decoders

 doc/APIchanges               |   4 +
 libavcodec/h264_slice.c      |  98 ++++++++-------
 libavcodec/h264dec.h         |   2 +
 libavcodec/hevcdec.c         | 117 +++++++++---------
 libavcodec/hevcdec.h         |   9 ++
 libavcodec/mpeg12.h          |  28 +++++
 libavcodec/mpeg12dec.c       |  40 +-----
 libavcodec/qsvdec.c          | 234 +++++++++++++++++++++++++++++++++++
 libavfilter/qsvvpp.c         |   6 +
 libavfilter/vf_overlay_qsv.c |  19 ++-
 libavutil/frame.c            |  67 ++++++----
 libavutil/frame.h            |  32 +++++
 libavutil/version.h          |   2 +-
 13 files changed, 485 insertions(+), 173 deletions(-)


base-commit: 6a82412bf33108111eb3f63076fd5a51349ae114
Published-As: https://github.com/ffstaging/FFmpeg/releases/tag/pr-ffstaging-31%2Fsoftworkz%2Fsubmit_qsv_sei-v4
Fetch-It-Via: git fetch https://github.com/ffstaging/FFmpeg pr-ffstaging-31/softworkz/submit_qsv_sei-v4
Pull-Request: https://github.com/ffstaging/FFmpeg/pull/31

Range-diff vs v3:

 1:  c442597a35 ! 1:  7656477360 avutil/frame: Add av_frame_copy_side_data() and av_frame_remove_all_side_data()
     @@ doc/APIchanges: libavutil:     2021-04-27
       
       API changes, most recent first:
       
     -+2022-05-26 - xxxxxxxxx - lavu 57.26.100 - frame.h
     ++2022-05-26 - xxxxxxxxx - lavu 57.28.100 - frame.h
      +  Add av_frame_remove_all_side_data(), av_frame_copy_side_data(),
      +  AV_FRAME_TRANSFER_SD_COPY, and AV_FRAME_TRANSFER_SD_FILTER.
      +
     - 2022-05-23 - xxxxxxxxx - lavu 57.25.100 - avutil.h
     -   Deprecate av_fopen_utf8() without replacement.
     - 
     + 2022-06-12 - xxxxxxxxxx - lavf 59.25.100 - avio.h
     +   Add avio_vprintf(), similar to avio_printf() but allow to use it
     +   from within a function taking a variable argument list as input.
      
       ## libavutil/frame.c ##
      @@ libavutil/frame.c: FF_ENABLE_DEPRECATION_WARNINGS
     @@ libavutil/frame.h: int av_frame_copy(AVFrame *dst, const AVFrame *src);
      + * @param src a frame from which to copy the side data.
      + * @param flags a combination of AV_FRAME_TRANSFER_SD_*
      + *
     -+ * @return >= 0 on success, a negative AVERROR on error.
     ++ * @return 0 on success, a negative AVERROR on error.
      + *
      + * @note This function will create new references to side data buffers in src,
      + * unless the AV_FRAME_TRANSFER_SD_COPY flag is passed.
     @@ libavutil/version.h
        */
       
       #define LIBAVUTIL_VERSION_MAJOR  57
     --#define LIBAVUTIL_VERSION_MINOR  25
     -+#define LIBAVUTIL_VERSION_MINOR  26
     +-#define LIBAVUTIL_VERSION_MINOR  27
     ++#define LIBAVUTIL_VERSION_MINOR  28
       #define LIBAVUTIL_VERSION_MICRO 100
       
       #define LIBAVUTIL_VERSION_INT   AV_VERSION_INT(LIBAVUTIL_VERSION_MAJOR, \
 2:  6f50d0bd57 = 2:  06976606c5 avcodec/vpp_qsv: Copy side data from input to output frame
 3:  f682b1d695 = 3:  320a8a535c avcodec/mpeg12dec: make mpeg_decode_user_data() accessible
 4:  995d835233 = 4:  e58ad6564f avcodec/hevcdec: make set_side_data() accessible
 5:  ac8dc06395 = 5:  a57bfaebb9 avcodec/h264dec: make h264_export_frame_props() accessible
 6:  27c3dded4d = 6:  3f2588563e avcodec/qsvdec: Implement SEI parsing for QSV decoders

-- 
ffmpeg-codebot