[FFmpeg-devel] [PATCH 31/92] Vulkan patchset part 1 - common code changes

[Lynne]

The attached patchset is all the common code changes that my Vulkan patchset needs.

In total lines of code, this part has 425 additions and 131 deletions.
Most of that is additions to HEVC parsing. Excluding them, the patchset is
200 lines of code added, which is manageable.

Apart from the parser changes, the following other changes have been
made to the API:

AVHWAccel.free_frame_priv exists due to Vulkan's way of using VkImageView
objects to wrap VkImage objects, which we need to free once they're no longer
in use. Every other API uses the direct objects in decoding, but with Vulkan,
they have to be represented by other objects.
We also use it to free the slice offsets buffer.

AVHWAccel.flush exists due to Vulkan keeping decoder state, despite being
stateless in theory. The decoder has to be notified of flushes in order to reset
decoding slots and other data it needs, such as motion vectors and reference
lists for AV1. Otherwise, inferring whether a flush has happened can be codec
dependent, and hacky.

hwaccel_params_buf exists due to Vulkan's way of compiling SPS/PPS data
into objects, making updating expensive. The change allows for hardware
to only upload new parameters if they have been changed.
It's insignificant for H264 and AV1, but HEVC's structures can reach 114
megabytes of data that has to be uploaded, for a specially crafted input,
which is enough to DDOS an ingest.
The data is set and managed by the hwaccel, but does need to be synchronized
between different decoding threads, which this patch performs.

Finally, the HWACCEL_CAP_THREAD_SAFE flag is added due to Vulkan being
actually threadsafe, and requiring no serialization. It does work and it
does actually make a difference, on average, it can increase performance
by 20% for an average B-frame using HEVC stream, depending on the
number of threads and the number of decode queues.
While hardware decoders are fast in general, certain vendors such as AMD
can choke up while playing 8k video, and this patch can significantly help
increase throughput.

In context, the changes can be viewed here:
https://github.com/cyanreg/FFmpeg/tree/vulkan

The rest of the whole patchset is either rewrites, filter code, or
the actual hardware accel code.

The patchset will not be pushed standalone, but as part of the greater
Vulkan patchset.

31 patches attached.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-h2645_vui-expose-aspect_ratio_idc.patch
Type: text/x-diff
Size: 1858 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20230314/97d138a0/attachment.patch>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0002-h2645_vui-expose-aspect_ratio_info_present_flag.patch
Type: text/x-diff
Size: 1470 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20230314/97d138a0/attachment-0001.patch>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0003-h264_ps-expose-pps_id.patch
Type: text/x-diff
Size: 1227 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20230314/97d138a0/attachment-0002.patch>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0004-h264_ps-set-pic_scaling_matrix_present_flag.patch
Type: text/x-diff
Size: 3224 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20230314/97d138a0/attachment-0003.patch>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0005-h264_parser-expose-idr_pic_id.patch
Type: text/x-diff
Size: 1438 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20230314/97d138a0/attachment-0004.patch>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0006-h264_ps-comment-pic_order_present-better.patch
Type: text/x-diff
Size: 998 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20230314/97d138a0/attachment-0005.patch>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0007-h264_ps-expose-max_dec_frame_buffering.patch
Type: text/x-diff
Size: 1397 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20230314/97d138a0/attachment-0006.patch>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0008-h264_ps-expose-bit-rate-and-CPB-size-fields.patch
Type: text/x-diff
Size: 2144 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20230314/97d138a0/attachment-0007.patch>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0009-h264_ps-expose-scaling_matrix_present_mask.patch
Type: text/x-diff
Size: 7405 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20230314/97d138a0/attachment-0008.patch>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0010-hevc_ps-expose-SPS-and-VPS-headers.patch
Type: text/x-diff
Size: 9069 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20230314/97d138a0/attachment-0009.patch>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0011-hevc_ps-expose-pps_id.patch
Type: text/x-diff
Size: 1214 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20230314/97d138a0/attachment-0010.patch>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0012-hevc_ps-expose-vps_id.patch
Type: text/x-diff
Size: 1163 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20230314/97d138a0/attachment-0011.patch>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0013-hevc_ps-expose-pps_extension_present_flag.patch
Type: text/x-diff
Size: 1459 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20230314/97d138a0/attachment-0012.patch>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0014-hevcdec-expose-bits_used_for_short_term_rps.patch
Type: text/x-diff
Size: 1229 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20230314/97d138a0/attachment-0013.patch>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0015-hevc_ps-expose-log2_diff_max_min_transform_block_siz.patch
Type: text/x-diff
Size: 3169 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20230314/97d138a0/attachment-0014.patch>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0016-hevc_ps-expose-rps-fields.patch
Type: text/x-diff
Size: 4901 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20230314/97d138a0/attachment-0015.patch>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0017-hevc_ps-expose-vui_present-flag.patch
Type: text/x-diff
Size: 1635 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20230314/97d138a0/attachment-0016.patch>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0018-hevc_ps-expose-sublayer_ordering_info_flag.patch
Type: text/x-diff
Size: 2208 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20230314/97d138a0/attachment-0017.patch>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0019-hevc_ps-expose-conformance_window_flag.patch
Type: text/x-diff
Size: 1337 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20230314/97d138a0/attachment-0018.patch>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0020-hevc_ps-expose-sps_extension_present_flag.patch
Type: text/x-diff
Size: 1322 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20230314/97d138a0/attachment-0019.patch>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0021-av1dec-expose-ref_order_hint.patch
Type: text/x-diff
Size: 2178 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20230314/97d138a0/attachment-0020.patch>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0022-av1-set-skip-mode-frames-properly.patch
Type: text/x-diff
Size: 1102 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20230314/97d138a0/attachment-0021.patch>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0023-cbs_av1-expose-tile-col-row-starts-in-SBs.patch
Type: text/x-diff
Size: 2953 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20230314/97d138a0/attachment-0022.patch>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0024-hwconfig-add-a-new-HWACCEL_CAP_THREAD_SAFE-for-threa.patch
Type: text/x-diff
Size: 1370 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20230314/97d138a0/attachment-0023.patch>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0025-avcodec-add-AVHWAccel.free_frame_priv-callback.patch
Type: text/x-diff
Size: 7764 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20230314/97d138a0/attachment-0024.patch>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0026-avcodec-add-AVHWAccel.flush-callback.patch
Type: text/x-diff
Size: 3136 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20230314/97d138a0/attachment-0025.patch>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0027-h264dec-add-hwaccel_params_buf.patch
Type: text/x-diff
Size: 2738 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20230314/97d138a0/attachment-0026.patch>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0028-hevcdec-add-hwaccel_params_buf.patch
Type: text/x-diff
Size: 2847 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20230314/97d138a0/attachment-0027.patch>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0029-av1dec-add-hwaccel_params_buf.patch
Type: text/x-diff
Size: 1469 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20230314/97d138a0/attachment-0028.patch>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0030-lavu-add-12-bit-2-plane-422-and-444-pixel-formats.patch
Type: text/x-diff
Size: 5055 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20230314/97d138a0/attachment-0029.patch>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0031-lsws-add-in-out-support-for-the-new-12-bit-2-plane-4.patch
Type: text/x-diff
Size: 3502 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20230314/97d138a0/attachment-0030.patch>