[FFmpeg-devel] [PATCH 1/4] avutil: add generic side data for video coding info
Michael Niedermayer
michael at niedermayer.cc
Fri Jul 18 18:48:52 EEST 2025
Hi
On Fri, Jul 18, 2025 at 12:30:52PM +0200, Timothée Regaud wrote:
> From: Timothee Regaud <timothee.informatique at regaud-chapuy.fr>
>
> Adds the generic data structures to libavutil. The design is recursive to support other codecs, even though the implementation is only for H.264 for now.
>
> Signed-off-by: Timothee Regaud <timothee.informatique at regaud-chapuy.fr>
> ---
> libavutil/Makefile | 1 +
> libavutil/frame.h | 7 ++
> libavutil/side_data.c | 1 +
> libavutil/video_coding_info.h | 163 ++++++++++++++++++++++++++++++++++
> 4 files changed, 172 insertions(+)
> create mode 100644 libavutil/video_coding_info.h
>
> diff --git a/libavutil/Makefile b/libavutil/Makefile
> index 94a56bb72f..44e51ab7ae 100644
> --- a/libavutil/Makefile
> +++ b/libavutil/Makefile
> @@ -93,6 +93,7 @@ HEADERS = adler32.h \
> tree.h \
> twofish.h \
> uuid.h \
> + video_coding_info.h \
> version.h \
> video_enc_params.h \
> xtea.h \
> diff --git a/libavutil/frame.h b/libavutil/frame.h
> index c50cd263d9..f4404472a0 100644
> --- a/libavutil/frame.h
> +++ b/libavutil/frame.h
> @@ -254,6 +254,13 @@ enum AVFrameSideDataType {
> * libavutil/tdrdi.h.
> */
> AV_FRAME_DATA_3D_REFERENCE_DISPLAYS,
> +
> + /**
> + * Detailed block-level coding information. The data is an AVVideoCodingInfo
> + * structure. This is exported by video decoders and can be used by filters
> + * for analysis and visualization.
> + */
> + AV_FRAME_DATA_VIDEO_CODING_INFO,
> };
>
> enum AVActiveFormatDescription {
> diff --git a/libavutil/side_data.c b/libavutil/side_data.c
> index fa2a2c2a13..b938ef6f52 100644
> --- a/libavutil/side_data.c
> +++ b/libavutil/side_data.c
> @@ -56,6 +56,7 @@ static const AVSideDataDescriptor sd_props[] = {
> [AV_FRAME_DATA_SEI_UNREGISTERED] = { "H.26[45] User Data Unregistered SEI message", AV_SIDE_DATA_PROP_MULTI },
> [AV_FRAME_DATA_VIDEO_HINT] = { "Encoding video hint", AV_SIDE_DATA_PROP_SIZE_DEPENDENT },
> [AV_FRAME_DATA_3D_REFERENCE_DISPLAYS] = { "3D Reference Displays Information", AV_SIDE_DATA_PROP_GLOBAL },
> + [AV_FRAME_DATA_VIDEO_CODING_INFO] = { "Video Coding Info", AV_SIDE_DATA_PROP_SIZE_DEPENDENT },
> };
>
> const AVSideDataDescriptor *av_frame_side_data_desc(enum AVFrameSideDataType type)
> diff --git a/libavutil/video_coding_info.h b/libavutil/video_coding_info.h
> new file mode 100644
> index 0000000000..17e9345892
> --- /dev/null
> +++ b/libavutil/video_coding_info.h
> @@ -0,0 +1,163 @@
> +/*
> + * This file is part of FFmpeg.
> + *
> + * FFmpeg is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU Lesser General Public
> + * License as published by the Free Software Foundation; either
> + * version 2.1 of the License, or (at your option) any later version.
> + *
> + * FFmpeg is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> + * Lesser General Public License for more details.
> + *
> + * You should have received a copy of the GNU Lesser General Public
> + * License along with FFmpeg; if not, write to the Free Software
> + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
> + */
> +
> +#ifndef AVUTIL_VIDEO_CODING_INFO_H
> +#define AVUTIL_VIDEO_CODING_INFO_H
> +
> +#include <stdint.h>
> +#include <stddef.h>
> +
> +/**
> + * @file
> + * @ingroup lavu_frame
> + * Structures for describing block-level video coding information.
> + */
> +
> +/**
> + * @defgroup lavu_video_coding_info Video Coding Info
> + * @ingroup lavu_frame
> + *
> + * @{
> + * Structures for describing block-level video coding information, to be
> + * attached to an AVFrame as side data.
> + *
> + * All pointer-like members in these structures are offsets relative to the
> + * start of the AVVideoCodingInfo struct to ensure the side data is
> + * self-contained and relocatable. This is critical as the underlying buffer
> + * may be moved in memory.
> + */
> +
> +/**
> + * Structure to hold inter-prediction information for a block.
> + */
> +typedef struct AVBlockInterInfo {
> + /**
> + * Offsets to motion vectors for list 0 and list 1, relative to the
> + * start of the AVVideoCodingInfo struct.
> + * The data for each list is an array of [x, y] pairs of int16_t.
> + * The number of vectors is given by num_mv.
> + * An offset of 0 indicates this data is not present.
> + */
> + size_t mv_offset[2];
int16 is not enough, with growing picture sizes and growing precission of
motion vectors
also the MV precssion is needed somewhere somehow or they could not be
vissualized by generic code
> +
> + /**
> + * Offsets to reference indices for list 0 and list 1, relative to the
> + * start of the AVVideoCodingInfo struct.
> + * The data is an array of int8_t. A value of -1 indicates the reference
> + * is not used for a specific partition.
> + * An offset of 0 indicates this data is not present.
> + */
> + size_t ref_idx_offset[2];
> + /**
> + * Number of motion vectors for list 0 and list 1.
> + */
> + uint8_t num_mv[2];
> +} AVBlockInterInfo;
weighted bi pred needs the weights too
and for more than 1 MV, the question becomes what the other vectors
are, bipred ?, affine MC ?, ...
Also if you want to be really generic you need to allow blocks
that dont span accross the luma and chroma planes but allow
different block structures (and motion vectors) per plane
Iam not sure how generic we want to be and how useful that is.
But it seemes you want your patch to be quite generic ?
I think its more important to allow this to be extensible
than suporting everything we can think of.
That is maybe store the size of the struct also somewhere so
that elements can be added to their end without breaking
anything. At least for the main block structure
I mean a future codec might allow non rectangular blocks but we
dont want to think about that today.
Maybe its best to keep this as simple as possible but extensible
> +
> +/**
> + * Structure to hold intra-prediction information for a block.
> + */
> +typedef struct AVBlockIntraInfo {
> + /**
> + * Offset to an array of intra prediction modes, relative to the
> + * start of the AVVideoCodingInfo struct.
> + * The number of modes is given by num_pred_modes.
> + */
> + size_t pred_mode_offset;
> +
> + /**
> + * Number of intra prediction modes.
> + */
> + uint8_t num_pred_modes;
> +
> + /**
> + * Chroma intra prediction mode.
> + */
> + uint8_t chroma_pred_mode;
> +} AVBlockIntraInfo;
classifying the predition in directional, DC, and non directional and
for directional the direction. Could be usefull.
Otherwise the prediction mode number requires codec specific knowledge
to interpret
thx
[...]
--
Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB
While the State exists there can be no freedom; when there is freedom there
will be no State. -- Vladimir Lenin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20250718/c2b9d89d/attachment.sig>
More information about the ffmpeg-devel
mailing list