[Libav-user] Injecting and reading back custom per-frame side data (UDU SEI) in HEVC / H264

[Tom Vercauteren]

>
> I am interested in storing custom metadata on a per frame level in a mov
> or mp4 file. It looks like using user data unregistered (UDU)
> supplemental enhancement information (SEI) may be a suitable approach for
> this. I am however struggling to get this to work as I am always reading
> back the custom message I used for the first frame.


Turns out I was wrongly assuming from the API that there can be only one
item of AV_FRAME_DATA_SEI_UNREGISTERED. I was accumulating messages in a
single frame and only displaying the first one.

This is now fixed in my gist by calling `av_frame_remove_side_data`:
https://gist.github.com/tvercaut/94c68e46d0d0321a2c83b88024d9cd69

It seems a bit unclear to me though why `av_frame_get_side_data` only
returns the first item rather than a list of items.

> I am however struggling to get this to work as I am always reading back
> the custom message I used for the first frame.
>
> You can't, at least using FFmpeg Tools. I am not sure if there is an
> libavXXX function which would allow you to, but FFMpeg tools themselves
> cannot. Looking at the FFmpeg patches for Apple's ambient viewing
> brightness, and from Discussions in the FFmpeg IRC, I understand that to
> extract that data you need to register a UUID for your usecase in
> libavcodec to be able to pull information, which means modifying
> libavcodec. Don't know how accurate my understanding is though.
>
> You can follow <https://stackoverflow.com/questions/67283568 to get
> FFMpeg to announce the presence of an unrecognized custom SEI Message, but
> there is no function to actually display it.
>
> In my project I resorted to reading the H264 Stream binary as a whole,
> scanning for the UUID / Magic Number of the SEI Message and extracting the
> JSON Data that way. Performance wise this was actually really good, as I
> could export nice Charts of my Data without interfacing with FFmpeg, but
> that is really more of a hackye workaround.
>
> I think injecting custom data through SEI messages is really cool, as you
> can Livestream sensor data in tandem with video, without sync issues. Hope
> this workflow can become more elegant and less hacky.
>

Thanks. I found a way with libav as mentioned above but would still be
interested in being able to extract that SEI from the ffmpeg command line.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://ffmpeg.org/pipermail/libav-user/attachments/20240521/1d361638/attachment.htm>