[Libav-user] Injecting and reading back custom per-frame side data (UDU SEI) in HEVC / H264

[Wladislav Artsimovich]

Hello there,

we are in the same boat!
For a project, I'm also injecting custom per-Frame data using SEI Messages into an H.264 Stream. JSON Data of a bunch of sensors in my case. Works great.

Also turns out, Apple does the same on iPhones to inject per frame brightness, as read by their environment sensors for use in HDR video: https://patchwork.ffmpeg.org/project/ffmpeg/patch/1648471271-22016-2-git-send-email-lance.lmwang@gmail.com/#71019

For reference, here is my implementation I'm C# in the context of a memory leak I was Chasing, as archived by this mailing list:
https://lists.ffmpeg.org/pipermail/libav-user/2023-March/013296.html
https://lists.ffmpeg.org/pipermail/libav-user/2023-March/013340.html


> I am however struggling to get this to work as I am always reading back the custom message I used for the first frame.

You can't, at least using FFmpeg Tools. I am not sure if there is an libavXXX function which would allow you to, but FFMpeg tools themselves cannot. Looking at the FFmpeg patches for Apple's ambient viewing brightness, and from Discussions in the FFmpeg IRC, I understand that to extract that data you need to register a UUID for your usecase in libavcodec to be able to pull information, which means modifying libavcodec. Don't know how accurate my understanding is though.

You can follow <https://stackoverflow.com/questions/67283568 to get FFMpeg to announce the presence of an unrecognized custom SEI Message, but there is no function to actually display it.

In my project I resorted to reading the H264 Stream binary as a whole, scanning for the UUID / Magic Number of the SEI Message and extracting the JSON Data that way. Performance wise this was actually really good, as I could export nice Charts of my Data without interfacing with FFmpeg, but that is really more of a hackye workaround.

I think injecting custom data through SEI messages is really cool, as you can Livestream sensor data in tandem with video, without sync issues. Hope this workflow can become more elegant and less hacky.


Best regards 

Vlad

On May 21, 2024 6:00:05 PM GMT+09:00, Tom Vercauteren <tom.vercauteren at gmail.com> wrote:
>Hello,
>
>I am interested in storing custom metadata on a per frame level in a mov or
>mp4 file. It looks like using user data unregistered (UDU) supplemental
>enhancement information (SEI) may be a suitable approach for this. I am
>however struggling to get this to work as I am always reading back the
>custom message I used for the first frame.
>
>Below is a basic change I made to the mux.c and demux_decode.c examples
>from ffmpeg that shows my attempt:
>https://gist.github.com/tvercaut/94c68e46d0d0321a2c83b88024d9cd69/revisions
>
>Essentially, I have tried to use
>`av_frame_new_side_data_from_buf(frame, AV_FRAME_DATA_SEI_UNREGISTERED,
>my_sei_buffer);`
>at muxing time and
>`av_frame_get_side_data(frame, AV_FRAME_DATA_SEI_UNREGISTERED);`
>at demuxing/decoding time.
>
>I also realised I had to set the `udu_sei` flag when muxing:
>`av_dict_set_int(&opt, "udu_sei", 1, 0);`
>
>I have looked for additional material / documentation on how to do this but
>couldn't find a lot. Am I missing a key step here?
>
>Also, is there a way to display the UDU SEI messages using ffmpeg or
>ffprobe command line tools? I tried using
>`ffmpeg -i test.mov -vf showinfo -f null -`
>but this doesn't display the content to the UDU SEI messages I provided.
>
>P.S.: This is a bit of a follow-up from an old post of mine but trying a
>different approach for per-frame metadata:
>https://lists.ffmpeg.org/pipermail/libav-user/2022-April/013051.html
>
>Best wishes,
>Tom
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://ffmpeg.org/pipermail/libav-user/attachments/20240521/ea7618c7/attachment.htm>