[Libav-user] Recomended approach for a timed metadata track within an mp4 (ISOBMFF section 12.3.3.2)

[Tom Vercauteren]

On Fri, 1 Apr 2022 at 04:20, Brad Hards <bradh at frogmouth.net> wrote:

> On Thursday, 31 March 2022 9:15:42 PM AEDT Tom Vercauteren wrote:
> > I am interested in embeding metadata corresponding to each video frame
> > within an mp4 file. I am however a bit unsure how to best approach this
> > with ffmpeg / libav. I tried looking for exisiting tutorials / guidelines
> > but couldn't find it so far and would thus appreciate further pointers.
>
> It might depend on what you are trying to achieve. One thing you might
> like to
> look at is the MISB ST 1910.1 approach, which uses an EMSG box.
>
> Its not clear to me how well that is supported within ffmpeg - it'll
> probably
> require some additional code.
>

Thanks. I am fairly new to this, so the mention of MISB ST 1910.1 was
insightful. I read this blog post which helped me understand:
https://aws.amazon.com/blogs/media/misb-st-1910-and-klv-changing-the-game-in-intelligence-entertainment-and-beyond/

As far as I understand, MISB ST 1910.1 with EMSG box would be mostly
important for web streaming purpose, right? In my case, I am moslty
interested in being able to store (and read back) some metadata alongside
each video frame of an mp4 file. From that article, it seems like adding
KLV metadata would be sufficient for me and would be supported by ffmpeg /
libav already.

In such a case, am I right in assuming that using a stream of type
AV_MEDIA_TYPE_DATA that uses codec AV_CODEC_ID_SMPTE_KLV is a good way to
go? Something similar to how it is suggested here?
https://stackoverflow.com/q/68304330/17261462

Now, for the message itself, is there a recommended MISB compliant KLV
library or is it OK to "hack" it by say using protobuf for serialising the
metadata (similar to https://github.com/Parrot-Developers/libvideo-metadata
)?

Also, is it ok to leave the interleaving to ffmpeg, i.e. using
av_interleaved_write_frame rather than write_frame (which is shown in many
of the questioned I read so far)?

Many thanks,
Tom
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://ffmpeg.org/pipermail/libav-user/attachments/20220401/03c0c4e7/attachment.htm>