[Libav-user] Recomended approach for a timed metadata track within an mp4 (ISOBMFF section 12.3.3.2)

[Tom Vercauteren]

Hello,

I am interested in embeding metadata corresponding to each video frame
within an mp4 file. I am however a bit unsure how to best approach this
with ffmpeg / libav. I tried looking for exisiting tutorials / guidelines
but couldn't find it so far and would thus appreciate further pointers.

>From what I read so far, it seems like frame-level metadata, also refered
to as timed metadata, is defined in section 12.3.3.2 of the ISO/IEC base
media file format (ISOBMFF standardised in ISO/IEC 14496-12 – MPEG-4 Part
12) specification. Below is an excerpt for reference.

I have seen reference to such timed metadata tracks in other frameworks:
https://developer.parrot.com/docs/pdraw/video-metadata.html
https://developer.android.com/reference/android/media/MediaMuxer#metadata-track

Reading though previous queries related to frame-level metadata with ffmpeg
/ libav, it seems like creating a stream of type AV_MEDIA_TYPE_DATA that
uses codec AV_CODEC_ID_BIN_DATA or AV_CODEC_ID_FFMETADATA or
AV_CODEC_ID_TEXT or AV_CODEC_ID_BINTEXT may be a way forward but I am not
quite sure how to best approach it. Links to some of the most relevant
questions I found so far are provided below:
https://stackoverflow.com/a/59110534/17261462
https://stackoverflow.com/a/48497299/17261462
https://stackoverflow.com/q/40658697/17261462

Is AV_MEDIA_TYPE_DATA indeed the right approach to achieving compliant
timed metadata and if so, what codec id would be appropriate? Also, how
should one set the MIME type format for the track? Any other help would be
great.

Best wishes,
Tom

**** ISOBMFF excerpt
12.3 Metadata media
12.3.1 Media handler
Timed metadata media uses the ‘meta’ handler type in the handler box of the
media box [...].
12.3.2 Media header
Metadata tracks use a null media header (‘nmhd’), as defined in subclause
8.4.5.2.
12.3.3 Sample entry
12.3.3.1 Definition
Timed metadata tracks use MetaDataSampleEntry.
[...]
12.3.3.2 Syntax
[...]
****
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://ffmpeg.org/pipermail/libav-user/attachments/20220331/3b6afbd3/attachment.htm>