[FFmpeg-devel] [FEATURE PROPOSAL] Extracting codec-level data to binary files

[Ramiro Polla]

On Thu, May 22, 2025 at 7:03 PM Timothée
<timothee.informatique at regaud-chapuy.fr> wrote:
> The 2025-05-22T14:33:22.000+02:00, Ramiro Polla
> <ramiro.polla at gmail.com> wrote :
> > On Thu, May 22, 2025 at 1:59 PM Ronald S. Bultje <rsbultje at gmail.com> wrote:
> >>  On Wed, May 21, 2025 at 9:34 AM Timothée <
> >>  timothee.informatique at regaud-chapuy.fr> wrote:
> >>>   I am interested in expanding ffmpeg's capabilities to extract
> >>>   low-level data from video codecs. Specifically, I'd like to
> >>>   implement functionality that would allow exporting frame data,
> >>>   macroblock information, quantization tables, and similar
> >>>   codec-specific elements to binary files for further analysis.
> >>>   After searching through the documentation and existing features,
> >>>   I haven't found similar functionality, though I may have missed
> >>>   something. Has this been implemented before, or are there
> >>>   related features I should examine?
> >>
> >>   Some older codecs implement minor variants for this, e.g. grep
> >>  for AV_FRAME_DATA_MOTION_VECTORS, which attaches a frame's motion
> >>  vectors to the picture data. I believe there's an example app and
> >>  possibly a filter to overlay MVs on top of the video frame based
> >>  on this concept. You could extend this to cover other (macro)block
> >>  info. There used to be a variant of this for quant-tables also but
> >>  I can't find it, maybe it was removed.
> >
> >  If I recall correctly, we wanted to move away from exporting more
> > of this kind of codec-specific information as picture data.
> > Timothée, this kind of feature depends a lot on the codec that you
> > want to work with. It will also depend on which syntax elements you
> > need from the codec. In FFmpeg, there is the CBS code which
> > describes the bitstream for a few codecs, which might help you. I
> > have a separate project (called FFglitch), which lets you export
> > some elements (such as quantization tables, dct coefficients, and
> > motion vectors) from some codecs (jpeg, mpeg2, mpeg4) to JSON
> > format, and even modify them in the bitstream. In the end, it will
> > depend on what you want to do with the extracted low-level data, and
> > for which codecs. Ramiro
>
> Thank you for your response and for mentioning FFglitch - it looks
> like an excellent project that's very relevant to what I'm trying to
> accomplish.
>
> I have a question about FFglitch's capabilities: is it possible to
> extract quantization parameter (QP) tables using ffedit? I've reviewed
> the documentation but may have missed this functionality if it exists.

You can get the dqt for jpeg and qscale for mpeg-2, but that's about
it for the moment.

> Ideally I would want to extract data for all existing codecs, but I
> know this is impossible. My current focus is on implementing this
> feature for H.264, with plans to extend support to H.265 and AVI
> formats as well. I want to extract this data for scientific use.

I see a few qp fields in libavcodec/cbs_h264.h and
libavcodec/cbs_h265.h. I've never used this functionality before, but
it might help you.

OTOH, if you want to add h264 support to FFglitch I would be very happy :)

Ramiro