[FFmpeg-devel] [FEATURE PROPOSAL] Extracting codec-level data to binary files

[Timothée]

The 2025-05-22T14:33:22.000+02:00, Ramiro Polla
<ramiro.polla at gmail.com> wrote :

> Hi,
> 
> On Thu, May 22, 2025 at 1:59 PM Ronald S. Bultje <rsbultje at gmail.com> wrote:
> 
>>  On Wed, May 21, 2025 at 9:34 AM Timothée <
>>  timothee.informatique at regaud-chapuy.fr> wrote: 
>>  
>>>   I am interested in expanding ffmpeg's capabilities to extract
>>>   low-level data from video codecs. Specifically, I'd like to
>>>   implement functionality that would allow exporting frame data,
>>>   macroblock information, quantization tables, and similar
>>>   codec-specific elements to binary files for further analysis.
>>>   After searching through the documentation and existing features,
>>>   I haven't found similar functionality, though I may have missed
>>>   something. Has this been implemented before, or are there
>>>   related features I should examine?
>>  
>>   Some older codecs implement minor variants for this, e.g. grep
>>  for AV_FRAME_DATA_MOTION_VECTORS, which attaches a frame's motion
>>  vectors to the picture data. I believe there's an example app and
>>  possibly a filter to overlay MVs on top of the video frame based
>>  on this concept. You could extend this to cover other (macro)block
>>  info. There used to be a variant of this for quant-tables also but
>>  I can't find it, maybe it was removed.
> 
>  If I recall correctly, we wanted to move away from exporting more
> of this kind of codec-specific information as picture data.
> Timothée, this kind of feature depends a lot on the codec that you
> want to work with. It will also depend on which syntax elements you
> need from the codec. In FFmpeg, there is the CBS code which
> describes the bitstream for a few codecs, which might help you. I
> have a separate project (called FFglitch), which lets you export
> some elements (such as quantization tables, dct coefficients, and
> motion vectors) from some codecs (jpeg, mpeg2, mpeg4) to JSON
> format, and even modify them in the bitstream. In the end, it will
> depend on what you want to do with the extracted low-level data, and
> for which codecs. Ramiro

Thank you for your response and for mentioning FFglitch - it looks
like an excellent project that's very relevant to what I'm trying to
accomplish.

I have a question about FFglitch's capabilities: is it possible to
extract quantization parameter (QP) tables using ffedit? I've reviewed
the documentation but may have missed this functionality if it exists.

Ideally I would want to extract data for all existing codecs, but I
know this is impossible. My current focus is on implementing this
feature for H.264, with plans to extend support to H.265 and AVI
formats as well. I want to extract this data for scientific use.

Thank you again for your time and suggestions.

Timothée