[FFmpeg-devel] [FEATURE PROPOSAL] Extracting codec-level data to binary files

[Timothée]

On 2025-05-23T02:57:36.000+02:00, Michael Niedermayer
<michael at niedermayer.cc> wrote:

> On Fri, May 23, 2025 at 02:45:59AM +0200, Michael Niedermayer wrote:
> 
>>  Hi Ronald On Thu, May 22, 2025 at 07:59:06AM -0400, Ronald S.
>>  Bultje wrote: 
>>  
>>>   Hi, On Wed, May 21, 2025 at 9:34 AM Timothée <
>>>   timothee.informatique at regaud-chapuy.fr> wrote: 
>>>   
>>>>    Hello, I am interested in expanding ffmpeg's capabilities to
>>>>    extract low-level data from video codecs. Specifically, I'd
>>>>    like to implement functionality that would allow exporting
>>>>    frame data, macroblock information, quantization tables, and
>>>>    similar codec-specific elements to binary files for further
>>>>    analysis. After searching through the documentation and
>>>>    existing features, I haven't found similar functionality,
>>>>    though I may have missed something. Has this been implemented
>>>>    before, or are there related features I should examine?
>>>   
>>>    Some older codecs implement minor variants for this, e.g. grep
>>>   for AV_FRAME_DATA_MOTION_VECTORS, which attaches a frame's
>>>   motion vectors to the picture data. I believe there's an example
>>>   app and possibly a filter to overlay MVs on top of the video
>>>   frame based on this concept. You could extend this to cover
>>>   other (macro)block info. There used to be a variant of this for
>>>   quant-tables also but I can't find it, maybe it was removed.
>>  
>>   For motion vectors: ./ffplay -flags2 +export_mvs -i
>>  matrixbench_mpeg2.mpg -vf codecview=mv=pf+bf+bb For macroblock
>>  segmentation and type vissualization + also motion vectors:
>>  ffplay-3.4.13 -debug vis_mb_type matrixbench_mpeg2.mpg -vf
>>  codecview=mv=pf+bf+bb For QP vissualization + also motion vectors:
>>  ffplay-3.4.13 -debug vis_qp matrixbench_mpeg2.mpg -vf
>>  codecview=mv=pf+bf+bb For qp values dumped on the console ./ffplay
>>  -debug qp -i matrixbench_mpeg2.mpg
> 
>  And this can easily be extended to other codecs, ATM it should work
> with all 16x16 MB based codecs like
> msmpeg4*/wmv*/mpeg1/2/4/h263/h264 mbtype and qp vissualization need
> codecview to be extended or versions around 3.4 which implemented it
> differently Implementing vissualization as done currently with
> sidedata and codecview is simple and efficient. It also would allow
> exporting the data to json by writing a codec2json filter in place
> of codecview Also all decoders already have all this data parsed and
> available so its simpler than trying to do it in a decoder
> independant way I would thus suggest implementations of this for
> modern codecs to follow the same path as the existing code. thx

Thanks for the helpful pointers!

I will work on the codec2json filter.

Looking at the code, I see where I can access sidedata but extracting
qb table seems to fail. (in codecview.c l.233: ff_qp_table_extract()
return 0 and qp_table is empty) (I use ./ffmpeg -flags2 +export_mvs -i
input.mp4 -vf codecview=qp=1 output.mp4 -y)

Is is qp extraction not implemented yet? Or is it because I have h264
video? If it's not implemented, I'm curious why there’s already code
that appears to handle it.

Timothée