[FFmpeg-devel] [FEATURE PROPOSAL] Extracting codec-level data to binary files

[Timothée]

On 2025-05-23T13:55:57.000+02:00, Michael Niedermayer
<michael at niedermayer.cc> wrote:

> On Fri, May 23, 2025 at 11:33:40AM +0200, Timothée wrote:
> 
>>  On 2025-05-23T02:57:36.000+02:00, Michael Niedermayer
>>  <michael at niedermayer.cc> wrote: 
>>  
>>>   On Fri, May 23, 2025 at 02:45:59AM +0200, Michael Niedermayer
>>>   wrote: 
>>>   
>>>>    Hi Ronald On Thu, May 22, 2025 at 07:59:06AM -0400, Ronald S.
>>>>    Bultje wrote: 
>>>>    
>>>>>     Hi, On Wed, May 21, 2025 at 9:34 AM Timothée <
>>>>>     timothee.informatique at regaud-chapuy.fr> wrote: 
>>>>>     
>>>>>>      Hello, I am interested in expanding ffmpeg's capabilities
>>>>>>      to extract low-level data from video codecs. Specifically,
>>>>>>      I'd like to implement functionality that would allow
>>>>>>      exporting frame data, macroblock information, quantization
>>>>>>      tables, and similar codec-specific elements to binary
>>>>>>      files for further analysis. After searching through the
>>>>>>      documentation and existing features, I haven't found
>>>>>>      similar functionality, though I may have missed something.
>>>>>>      Has this been implemented before, or are there related
>>>>>>      features I should examine?
>>>>>     
>>>>>      Some older codecs implement minor variants for this, e.g.
>>>>>     grep for AV_FRAME_DATA_MOTION_VECTORS, which attaches a
>>>>>     frame's motion vectors to the picture data. I believe
>>>>>     there's an example app and possibly a filter to overlay MVs
>>>>>     on top of the video frame based on this concept. You could
>>>>>     extend this to cover other (macro)block info. There used to
>>>>>     be a variant of this for quant-tables also but I can't find
>>>>>     it, maybe it was removed.
>>>>    
>>>>     For motion vectors: ./ffplay -flags2 +export_mvs -i
>>>>    matrixbench_mpeg2.mpg -vf codecview=mv=pf+bf+bb For macroblock
>>>>    segmentation and type vissualization + also motion vectors:
>>>>    ffplay-3.4.13 -debug vis_mb_type matrixbench_mpeg2.mpg -vf
>>>>    codecview=mv=pf+bf+bb For QP vissualization + also motion
>>>>    vectors: ffplay-3.4.13 -debug vis_qp matrixbench_mpeg2.mpg -vf
>>>>    codecview=mv=pf+bf+bb For qp values dumped on the console
>>>>    ./ffplay -debug qp -i matrixbench_mpeg2.mpg
>>>   
>>>    And this can easily be extended to other codecs, ATM it should
>>>   work with all 16x16 MB based codecs like
>>>   msmpeg4*/wmv*/mpeg1/2/4/h263/h264 mbtype and qp vissualization
>>>   need codecview to be extended or versions around 3.4 which
>>>   implemented it differently Implementing vissualization as done
>>>   currently with sidedata and codecview is simple and efficient.
>>>   It also would allow exporting the data to json by writing a
>>>   codec2json filter in place of codecview Also all decoders
>>>   already have all this data parsed and available so its simpler
>>>   than trying to do it in a decoder independant way I would thus
>>>   suggest implementations of this for modern codecs to follow the
>>>   same path as the existing code. thx
>>  
>>   Thanks for the helpful pointers! I will work on the codec2json
>>  filter. Looking at the code, I see where I can access sidedata but
>>  extracting qb table seems to fail. (in codecview.c l.233:
>>  ff_qp_table_extract() return 0 and qp_table is empty) (I use
>>  ./ffmpeg -flags2 +export_mvs -i input.mp4 -vf codecview=qp=1
>>  output.mp4 -y) Is is qp extraction not implemented yet? Or is it
>>  because I have h264 video? If it's not implemented, I'm curious
>>  why there’s already code that appears to handle it.
> 
>  look at ff_print_debug_info2() theres probably something missing
> The QP and MB type code was changed from being inside arrays of
> pictures to sidedata. Something likely was lost/forgotten in the
> process -debug qp works with h264 so likely teh export into sidedata
> is not fully implemented

Yes, this is exactly what is happening.

After some research, I have found: `// FIXME qscale / qp ... stuff` in
libavcodec/h264_slice.c, line 1871

and also in libavfilter/qp_table.h: 
34 /** 
35 * Normalize the qscale factor 
36 * FIXME Add support for other values of enum AVVideoEncParamsType 
37 * besides AV_VIDEO_ENC_PARAMS_MPEG2. 
38 */ 
39 static inline int ff_norm_qscale(int qscale, enum
AVVideoEncParamsType type) 
40 { 
41     switch (type) { 
42        case AV_VIDEO_ENC_PARAMS_MPEG2: return qscale >> 1; 
43     } 
44     return qscale; 
45 } 

So if I understand correctly, QP tables are not attached to the frames
except for MPEG2. This has been left as a FIXME for other codecs. 
Should I extend it to work for H.264, and maybe later for other
codecs, thus removing the first FIXME and partially fixing the second?
Or is this too specific and too complicated to maintain if it needs to
be added for every codec?

Thx,

Timothée