[FFmpeg-devel] [FEATURE PROPOSAL] Extracting codec-level data to binary files

[Michael Niedermayer]

On Fri, May 23, 2025 at 11:33:40AM +0200, Timothée wrote:
> On 2025-05-23T02:57:36.000+02:00, Michael Niedermayer
> <michael at niedermayer.cc> wrote:
> 
> > On Fri, May 23, 2025 at 02:45:59AM +0200, Michael Niedermayer wrote:
> > 
> >>  Hi Ronald On Thu, May 22, 2025 at 07:59:06AM -0400, Ronald S.
> >>  Bultje wrote: 
> >>  
> >>>   Hi, On Wed, May 21, 2025 at 9:34 AM Timothée <
> >>>   timothee.informatique at regaud-chapuy.fr> wrote: 
> >>>   
> >>>>    Hello, I am interested in expanding ffmpeg's capabilities to
> >>>>    extract low-level data from video codecs. Specifically, I'd
> >>>>    like to implement functionality that would allow exporting
> >>>>    frame data, macroblock information, quantization tables, and
> >>>>    similar codec-specific elements to binary files for further
> >>>>    analysis. After searching through the documentation and
> >>>>    existing features, I haven't found similar functionality,
> >>>>    though I may have missed something. Has this been implemented
> >>>>    before, or are there related features I should examine?
> >>>   
> >>>    Some older codecs implement minor variants for this, e.g. grep
> >>>   for AV_FRAME_DATA_MOTION_VECTORS, which attaches a frame's
> >>>   motion vectors to the picture data. I believe there's an example
> >>>   app and possibly a filter to overlay MVs on top of the video
> >>>   frame based on this concept. You could extend this to cover
> >>>   other (macro)block info. There used to be a variant of this for
> >>>   quant-tables also but I can't find it, maybe it was removed.
> >>  
> >>   For motion vectors: ./ffplay -flags2 +export_mvs -i
> >>  matrixbench_mpeg2.mpg -vf codecview=mv=pf+bf+bb For macroblock
> >>  segmentation and type vissualization + also motion vectors:
> >>  ffplay-3.4.13 -debug vis_mb_type matrixbench_mpeg2.mpg -vf
> >>  codecview=mv=pf+bf+bb For QP vissualization + also motion vectors:
> >>  ffplay-3.4.13 -debug vis_qp matrixbench_mpeg2.mpg -vf
> >>  codecview=mv=pf+bf+bb For qp values dumped on the console ./ffplay
> >>  -debug qp -i matrixbench_mpeg2.mpg
> > 
> >  And this can easily be extended to other codecs, ATM it should work
> > with all 16x16 MB based codecs like
> > msmpeg4*/wmv*/mpeg1/2/4/h263/h264 mbtype and qp vissualization need
> > codecview to be extended or versions around 3.4 which implemented it
> > differently Implementing vissualization as done currently with
> > sidedata and codecview is simple and efficient. It also would allow
> > exporting the data to json by writing a codec2json filter in place
> > of codecview Also all decoders already have all this data parsed and
> > available so its simpler than trying to do it in a decoder
> > independant way I would thus suggest implementations of this for
> > modern codecs to follow the same path as the existing code. thx
> 
> Thanks for the helpful pointers!
> 
> I will work on the codec2json filter.
> 
> Looking at the code, I see where I can access sidedata but extracting
> qb table seems to fail. (in codecview.c l.233: ff_qp_table_extract()
> return 0 and qp_table is empty) (I use ./ffmpeg -flags2 +export_mvs -i
> input.mp4 -vf codecview=qp=1 output.mp4 -y)
> 
> Is is qp extraction not implemented yet? Or is it because I have h264
> video? If it's not implemented, I'm curious why there’s already code
> that appears to handle it.

look at ff_print_debug_info2()
theres probably something missing

The QP and MB type code was changed from being inside arrays of pictures
to sidedata. Something likely was lost/forgotten in the process

-debug qp
works with h264 so likely teh export into sidedata is not fully
implemented

thx

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

"Nothing to hide" only works if the folks in power share the values of
you and everyone you know entirely and always will -- Tom Scott

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20250523/b1e8d12d/attachment.sig>