[FFmpeg-devel] [FEATURE PROPOSAL] Extracting codec-level data to binary files

[Michael Niedermayer]

On Fri, May 23, 2025 at 02:45:59AM +0200, Michael Niedermayer wrote:
> Hi Ronald
> 
> On Thu, May 22, 2025 at 07:59:06AM -0400, Ronald S. Bultje wrote:
> > Hi,
> > 
> > On Wed, May 21, 2025 at 9:34 AM Timothée <
> > timothee.informatique at regaud-chapuy.fr> wrote:
> > 
> > > Hello,
> > >
> > > I am interested in expanding ffmpeg's capabilities to extract
> > > low-level data from video codecs. Specifically, I'd like to implement
> > > functionality that would allow exporting frame data, macroblock
> > > information, quantization tables, and similar codec-specific elements
> > > to binary files for further analysis.
> > >
> > > After searching through the documentation and existing features, I
> > > haven't found similar functionality, though I may have missed
> > > something. Has this been implemented before, or are there related
> > > features I should examine?
> > 
> > 
> > Some older codecs implement minor variants for this, e.g. grep
> > for AV_FRAME_DATA_MOTION_VECTORS, which attaches a frame's motion vectors
> > to the picture data. I believe there's an example app and possibly a filter
> > to overlay MVs on top of the video frame based on this concept. You could
> > extend this to cover other (macro)block info. There used to be a variant of
> > this for quant-tables also but I can't find it, maybe it was removed.
> 
> For motion vectors:
> ./ffplay -flags2 +export_mvs -i matrixbench_mpeg2.mpg  -vf codecview=mv=pf+bf+bb
> 
> For macroblock segmentation and type vissualization + also motion vectors:
> ffplay-3.4.13 -debug vis_mb_type matrixbench_mpeg2.mpg  -vf codecview=mv=pf+bf+bb
> 
> For QP vissualization + also motion vectors:
> ffplay-3.4.13 -debug vis_qp matrixbench_mpeg2.mpg  -vf codecview=mv=pf+bf+bb
> 
> For qp values dumped on the console
> ./ffplay  -debug qp  -i matrixbench_mpeg2.mpg

And this can easily be extended to other codecs, ATM it should work with
all 16x16 MB based codecs like msmpeg4*/wmv*/mpeg1/2/4/h263/h264

mbtype and qp vissualization need codecview to be extended or versions around 3.4
which implemented it differently

Implementing vissualization as done currently with sidedata and codecview
is simple and efficient. It also would allow exporting the data to json
by writing a codec2json filter in place of codecview

Also all decoders already have all this data parsed and available so
its simpler than trying to do it in a decoder independant way

I would thus suggest implementations of this for modern codecs to
follow the same path as the existing code.

thx

[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

"You are 36 times more likely to die in a bathtub than at the hands of a
terrorist. Also, you are 2.5 times more likely to become a president and
2 times more likely to become an astronaut, than to die in a terrorist
attack." -- Thoughty2

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20250523/418197fa/attachment.sig>