[FFmpeg-devel] [PATCH] [RFC] avcodec: export MB information in frame side data

[wm4]

On Tue, 22 Jul 2014 01:51:46 +0200
Michael Niedermayer <michaelni at gmx.at> wrote:

> On Wed, Jul 16, 2014 at 05:06:27PM +0200, Clément Bœsch wrote:
> > Also add an example exporting the MB information as a CSV stream.
> > 
> > ---
> > 
> > A bunch of questions stand:
> > 
> > - Ideally the "source" for a given macroblock should be a specific
> >   reference to a future or paste frame (something like -4 or +2).
> >   Currently it's just ±1 depending on the direction. I don't see how I
> >   can extract that information.
> > 
> > - how the MB "type" should be exported? Like, what "generic" type we
> >   need to expose for the users
> > 
> > - Who is motivated to port -vismv & various other vis_* debug to a video
> >   filter? (The hard part will probably be writing its documentation...)
> > 
> > (TODO: avcodec version bump & APIChanges entry at least)
> > ---
> [...]
> > @@ -2109,6 +2129,93 @@ void ff_print_debug_info2(AVCodecContext *avctx, AVFrame *pict, uint8_t *mbskip_
> >                           int *low_delay,
> >                           int mb_width, int mb_height, int mb_stride, int quarter_sample)
> >  {
> > +    if ((avctx->flags2 & CODEC_FLAG2_SD_MB_INFO) && mbtype_table && motion_val[0]) {
> > +        const int shift = 1 + quarter_sample;
> > +        const int mv_sample_log2 = avctx->codec_id == AV_CODEC_ID_H264 || avctx->codec_id == AV_CODEC_ID_SVQ3 ? 2 : 1;
> > +        const int mv_stride      = (mb_width << mv_sample_log2) +
> > +                                   (avctx->codec->id == AV_CODEC_ID_H264 ? 0 : 1);
> > +        int mb_x, mb_y, mbcount = 0;
> > +
> > +        /* width * height * directions * 4MB (4MB for IS_8x8) */
> > +        AVMBInfo_MB *mbs = av_malloc_array(mb_width * mb_height, 2 * 4 * sizeof(AVMBInfo_MB));
> > +        if (!mbs)
> > +            return;
> > +
> > +        // TODO: refactor with the following code
> > +        for (mb_y = 0; mb_y < mb_height; mb_y++) {
> > +            for (mb_x = 0; mb_x < mb_width; mb_x++) {
> > +                int i, direction, mb_type = mbtype_table[mb_x + mb_y * mb_stride];
> > +                for (direction = 0; direction < 2; direction++) {
> 
> 
> > +                    if (direction == 0 &&
> > +                        pict->pict_type != AV_PICTURE_TYPE_B &&
> > +                        pict->pict_type != AV_PICTURE_TYPE_P)
> > +                        continue;
> > +                    if (direction == 1 &&
> > +                        pict->pict_type != AV_PICTURE_TYPE_B)
> > +                        continue;
> > +                    if (!USES_LIST(mb_type, direction))
> > +                        continue;
> 
> these checks seems a bit redundant
> 
> 
> [...]
> > diff --git a/libavutil/mbinfo.h b/libavutil/mbinfo.h
> > new file mode 100644
> > index 0000000..89538ba
> > --- /dev/null
> > +++ b/libavutil/mbinfo.h
> > @@ -0,0 +1,32 @@
> > +/*
> > + * This file is part of FFmpeg.
> > + *
> > + * FFmpeg is free software; you can redistribute it and/or
> > + * modify it under the terms of the GNU Lesser General Public
> > + * License as published by the Free Software Foundation; either
> > + * version 2.1 of the License, or (at your option) any later version.
> > + *
> > + * FFmpeg is distributed in the hope that it will be useful,
> > + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
> > + * Lesser General Public License for more details.
> > + *
> > + * You should have received a copy of the GNU Lesser General Public
> > + * License along with FFmpeg; if not, write to the Free Software
> > + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
> > + */
> > +
> > +#ifndef AVUTIL_MBINFO_H
> > +#define AVUTIL_MBINFO_H
> > +
> > +#include <stdint.h>
> > +
> > +typedef struct AVMBInfo_MB {
> 
>     > +    int8_t source; /* -1/+1 XXX: set exact relative ref frame instead of "direction" */
> 
> i suspect this is too small for long term references
> 
> 
> > +    uint32_t type; /* how much codec specific can this be? */
> > +    uint8_t w, h;
> > +    uint16_t src_x, src_y;
> > +    uint16_t dst_x, dst_y;
> > +} AVMBInfo_MB;
> 
> as side data this is not optimal as it depends on native endianness
> making exchange of side data over the network or serialization
> into file storage more difficult

Note that there's a lot of side data now that uses native structs.

And it is certainly less awkward to use than an "in memory bitstream",
even if that should make serializing harder. (So what, you just need to
write a serialization function once -if that is even a valid use
case- but normal access is going to be much more common by users.)

> also QP, intra prediction mode, ... is missing 
> 
> 
> [...]