[FFmpeg-devel] [PATCH 2/2] avformat/mov: Support parsing of still AVIF Alpha Channel

Jan Ekström jeebjp at gmail.com
Sat Jul 2 22:34:44 EEST 2022


On Sat, Jul 2, 2022 at 7:32 PM Vignesh Venkatasubramanian
<vigneshv-at-google.com at ffmpeg.org> wrote:
>
> On Sat, Jul 2, 2022 at 2:35 AM Anton Khirnov <anton at khirnov.net> wrote:
> >
> > Quoting Vignesh Venkatasubramanian (2022-06-30 23:04:34)
> > > Parse the alpha channel for still AVIF images and expose it as a
> > > separate track. This is the simplest way of supporting AVIF alpha
> > > channel in a codec independent manner (similar to how ffmpeg
> > > supports animated AVIF with alpha channel).
> > >
> > > One can use the alphamerge filter to get a transparent image with
> > > a single command. For example:
> > >
> > > ffmpeg -i image_with_alpha.avif -filter_complex alphamerge image_with_alpha.png
> > >
> > > Signed-off-by: Vignesh Venkatasubramanian <vigneshv at google.com>
> > > ---
> > >  libavformat/isom.h |  1 +
> > >  libavformat/mov.c  | 66 ++++++++++++++++++++++++++++++++++++++++++++++
> > >  2 files changed, 67 insertions(+)
> >
> > I am against this patch, this is a digusting hack.
> >
> > These are not two streams, it is a single stream and should be exposed
> > as such.
> >
>
> Yes, while it is a hack, it is also the simplest way of supporting
> AVIF images with alpha. Since AVIF alpha images require two separate
> decoders, i could not think of any other solution that does not
> involve modifying the underlying av1 codec itself.
>
> If there are any alternative solutions where we can expose a single
> stream that needs two codecs, then i am open to implementing that.
>
> Otherwise, this solution improves the current situation where there is
> no way for the user to extract the alpha channel (albeit in a hacky
> way).

I have discussed this on IRC, and as far as I can see we have multiple
cases where an additional image has to be decoded by a separate
decoder context to get alpha, namely:

- vp8/9 (currently handled by AV_PKT_DATA_MATROSKA_BLOCKADDITIONAL)
- HEIF-like formats (such as AVIF), which are codec agnostic (you can
put JPEG, HEVC, AV1 etc into HEIF)

This means the logic (preferably) should not be codec-specific, and
would allow to implement at least two separate

My proposal - which Anton didn't really like - was to add a side data
entry AV_PKT_DATA_AUXILIARY_IMAGE , which could have type (ALPHA,
DEPTH etc) as well as the relevant packet(s) - as well as if any of
specifications define codec initialization data, that. Then
libavcodec/decode or so could initialize a secondary decoder context
for alpha images and add the third plane if parameters match what's
required.

This would also make remux possible, since containers that do not
support such auxiliary images would ignore the alpha plane, and those
that do would properly pass it through.

As for encoding, not fully sure how it should be integrated, if any
encoders actually at this moment do proper alpha coding, or do all API
clients have to separately encode with one context the primary image,
and the alpha with another? That said, the multi-stream implementation
is already merged on the muxer side, so that can be utilized in the
mean time, even though the auxiliary images are not technically a
separate stream within HEIF-likes.

Jan


More information about the ffmpeg-devel mailing list