[FFmpeg-devel] [PATCH 2/2] avformat/mov: Support parsing of still AVIF Alpha Channel

[Jan Ekström]

On Sun, Jul 3, 2022 at 12:15 AM Vignesh Venkatasubramanian
<vigneshv-at-google.com at ffmpeg.org> wrote:
>
> On Sat, Jul 2, 2022 at 12:35 PM Jan Ekström <jeebjp at gmail.com> wrote:
> >
> > On Sat, Jul 2, 2022 at 7:32 PM Vignesh Venkatasubramanian
> > <vigneshv-at-google.com at ffmpeg.org> wrote:
> > >
> > > On Sat, Jul 2, 2022 at 2:35 AM Anton Khirnov <anton at khirnov.net> wrote:
> > > >
> > > > Quoting Vignesh Venkatasubramanian (2022-06-30 23:04:34)
> > > > > Parse the alpha channel for still AVIF images and expose it as a
> > > > > separate track. This is the simplest way of supporting AVIF alpha
> > > > > channel in a codec independent manner (similar to how ffmpeg
> > > > > supports animated AVIF with alpha channel).
> > > > >
> > > > > One can use the alphamerge filter to get a transparent image with
> > > > > a single command. For example:
> > > > >
> > > > > ffmpeg -i image_with_alpha.avif -filter_complex alphamerge image_with_alpha.png
> > > > >
> > > > > Signed-off-by: Vignesh Venkatasubramanian <vigneshv at google.com>
> > > > > ---
> > > > >  libavformat/isom.h |  1 +
> > > > >  libavformat/mov.c  | 66 ++++++++++++++++++++++++++++++++++++++++++++++
> > > > >  2 files changed, 67 insertions(+)
> > > >
> > > > I am against this patch, this is a digusting hack.
> > > >
> > > > These are not two streams, it is a single stream and should be exposed
> > > > as such.
> > > >
> > >
> > > Yes, while it is a hack, it is also the simplest way of supporting
> > > AVIF images with alpha. Since AVIF alpha images require two separate
> > > decoders, i could not think of any other solution that does not
> > > involve modifying the underlying av1 codec itself.
> > >
> > > If there are any alternative solutions where we can expose a single
> > > stream that needs two codecs, then i am open to implementing that.
> > >
> > > Otherwise, this solution improves the current situation where there is
> > > no way for the user to extract the alpha channel (albeit in a hacky
> > > way).
> >
> > I have discussed this on IRC,
>
> Thank you for discussing this!
>
> > and as far as I can see we have multiple
> > cases where an additional image has to be decoded by a separate
> > decoder context to get alpha, namely:
> >
> > - vp8/9 (currently handled by AV_PKT_DATA_MATROSKA_BLOCKADDITIONAL)
> > - HEIF-like formats (such as AVIF), which are codec agnostic (you can
> > put JPEG, HEVC, AV1 etc into HEIF)
> >
> > This means the logic (preferably) should not be codec-specific, and
> > would allow to implement at least two separate
> >
> > My proposal - which Anton didn't really like - was to add a side data
> > entry AV_PKT_DATA_AUXILIARY_IMAGE , which could have type (ALPHA,
> > DEPTH etc) as well as the relevant packet(s) - as well as if any of
> > specifications define codec initialization data, that. Then
> > libavcodec/decode or so could initialize a secondary decoder context
> > for alpha images and add the third plane if parameters match what's
> > required.
> >
>
> My only concern about this solution is that, while the demuxer can
> easily put the alpha data in the side data, we will then have to
> modify the decoder (in case of HEIF, we will have to change each
> underlying codec like av1, hevc, jpeg, etc) to support handling of
> this side data. I was just hoping it would be nicer if we can have
> something simpler without having to change every individual libavcodec
> implementation. ffmpeg already has a way to cleanly initialize codecs
> and pass data to them, so it would be nicer if we can leverage that
> somehow instead of making each libavcodec implementation handle this
> side data type.

Yes This is why I specifically noted that general decode logic should
be modified, not separate decoders. Just like the recent ICC profile
<-> color space information patch set by Niklas.

It is clearly a thing that is going to get utilized by multiple
things, so having it specific to specific codecs makes no sense :) .

Jan