[FFmpeg-devel] [PATCH 2/2] avformat/mov: Support parsing of still AVIF Alpha Channel

[Vignesh Venkatasubramanian]

On Sun, Jul 3, 2022 at 5:18 AM Jan Ekström <jeebjp at gmail.com> wrote:
>
> On Sun, Jul 3, 2022 at 12:15 AM Vignesh Venkatasubramanian
> <vigneshv-at-google.com at ffmpeg.org> wrote:
> >
> > On Sat, Jul 2, 2022 at 12:35 PM Jan Ekström <jeebjp at gmail.com> wrote:
> > >
> > > On Sat, Jul 2, 2022 at 7:32 PM Vignesh Venkatasubramanian
> > > <vigneshv-at-google.com at ffmpeg.org> wrote:
> > > >
> > > > On Sat, Jul 2, 2022 at 2:35 AM Anton Khirnov <anton at khirnov.net> wrote:
> > > > >
> > > > > Quoting Vignesh Venkatasubramanian (2022-06-30 23:04:34)
> > > > > > Parse the alpha channel for still AVIF images and expose it as a
> > > > > > separate track. This is the simplest way of supporting AVIF alpha
> > > > > > channel in a codec independent manner (similar to how ffmpeg
> > > > > > supports animated AVIF with alpha channel).
> > > > > >
> > > > > > One can use the alphamerge filter to get a transparent image with
> > > > > > a single command. For example:
> > > > > >
> > > > > > ffmpeg -i image_with_alpha.avif -filter_complex alphamerge image_with_alpha.png
> > > > > >
> > > > > > Signed-off-by: Vignesh Venkatasubramanian <vigneshv at google.com>
> > > > > > ---
> > > > > >  libavformat/isom.h |  1 +
> > > > > >  libavformat/mov.c  | 66 ++++++++++++++++++++++++++++++++++++++++++++++
> > > > > >  2 files changed, 67 insertions(+)
> > > > >
> > > > > I am against this patch, this is a digusting hack.
> > > > >
> > > > > These are not two streams, it is a single stream and should be exposed
> > > > > as such.
> > > > >
> > > >
> > > > Yes, while it is a hack, it is also the simplest way of supporting
> > > > AVIF images with alpha. Since AVIF alpha images require two separate
> > > > decoders, i could not think of any other solution that does not
> > > > involve modifying the underlying av1 codec itself.
> > > >
> > > > If there are any alternative solutions where we can expose a single
> > > > stream that needs two codecs, then i am open to implementing that.
> > > >
> > > > Otherwise, this solution improves the current situation where there is
> > > > no way for the user to extract the alpha channel (albeit in a hacky
> > > > way).
> > >
> > > I have discussed this on IRC,
> >
> > Thank you for discussing this!
> >
> > > and as far as I can see we have multiple
> > > cases where an additional image has to be decoded by a separate
> > > decoder context to get alpha, namely:
> > >
> > > - vp8/9 (currently handled by AV_PKT_DATA_MATROSKA_BLOCKADDITIONAL)
> > > - HEIF-like formats (such as AVIF), which are codec agnostic (you can
> > > put JPEG, HEVC, AV1 etc into HEIF)
> > >
> > > This means the logic (preferably) should not be codec-specific, and
> > > would allow to implement at least two separate
> > >
> > > My proposal - which Anton didn't really like - was to add a side data
> > > entry AV_PKT_DATA_AUXILIARY_IMAGE , which could have type (ALPHA,
> > > DEPTH etc) as well as the relevant packet(s) - as well as if any of
> > > specifications define codec initialization data, that. Then
> > > libavcodec/decode or so could initialize a secondary decoder context
> > > for alpha images and add the third plane if parameters match what's
> > > required.
> > >
> >
> > My only concern about this solution is that, while the demuxer can
> > easily put the alpha data in the side data, we will then have to
> > modify the decoder (in case of HEIF, we will have to change each
> > underlying codec like av1, hevc, jpeg, etc) to support handling of
> > this side data. I was just hoping it would be nicer if we can have
> > something simpler without having to change every individual libavcodec
> > implementation. ffmpeg already has a way to cleanly initialize codecs
> > and pass data to them, so it would be nicer if we can leverage that
> > somehow instead of making each libavcodec implementation handle this
> > side data type.
>
> Yes This is why I specifically noted that general decode logic should
> be modified, not separate decoders. Just like the recent ICC profile
> <-> color space information patch set by Niklas.
>

Ah okay, i see what you are saying now. This makes sense. I am also in
favor of this solution then.

We can abandon this second patch for now and i will see if i can
implement the side data based solution.

The first patch in this series is still a refactor that will be useful
for the future. So i will leave that open for review.

> It is clearly a thing that is going to get utilized by multiple
> things, so having it specific to specific codecs makes no sense :) .

Yes, sounds good!

>
> Jan
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".



-- 
Vignesh