[FFmpeg-devel] [PATCH 2/2] avformat/mov: Support parsing of still AVIF Alpha Channel

[Vignesh Venkatasubramanian]

On Sat, Jul 2, 2022 at 12:35 PM Jan Ekström <jeebjp at gmail.com> wrote:
>
> On Sat, Jul 2, 2022 at 7:32 PM Vignesh Venkatasubramanian
> <vigneshv-at-google.com at ffmpeg.org> wrote:
> >
> > On Sat, Jul 2, 2022 at 2:35 AM Anton Khirnov <anton at khirnov.net> wrote:
> > >
> > > Quoting Vignesh Venkatasubramanian (2022-06-30 23:04:34)
> > > > Parse the alpha channel for still AVIF images and expose it as a
> > > > separate track. This is the simplest way of supporting AVIF alpha
> > > > channel in a codec independent manner (similar to how ffmpeg
> > > > supports animated AVIF with alpha channel).
> > > >
> > > > One can use the alphamerge filter to get a transparent image with
> > > > a single command. For example:
> > > >
> > > > ffmpeg -i image_with_alpha.avif -filter_complex alphamerge image_with_alpha.png
> > > >
> > > > Signed-off-by: Vignesh Venkatasubramanian <vigneshv at google.com>
> > > > ---
> > > >  libavformat/isom.h |  1 +
> > > >  libavformat/mov.c  | 66 ++++++++++++++++++++++++++++++++++++++++++++++
> > > >  2 files changed, 67 insertions(+)
> > >
> > > I am against this patch, this is a digusting hack.
> > >
> > > These are not two streams, it is a single stream and should be exposed
> > > as such.
> > >
> >
> > Yes, while it is a hack, it is also the simplest way of supporting
> > AVIF images with alpha. Since AVIF alpha images require two separate
> > decoders, i could not think of any other solution that does not
> > involve modifying the underlying av1 codec itself.
> >
> > If there are any alternative solutions where we can expose a single
> > stream that needs two codecs, then i am open to implementing that.
> >
> > Otherwise, this solution improves the current situation where there is
> > no way for the user to extract the alpha channel (albeit in a hacky
> > way).
>
> I have discussed this on IRC,

Thank you for discussing this!

> and as far as I can see we have multiple
> cases where an additional image has to be decoded by a separate
> decoder context to get alpha, namely:
>
> - vp8/9 (currently handled by AV_PKT_DATA_MATROSKA_BLOCKADDITIONAL)
> - HEIF-like formats (such as AVIF), which are codec agnostic (you can
> put JPEG, HEVC, AV1 etc into HEIF)
>
> This means the logic (preferably) should not be codec-specific, and
> would allow to implement at least two separate
>
> My proposal - which Anton didn't really like - was to add a side data
> entry AV_PKT_DATA_AUXILIARY_IMAGE , which could have type (ALPHA,
> DEPTH etc) as well as the relevant packet(s) - as well as if any of
> specifications define codec initialization data, that. Then
> libavcodec/decode or so could initialize a secondary decoder context
> for alpha images and add the third plane if parameters match what's
> required.
>

My only concern about this solution is that, while the demuxer can
easily put the alpha data in the side data, we will then have to
modify the decoder (in case of HEIF, we will have to change each
underlying codec like av1, hevc, jpeg, etc) to support handling of
this side data. I was just hoping it would be nicer if we can have
something simpler without having to change every individual libavcodec
implementation. ffmpeg already has a way to cleanly initialize codecs
and pass data to them, so it would be nicer if we can leverage that
somehow instead of making each libavcodec implementation handle this
side data type.

For example, the proposed solution is what we do for the VP8 alpha
demuxing and there is already some fragmentation. The libvpx based vp8
decoder supports alpha channel decoding while the native ffmpeg vp8
decoder does not support it. So, a user has to explicitly pass -c:v
libvpx-vp8 to be able to decode files with alpha channel properly.

> This would also make remux possible, since containers that do not
> support such auxiliary images would ignore the alpha plane, and those
> that do would properly pass it through.
>

The remux is possible in the multi-stream approach/hack as well. But I
do see your point.

> As for encoding, not fully sure how it should be integrated, if any
> encoders actually at this moment do proper alpha coding, or do all API
> clients have to separately encode with one context the primary image,
> and the alpha with another?

I am not sure about other codecs, but in the case of AVIF/AV1, the
encoder does not understand/support alpha channels. The only way to do
it is to use two separate encoders.

> That said, the multi-stream implementation
> is already merged on the muxer side, so that can be utilized in the
> mean time, even though the auxiliary images are not technically a
> separate stream within HEIF-likes.
>
> Jan
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".



-- 
Vignesh