[FFmpeg-cvslog] avcodec/ac3enc: Don't presume ch_layout to be AV_CHANNEL_ORDER_NATIVE

[Andreas Rheinhardt]

ffmpeg | branch: master | Andreas Rheinhardt <andreas.rheinhardt at outlook.com> | Sun Apr  7 12:31:58 2024 +0200| [c3fb0c5babb2ae3b8985d038dd0ce228d52f85d7] | committer: Andreas Rheinhardt

avcodec/ac3enc: Don't presume ch_layout to be AV_CHANNEL_ORDER_NATIVE

It is perfectly legal for users to use a custom layout
that is equivalent to a supported native one.
In this case the union in AVChannelLayout is not an uint64_t mask,
but a pointer to a custom map.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt at outlook.com>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=c3fb0c5babb2ae3b8985d038dd0ce228d52f85d7
---

 libavcodec/ac3enc.c | 6 +-----
 1 file changed, 1 insertion(+), 5 deletions(-)

diff --git a/libavcodec/ac3enc.c b/libavcodec/ac3enc.c
index 7a6bcf7900..dc197c1517 100644
--- a/libavcodec/ac3enc.c
+++ b/libavcodec/ac3enc.c
@@ -2192,18 +2192,14 @@ av_cold int ff_ac3_encode_close(AVCodecContext *avctx)
 static av_cold int set_channel_info(AVCodecContext *avctx)
 {
     AC3EncodeContext *s = avctx->priv_data;
+    uint64_t mask = av_channel_layout_subset(&avctx->ch_layout, ~(uint64_t)0);
     int channels = avctx->ch_layout.nb_channels;
-    uint64_t mask = avctx->ch_layout.u.mask;
 
     if (channels < 1 || channels > AC3_MAX_CHANNELS)
         return AVERROR(EINVAL);
     if (mask > 0x7FF)
         return AVERROR(EINVAL);
 
-    if (!mask)
-        av_channel_layout_default(&avctx->ch_layout, channels);
-    mask = avctx->ch_layout.u.mask;
-
     s->lfe_on       = !!(mask & AV_CH_LOW_FREQUENCY);
     s->channels     = channels;
     s->fbw_channels = channels - s->lfe_on;