[FFmpeg-cvslog] avformat/matroskadec: fix setting channel layout using the Channels element

[James Almer]

ffmpeg | branch: master | James Almer <jamrial at gmail.com> | Mon Jul  4 14:00:21 2022 -0300| [94901a9518664a13f3ac8d0062f6600741b9c9aa] | committer: James Almer

avformat/matroskadec: fix setting channel layout using the Channels element

If the stream's channel layout is first set into a native layout using codec
private parameters, this code here could potentially result in an invalid
native layout where popcnt(ch_layout.u.mask) != ch_layout.nb_channels being
propagated.

Fixes: Timeout printing a billion channels
Fixes: 48099/clusterfuzz-testcase-minimized-ffmpeg_dem_MATROSKA_fuzzer-6754782204788736

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Tested-by: Michael Niedermayer <michael at niedermayer.cc>
Signed-off-by: James Almer <jamrial at gmail.com>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=94901a9518664a13f3ac8d0062f6600741b9c9aa
---

 libavformat/matroskadec.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/libavformat/matroskadec.c b/libavformat/matroskadec.c
index 73ded761fd..ad7ee390a2 100644
--- a/libavformat/matroskadec.c
+++ b/libavformat/matroskadec.c
@@ -2950,10 +2950,10 @@ static int matroska_parse_tracks(AVFormatContext *s)
             st->codecpar->codec_tag   = fourcc;
             st->codecpar->sample_rate = track->audio.out_samplerate;
             // channel layout may be already set by codec private checks above
-            if (st->codecpar->ch_layout.order == AV_CHANNEL_ORDER_NATIVE &&
-                !st->codecpar->ch_layout.u.mask)
+            if (!av_channel_layout_check(&st->codecpar->ch_layout)) {
                 st->codecpar->ch_layout.order = AV_CHANNEL_ORDER_UNSPEC;
-            st->codecpar->ch_layout.nb_channels = track->audio.channels;
+                st->codecpar->ch_layout.nb_channels = track->audio.channels;
+            }
             if (!st->codecpar->bits_per_coded_sample)
                 st->codecpar->bits_per_coded_sample = track->audio.bitdepth;
             if (st->codecpar->codec_id == AV_CODEC_ID_MP3 ||