[FFmpeg-devel] cur_channel is wrong in AAC encoder / search_for_ms

[Alex Converse]

On Mon, May 10, 2010 at 4:06 AM, Kostya <kostya.shishkov at gmail.com> wrote:
> On Mon, May 10, 2010 at 12:27:57AM -0700, Isak Jonsson wrote:
>> [Erroneously sent this to libav-user earlier]
>>
>> Hello ffmpeg-devel. I'm doing stereo encoding with the AAC encoder (r23058) but occasionally get SIGSEGV. I have tracked things down to this (probable) cause:
>>
>> In aacenc.c:546 (large for-loop of aac_encode_frame), s->cur_channel is set to 0, then 1. In aacenc.c:580, search_for_ms gets called with cur_channel == 1. So in ?aaccoder.c:890-891 of (search_for_ms, calculating band1 address and then using it), the array index is too high. In a heavy loaded environment, this leads to segfaults.
>>

Fixed in r23094.

>> My workaround for now is to set cur_channel to 0 right before call to search_for_ms. No more segfaults, but probably not the totally right thing<tm> either?
>
> Gee, IIRC that was intended for searching M/S on all available channel
> pairs, it shan't be called for the second channel at all. Hopefully Alex
> will fix it.
>

We were only ever attempting to search once per CPE however the active
channel was set to right after some preparatory work on the right
channel. The logic in r23094 should be apparent.