[FFmpeg-user] Working with HLS Multiple Audio Renditions

[Alfredo Di Napoli]

Hey folks,

sorry for the noise, but I posted this question one week ago and I’m still
struggling with HLS and multiple audio renditions.

I have now upgraded to use ffmpeg 3.1.1, and I have formalised the problem
in this Super User
question, for which I will paste the link (I hope I’m not violanting the
netiquette of this list!):

http://superuser.com/questions/1099442/ffmpeg-and-hls-multiple-audio-renditions

It would be great if you guys could give me 5 mins of your time to point me
on the right track!

Thanks,

Alfredo

<http://superuser.com/questions/1099442/ffmpeg-and-hls-multiple-audio-renditions>
<http://superuser.com/questions/1099442/ffmpeg-and-hls-multiple-audio-renditions>
On 4 July 2016 at 12:58, Alfredo Di Napoli <alfredo.dinapoli at gmail.com>
wrote:

> Hey guys!
>
> First of all thanks for all the outstanding work you guys are doing on
> FFmpeg, it's my go-to
> solution to work with audio and video streams.
>
> I'm currently trying to crack the "Multiple Audio Renditions" nut; I use
> JWPlayer 7 in my
> everyday job and they recently released support for alternate audio
> streams for HLS as
> described in this part of the standard (Master Playlist with Alternative
> audio):
>
> https://tools.ietf.org/html/draft-pantos-http-live-streaming-19#section-8.6
>
> An example of how this should look like is visible here:
>
>
> https://support.jwplayer.com/customer/portal/articles/1761348-multiple-audio-renditions
>
> Without going into the gory details, at work we allow the users to record
> videos using an iPad camera,
> and we now want to allow them to also record multiple audio from different
> sources (BT mics, for example),
> and then having our transcoding server produce an HLS playlist where they
> can switch the different
> audio tracks with ease.
>
> **I'm using ffmpeg 2.7.2**
>
> As far as I know, FFmpeg does not offer any "out of the box" support for
> this in the `hls` muxer, by
> which I mean the muxer only segment an input media into `.ts` segments
> (audio + video) and creates a simple .m3u8 playlist,
> so my current flow is the following:
>
> * I use the `hls` muxer with the `-an` to produce audioless segments, all
> packed up in a valid manifest which
> ffmpeg generates. Example (`video.m3u8`):
>
> ```
> #EXTM3U
> #EXT-X-VERSION:3
> #EXT-X-TARGETDURATION:63
> #EXT-X-MEDIA-SEQUENCE:0
> #EXTINF:60.458333,
> video_0.ts
> #EXTINF:59.625000,
> video_1.ts
> #EXTINF:62.250000,
> video_2.ts
> ....
> #EXT-X-ENDLIST
> ```
>
> * I get the audio tracks I receive from the outside, and I use the
> `segment` muxer to map those into valid segments:
>
> ```
> ffmpeg -i foo.mp3 -threads 0 -ac 2 -acodec libmp3lame -async 1 -ar 44100
> -muxdelay 0 -f segment \
>         -segment_time 60 -segment_list_size 0 -segment_format mp3
> -segment_list "foo_%d.mp3" audio1.m3u8
> ```
>
> Which yields:
>
> ```
> #EXTM3U
> #EXT-X-VERSION:3
> #EXT-X-MEDIA-SEQUENCE:0
> #EXT-X-ALLOW-CACHE:YES
> #EXT-X-TARGETDURATION:61
> #EXTINF:60.003265,
> foo_1.mp3
> #EXTINF:60.003265,
> foo_2.mp3
> #EXTINF:60.003265,
> foo_3.mp3
> ...
> #EXT-X-ENDLIST
> ```
>
> * I manually assemble the master playlist myself to conform with the
> standard. Real wold example of what I'm producing (`index.m3u8`):
>
> ```
> #EXTM3U
> #EXT-X-VERSION:3
>
>
> #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=1200000,RESOLUTION=960x540,NAME="Main",CODECS="avc1.66.30",AUDIO="mp3"
> video.m3u8
>
> #EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="mp3",LANGUAGE="en",NAME="Left
> Part",DEFAULT=YES,AUTOSELECT=YES,URI="audio1.m3u8"
> #EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="mp3",LANGUAGE="en",NAME="Right
> Part",DEFAULT=NO,AUTOSELECT=NO,URI="audio2.m3u8"
> ```
>
> This produces a valid playlist which I'm then able to stream, but
> unfortunately it seems to lose the audio/video sync pretty quickly,
> and generally speaking the way I'm doing things feels a bit wrong. If I
> jump to a particular point in the video, the audio stops
> playing or simply the wrong one is played. Maybe my segment size is too
> big?
>
> *Thus, my questions for you guys are:*
>
> 1. Which is the recommended workflow for producting HLS playlist with
> multiple audio?
>    Can you guys point me to any helpful tutorial/documentation?
>
> 2. Currently I'm doing several passes;
>     First pass: Produce the audioless video segments
>     Second .. N passes: For each audio input the user recorded, I ran the
> `segment` muxer.
>     Is this a correct way to approach the problem? Which are the
> constraints I need to enforce on the codecs/bitrates to make
>     sure I'm not producing audio segments which won't sync correctly?
>
> 3. Shall I simply give up any hope of making this work with separate audio
> files and simply switch working with media having 1 video
> stream and N audio streams? I can imagine at least 2 other scenarios to
> make this work:
>     a. I would run ffmpeg and map the video stream (0:0) to the HLS muxer,
> and all the N audio streams to the segment muxer, hoping
>        ffmpeg will be able to produce synced chunks.
>     b. I would run ffmpeg to produce a "normal" HLS playlist, then run it
> again, on each chunk, to separate the audio from the video.
>        The main reasoning here would be, again, to have ffmpeg deal with
> all the transcoding and the syncing.
>
> As you guys can see, I'm quite confused and starting to ramble, so I'm
> putting my trust in you audio/video gurus to help me out ;)
>
> Sorry for the lengthy email and I hope I've made myself clear enough!
>
> Peace,
>
> Alfredo
>
>