[FFmpeg-devel] [FFmpeg-user] fMP4 generation speed

[Ronak Patel]

Hi,

We’d like to start working on this real soon and want to make sure we properly fix this. We understand this code is used by more than just fMP4 assets and even live streaming use cases.

Can anyone please help shed some light on all use cases we should make sure are working after we make our fixes?

We’re afraid that the number of use cases here are unknown and we’ll be stuck fixing this for an unknown time.

Can anyone please shed some light on this?

Thanks

Ronak

> On Jul 13, 2018, at 8:05 AM, Ronak Patel <ronak2121-at-yahoo.com at ffmpeg.org> wrote:
> 
> I was wondering if anyone on the development list was familiar with this code and could provide pointers on how to fix up the code due to the below problem.
> 
> 
>> On Jul 12, 2018, at 7:27 PM, Ronak Patel <ronak2121-at-yahoo.com at ffmpeg.org> wrote:
>> 
>> 
>> 
>>> On Jul 12, 2018, at 6:21 PM, Ronak Patel <ronak2121-at-yahoo.com at ffmpeg.org> wrote:
>>> 
>>> Hey Carl,
>>> 
>>> So I dug into this more today and I have root caused what's exactly happening here. 
>>> 
>>> The problematic code is this: https://github.com/FFmpeg/FFmpeg/blob/master/libavformat/hlsenc.c#L1368 <https://github.com/FFmpeg/FFmpeg/blob/master/libavformat/hlsenc.c#L1368>
>>> This is where the filename is set and the next line actually opens the file. 
>>> 
>>> The logic for this hls_window method is the following:
>>> 
>>> 1. Make a new temporary file.
>>> 2. Write out a new HLS manifest header.
>>> 3. Loop through all available segments and write out all of the entries for them.
>>> 4. Close the temporary file when finished.
>>> 5. Rename the temporary file to the target file name.
>>> 6. Rinse and repeat for every single fragment.
>>> 
>>> Therefore, if you can imagine a 153 hour audio file, we write out a totally new HLS manifest 550800 times (153 * 60 * 60 assuming a 1s fragment duration) that gets progressively larger as each fragment is generated.
>>> 
>>> This is a classic O(N^2) algorithm implementation, instead of:
>>> 
>>> 1. Creating the destination file up front & write the manifest header.
>>> 2. Append the new segment to the file.
>>> 3. If this is the last segment, write out EXT-X-ENDLIST.
>>> 
>>> There's no looping involved, nor the need to make temporary files.
>>> 
>>> FYI that I've noticed the same sort of pattern being applied to MPEG DASH: https://github.com/FFmpeg/FFmpeg/blob/master/libavformat/dashenc.c#L786 <https://github.com/FFmpeg/FFmpeg/blob/master/libavformat/dashenc.c#L786>
>>> 
>>> To implement something like this, looks like we'd have to significantly re-engineer the code. Do you have any pointers on how to go about doing this? Or, would you be able to help do this?
>>> 
>> 
>> I was thinking about this and I think it must have been implemented this way because of live hls streams? Not writing to the main manifest directly and keeping that stream open does release any file system locks that there would be otherwise. 
>> 
>> Is this the reason why this was done this way at all?
>> 
>> 
>>> Thanks for all your help,
>>> 
>>> Ronak
>>> 
>>>> On Jun 27, 2018, at 2:04 PM, Carl Zwanzig <cpz at tuunq.com> wrote:
>>>> 
>>>> Hi,
>>>> 
>>>> I haven't traced it out completely, but take a look at the flag HLS_TEMP_FILE in libavformat/hlsenc.c.
>>>> 
>>>> Later,
>>>> 
>>>> z!
>>> _______________________________________________
>>> ffmpeg-user mailing list
>>> ffmpeg-user at ffmpeg.org
>>> http://ffmpeg.org/mailman/listinfo/ffmpeg-user
>>> 
>>> To unsubscribe, visit link above, or email
>>> ffmpeg-user-request at ffmpeg.org with subject "unsubscribe".
>> 
>> _______________________________________________
>> ffmpeg-user mailing list
>> ffmpeg-user at ffmpeg.org
>> http://ffmpeg.org/mailman/listinfo/ffmpeg-user
>> 
>> To unsubscribe, visit link above, or email
>> ffmpeg-user-request at ffmpeg.org with subject "unsubscribe".
> 
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel