[FFmpeg-user] fMP4 generation speed

Ronak ronak2121 at yahoo.com
Mon Jun 25 16:02:15 EEST 2018


> On Jun 23, 2018, at 12:07 AM, Ronak <ronak2121-at-yahoo.com at ffmpeg.org> wrote:
> 
> 
> 
>> On Jun 22, 2018, at 7:59 PM, Ronak Patel <ronak2121 at yahoo.com> wrote:
>> 
>> 
>>> On Jun 22, 2018, at 5:03 PM, Carl Zwanzig <cpz at tuunq.com> wrote:
>>> 
>>>> On 6/22/2018 1:37 PM, Ronak wrote:
>>>> We have audio files that are more than 100 hours long, and we need them
>>>> to be fragmented quickly. It totally seems like there's an I/O problem
>>>> here because fragmentation is faster the longer we set our fragment size
>>>> to.
>>> Sounds like it's time to break out iostat or sar (or even dtrace) and do some snooping.
>>> 
>>> Are you reading and writing to different drives?  (The usual metric for a spinning drive is 150 IOPS for random access, RAID doesn't necessarily speed that up.)
>>> 
>> 
>> Nope it’s the same disk that we’re reading and writing to. I’m really curious why we can’t just buffer fragments in ram and write them out in larger bursts. Why not make this an option we can set?
>> 
>> This would simulate the performance for a 10 sec fragment size.
>> 
>> I’d imagine you have the same reading buffer regardless of the fragment size.
>> 
>> I’m thinking of running this on even more powerful machines to see if I can get the times down. But is it possible to get an option to tweak the output buffer size?
>> 
> 
> The problem is probably because of this:
> 
> [dash @ 0x7f92cd005600] Opening 'media_0.m3u8.tmp' for writing
> [dash @ 0x7f92cd005600] Opening 'test_60_hr_10sec.m4s' for reading
> [dash @ 0x7f92cd005600] Opening 'test_60_hr_10sec.mpd.tmp' for writing
> [dash @ 0x7f92cd005600] Opening 'media_0.m3u8.tmp' for writing
> [dash @ 0x7f92cd005600] Opening 'test_60_hr_10sec.m4s' for reading
> [dash @ 0x7f92cd005600] Opening 'test_60_hr_10sec.mpd.tmp' for writing
> [dash @ 0x7f92cd005600] Opening 'media_0.m3u8.tmp' for writing
> [dash @ 0x7f92cd005600] Opening 'test_60_hr_10sec.m4s' for reading
> [dash @ 0x7f92cd005600] Opening 'test_60_hr_10sec.mpd.tmp' for writing
> [dash @ 0x7f92cd005600] Opening 'media_0.m3u8.tmp' for writing
> [dash @ 0x7f92cd005600] Opening 'test_60_hr_10sec.m4s' for reading
> [dash @ 0x7f92cd005600] Opening 'test_60_hr_10sec.mpd.tmp' for writing
> [dash @ 0x7f92cd005600] Opening 'media_0.m3u8.tmp' for writing
> [dash @ 0x7f92cd005600] Opening 'test_60_hr_10sec.m4s' for reading
> [dash @ 0x7f92cd005600] Opening 'test_60_hr_10sec.mpd.tmp' for writing
> [dash @ 0x7f92cd005600] Opening 'media_0.m3u8.tmp' for writing
> [dash @ 0x7f92cd005600] Opening 'test_60_hr_10sec.m4s' for reading
> [dash @ 0x7f92cd005600] Opening 'test_60_hr_10sec.mpd.tmp' for writing
> [dash @ 0x7f92cd005600] Opening 'media_0.m3u8.tmp' for writing
> [dash @ 0x7f92cd005600] Opening 'test_60_hr_10sec.m4s' for reading
> [dash @ 0x7f92cd005600] Opening 'test_60_hr_10sec.mpd.tmp' for writing
> [dash @ 0x7f92cd005600] Opening 'media_0.m3u8.tmp' for writing
> 
> Ffmpeg constantly makes this tmp files, and seems to be opening and closing the file handles constantly.
> Is there a way to just keep the main files open and just keep appending to it? Why make temp files at all?
> 

I just tested Apple's segmentation tools: mediafilesegmenter, and I can confirm that a 26 hour file took only:

real	1m35.576s
user	1m32.617s
sys	0m29.705s

time to fragment on a 2015 Macbook Pro 2.2 GHz, 16 GB of RAM.

The big differences I see are:

1. Apple does not create temp files nor open & close file handles repeatedly.
2. Apple does not write out sidx boxes, only moof for these fMP4 files. It seems like sidx boxes are not required when the content only contains audio. Is this true? Can ffmpeg be configured to skip generating them?
3. Apple's fragmentation is consistent. Every single fragment has exactly the same duration.
4. Apple sets the defaultSampleDuration in the moof atoms to 1024 for HE-AACv2 streams, the same as LC-AAC at the same sampling rate. 

I would like to see how I can fix these 4 flaws in ffmpeg

Can anyone please give me pointers on which source files to look at and pointers on how to make these code changes?
I'd like to submit patches for these issues soon.

Thanks,

Ronak

>>> Later,
>>> 
>>> z!
>>> _______________________________________________
>>> ffmpeg-user mailing list
>>> ffmpeg-user at ffmpeg.org <mailto:ffmpeg-user at ffmpeg.org>
>>> http://ffmpeg.org/mailman/listinfo/ffmpeg-user <http://ffmpeg.org/mailman/listinfo/ffmpeg-user>
>>> 
>>> To unsubscribe, visit link above, or email
>>> ffmpeg-user-request at ffmpeg.org <mailto:ffmpeg-user-request at ffmpeg.org> with subject "unsubscribe".
> 
> _______________________________________________
> ffmpeg-user mailing list
> ffmpeg-user at ffmpeg.org <mailto:ffmpeg-user at ffmpeg.org>
> http://ffmpeg.org/mailman/listinfo/ffmpeg-user <http://ffmpeg.org/mailman/listinfo/ffmpeg-user>
> 
> To unsubscribe, visit link above, or email
> ffmpeg-user-request at ffmpeg.org <mailto:ffmpeg-user-request at ffmpeg.org> with subject "unsubscribe".



More information about the ffmpeg-user mailing list