[FFmpeg-user] Dual Xeon VS Haswell-E, maximize fps

Henk D. Schoneveld belcampo at zonnet.nl
Thu Oct 2 20:12:54 CEST 2014 

On 02 Oct 2014, at 18:06, Ryan M <rymerrick at gmail.com> wrote:

> "The sum of several single threaded encodings will result in more total-fps
> then 1 multithreaded encoding."
> This is interesting.  So whatever route I take (many separate boxes or one
> many core system) feeding it with several jobs at once and limiting it to
> one core is the way to go.
> The issue with what your suggesting then is that if there is only one or
> two jobs to run, a lot of the boxes are going to be sitting there doing
> nothing.  Maximizing the throughput on your system requires it always being
> feed the maximum # of jobs.
With some clever thinking you can divide your source-file in  amount of pieces = # cores-1. This way all cores create 1 audio and # cores -1 video parts with can be catted together and muxed to 1 new file. Catting and muxing doesn’t cost any time in relation to the time needed for encoding. Your source file ProResHQ is afaik already a I-frames only file so dividing by # cores -1 is very easy.  
>   The dual Xeon (or lesser but still many core)
> is going to be more flexible in that regard (so the high price tag is for
> that flexibility).
Yes if the one time investment in thinking is more costly.
> 
> On Wed, Oct 1, 2014 at 12:21 PM, Henk D. Schoneveld <belcampo at zonnet.nl>
> wrote:
> 
>> Some additional things to consider.
>> 
>> Perofrm some short tests, with a 1000 frames file, try with threads 1,
>> threads 2, threads 3, etc
>> Threads 1 gives you 100% of a single core say 4fps. Threads 2 doesn’t
>> perform 8fps, because of overhead, and adding another thread adds less then
>> the 2nd core. The sum of several single threaded encodings will result in
>> more total-fps then 1 multithreaded encoding.
>> 
>> If you want 40fps you can easily add as many DualCelerons as mentioned
>> before to get the desired result, you only need to add the ‘missing’ fps to
>> the CPU you already have.
>> 
>> If you could send a 1000frames file, I’m happy to test te results for the
>> mentioned Celeron.
>> 
>> Henk
>> 
>> On 01 Oct 2014, at 04:07, Ryan M <rymerrick at gmail.com> wrote:
>> 
>>> It is definitely something to think of/consider. Thanks for the input.
>>> 
>>> On Mon, Sep 29, 2014 at 2:14 PM, Henk D. Schoneveld <belcampo at zonnet.nl>
>>> wrote:
>>> 
>>>> Size of 1 system 8 * 8.5 * 2.25 inches, so they are very small just a
>>>> little bigger then an Apple Mini.
>>>> So 8 of them doen’t take that much space, you can stack them.
>>>> 
>>>> On 29 Sep 2014, at 18:37, Ryan M <rymerrick at gmail.com> wrote:
>>>> 
>>>>> Yeah really the actual space/room is the issue there, not sure where
>> i'd
>>>>> put that.
>>>>> With PPSS, it looks like it distributes the encoding of different files
>>>>> over the available resources, it wouldn't spread a single encoding job
>>>> over
>>>>> them correct?
>>>> Yes that’s correct. If 1 resource becomes free, job done, it takes the
>>>> next file to be done.
>>>>> I do need a way to automate ffmpeg instances so more jobs
>>>>> can run at the same time.
>>>> You can run as many jobs as you do have CPU cores, with 8 systems being
>>>> DualCore, 16 jobs at once.
>>>>> My jobs aren't just  a bunch of files sitting in
>>>>> a directory, it is querying a db for files that need to be encoded in
>>>>> various ways.
>>>> You could run multiple instances of PPSS, with different encoding
>>>> parameters. You could query your DB for files with ffmpeg parameters X,
>>>> make symlinks to Directory X
>>>> and
>>>> another query for files with parameters Y and make  symlinks to
>> directory Y
>>>> One PPSS instance runs the X and another instance runs the Y symlinks
>>>> directory.
>>>>> 
>>>>> 
>>>>> On Mon, Sep 29, 2014 at 11:15 AM, Henk D. Schoneveld <
>> belcampo at zonnet.nl
>>>>> 
>>>>> wrote:
>>>>> 
>>>>>> Why wouldn’t it be realistic ?
>>>>>> Have a look at https://code.google.com/p/ppss/ to distribute jobs
>> over
>>>>>> the amount of systems you have and works like a charm.
>>>>>> On 29 Sep 2014, at 17:05, Ryan M <rymerrick at gmail.com> wrote:
>>>>>> 
>>>>>>> I don't know that it is realistic to have 8 systems running though.
>>>> If I
>>>>>>> can pay more and have the approx same amount of throughput with one
>>>>>> system,
>>>>>>> that would be better.
>>>>>> In what way would it be better, less room, yes but that’s very
>> expensive
>>>>>> room you pay for in my opinion.
>>>>>>> I just don't want to pay like $4k  more (for the
>>>>>>> Xeon vs Haswell-EP) and only get a few more fps out of it.  If
>> stepping
>>>>>> up
>>>>>>> to Xeon allows ~40fps like Sean said then it is probably worth it.
>>>>>> Thank
>>>>>>> you very much for the input.
>>>>>>> 
>>>>>>> On Sun, Sep 28, 2014 at 7:23 AM, Henk D. Schoneveld <
>>>> belcampo at zonnet.nl>
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> 
>>>>>>>> On 28 Sep 2014, at 13:13, Henk D. Schoneveld <belcampo at zonnet.nl>
>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On 26 Sep 2014, at 18:36, Ryan M <rymerrick at gmail.com> wrote:
>>>>>>>>> 
>>>>>>>>>> I am putting together hardware to encode a large and ever growing
>>>>>>>> catalog
>>>>>>>>>> of video using ffmpeg/x264.  Much of the source video is 1080p
>>>>>> ProResHQ.
>>>>>>>>>> Currently I have a box with a Haswell 4770k CPU which gets around
>>>>>> 12-15
>>>>>>>>>> fps, I am of course looking to increase that as much as possible.
>>>>>>>>> Have a look at
>>>>>>>> 
>>>>>> 
>>>> 
>> http://www.cpubenchmark.net/cpu.php?cpu=Intel+Core+i7-5960X+%40+3.00GHz&id=2332
>>>>>>>>> and compare with
>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>> 
>> http://www.cpubenchmark.net/cpu.php?cpu=Intel+Core+i7-4770K+%40+3.50GHz&id=1919
>>>>>>>>> or in 1 view
>>>>>>>>> http://www.cpubenchmark.net/compare.php?cmp[]=1919&cmp[]=2364
>>>>>>>>> 7 times the price for less then 2 times the performance.
>>>>>>>>> Spreading the task over more much cheaper CPU’s gives you more bang
>>>> for
>>>>>>>> the buck if realtime encoding isn’t needed.
>>>>>>>>> You could use 8 Dual Celeron systems to get the same performance
>> as 1
>>>>>>>> E5-2690 where CPU cost would be 8x52.45 vs. 1x2299.99 Of course
>>>>>>>> motherboards etc are needed and housing but a low cost MB and
>> Housing
>>>>>> can
>>>>>>>> be bought at < 100.00 a piece. 8*(100+52.45) ~ 1220.00
>>>>>>>>> Saved more then $1.000 And the Xeon still needs a motherboard and
>>>> case
>>>>>>>> of course.
>>>>>>>> The CPU Mark and single thread performance are good indicators for
>>>>>>>> relative ffmpeg encoding speeds. I do have i5 i7 and the named
>>>> Celerons
>>>>>>>> which I compared extensively with the same source files and encoding
>>>>>>>> parameters.
>>>>>>>> Disk I/O isn’t any problem, think of it, 5 times BluRay bitrates
>>>> results
>>>>>>>> in 90Kb/s < 12MB/s. Every modern disk has no problem with that and
>>>> even
>>>>>>>> old-fashioned nics, 100Mb/s can cope with that.
>>>>>>>>>> 
>>>>>>>>>> Trying to determine if getting the latest dual CPU Xeon (such as
>>>>>> E5-2690
>>>>>>>>>> v3) setup is going to be worth the significant additional cost
>> over
>>>>>> the
>>>>>>>>>> best Haswell-E (Core i7-5960X).
>>>>>>>>>> 
>>>>>>>>>> I've read lots of articles/posts and it is not clear to me.  I
>> know
>>>>>>>> worth
>>>>>>>>>> is subjective but looking to know if there'll be significant
>>>> increase
>>>>>> in
>>>>>>>>>> fps using 2 Xeons.  I need to justify the $6-7k price tag.
>>>>>>>>>> 
>>>>>>>>>> Any insight/experience would be appreciated.
>>>>>>>>>> 
>>>>>>>>>> thanks
>>>>>>>>>> Ryan
>>>>>>>>>> _______________________________________________
>>>>>>>>>> ffmpeg-user mailing list
>>>>>>>>>> ffmpeg-user at ffmpeg.org
>>>>>>>>>> http://ffmpeg.org/mailman/listinfo/ffmpeg-user
>>>>>>>>> 
>>>>>>>>> _______________________________________________
>>>>>>>>> ffmpeg-user mailing list
>>>>>>>>> ffmpeg-user at ffmpeg.org
>>>>>>>>> http://ffmpeg.org/mailman/listinfo/ffmpeg-user
>>>>>>>> 
>>>>>>>> _______________________________________________
>>>>>>>> ffmpeg-user mailing list
>>>>>>>> ffmpeg-user at ffmpeg.org
>>>>>>>> http://ffmpeg.org/mailman/listinfo/ffmpeg-user
>>>>>>>> 
>>>>>>> _______________________________________________
>>>>>>> ffmpeg-user mailing list
>>>>>>> ffmpeg-user at ffmpeg.org
>>>>>>> http://ffmpeg.org/mailman/listinfo/ffmpeg-user
>>>>>> 
>>>>>> _______________________________________________
>>>>>> ffmpeg-user mailing list
>>>>>> ffmpeg-user at ffmpeg.org
>>>>>> http://ffmpeg.org/mailman/listinfo/ffmpeg-user
>>>>>> 
>>>>> _______________________________________________
>>>>> ffmpeg-user mailing list
>>>>> ffmpeg-user at ffmpeg.org
>>>>> http://ffmpeg.org/mailman/listinfo/ffmpeg-user
>>>> 
>>>> _______________________________________________
>>>> ffmpeg-user mailing list
>>>> ffmpeg-user at ffmpeg.org
>>>> http://ffmpeg.org/mailman/listinfo/ffmpeg-user
>>>> 
>>> _______________________________________________
>>> ffmpeg-user mailing list
>>> ffmpeg-user at ffmpeg.org
>>> http://ffmpeg.org/mailman/listinfo/ffmpeg-user
>> 
>> _______________________________________________
>> ffmpeg-user mailing list
>> ffmpeg-user at ffmpeg.org
>> http://ffmpeg.org/mailman/listinfo/ffmpeg-user
>> 
> _______________________________________________
> ffmpeg-user mailing list
> ffmpeg-user at ffmpeg.org
> http://ffmpeg.org/mailman/listinfo/ffmpeg-user

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <https://ffmpeg.org/pipermail/ffmpeg-user/attachments/20141002/8bc9a7f0/attachment.asc>

More information about the ffmpeg-user mailing list