[FFmpeg-devel] [PATCH] Runtime detection for the number of processors/cores

[Alexander Strange]

On May 21, 2008, at 4:07 PM, Roman Shaposhnik wrote:

> On Wed, 2008-05-21 at 11:24 +0100, M?ns Rullg?rd wrote:
>> Philipp Meinen wrote:
>>> Hello FFmpeg Team
>>>
>>> The attached patch is an attempt to allow runtime detection for the
>>> number of online processors/cores instead of having to specify the
>>> number of threads. The idea is to type:
>>>    ffmpeg -threads 0 ....
>>> to use as many threads as processors/cores are online.
>>>
>>> I guess cmdutils.c/h is not the right file to place the new
>>> detection function. To which file should this function belong?
>>>
>>> Comments welcome :)
>>
>> This is the wrong way to go about it.  The number of processors in  
>> the
>> machine is not interesting, the number of processors we're running on
>> is.  On Linux, this information can be found from  
>> sched_get_affinity().
>> Other systems have other methods.
>
> This is a reasonable generic statement, but the nice thing about
> FFmpeg's approach to multithreading is that it sort of self-schedules.
> IOW, even if you create more (far more!) threads than you have actual
> execution units (could be CPUs, could be cores or could even be
> virtual threads -- think Hype-R-Threading) the only overhead is
> at the start-up phase. Thus, using _sysconf(_SC_NPROCESSORS_ONLN)
> seems quite reasonable of a compromise.
>
> On a related note: I still don't quite understand why MPEG based
> codecs don't allow for higher degree of parallelism. Unlike DV codec
> which I've just tested on a nice Maramba box (2 CPUs, 16 cores,
> 128 execution units) and it scaled pretty much linearly to the
> # of cores, the MPEG based ones just refused to go higher than 4.

They only scale to the number of slices in a frame, and then only to  
the first 8 of those, I think. Adding slices is the only way to  
guarantee parallelism in a codec like this, but it always hurts  
compressibility,, which is much more important.
I'm working on it - for intra codecs frame-threading works perfectly  
so far, but for all the other codecs with backreferences I ran into  
some trouble with the buffer API. That shouldn't take too long to get  
around, though, and then I can get on with reading mpegvideo.