[FFmpeg-user] CPU and GPU

[Mark Filipak]

On 03/02/2020 07:24 AM, Moritz Barsnick wrote:
> On Mon, Mar 02, 2020 at 04:57:23 -0500, Mark Filipak wrote:
>> I used to work at Intel as a product line architect. I sat on 3
>> divisional planning councils, including graphics processors, and
>> co-chaired one.
> 
> [OT, sorry]

OT, eh? Just like Congressional testimony.

> BTW, I have worked as Intel Corp. as well, incidentally both in
> graphics processors ("Petascale Development" ;-)) and in FPGA
> technology. Both in the same project, and before I was all too avid in
> video technology.

Oh goodie! Hey, Brother, when did you work at Intel? I'll bet in the 
early 00s. I was at San Tomas 4. I see from streetview that the building 
isn't even there any more. ;-(
I lived in Felton. $-)

>> I outlined what appeared to me to be the direction of open source video
>> development: That what is developed using open source code is migrated
>> to FPGAs and hence to GPUs. I asked if I have it right (correct).
> 
> I'd like to pitch in to what Carl Eugen doubted: You hardly want to
> "convert" software to hardware...

Agreed; totally. You must not have read the whole thread as I mentioned 
mimicking procedural code via finite state machines (FSMs). As you note 
below, such things as MMX & SSE are CPU opcode extensions that, though 
optimized for graphics, are still code and don't come close to the speed 
of FSMs. Most FPGAs operate as peripherals that require OS drivers, 
while some (few) FPGAs are populated entirely by FSMs that are fed (and 
operate directly on) streams.

> ...Simplified (not sure about all the
> terms): In software, execution is linear, procedular, while in
> hardware, everything is parallel by design (while data flow again can
> be "linear"). You wouldn't want to give up this advantage of hardware
> over software. You want to mix the best of both worlds.
> 
> (BTW, for those who aren't aware: hardware, such as chips, ASICs, and
> also FGPA content, is described in something you could call software.
> But it's not procedural in the same way.)
> 
> What hardware would do, is to build accelerated "calculators" for
> certain operations. This might be a complete codec (raw in, encoded
> out, like GPUs do, and probably like Amazon AWS does), or just bits of
> hardware capable of computations which standard CPU components aren't
> capable of at the same speed. Think MMX, SSE, but specialized for
> graphics or e.g. vector accelerations. Either of these would be hooked
> into a software flow, as ffmpeg does with hardware ("GPU") encoding.
> 
> Many smaller video related operations could already be shifted to GPU
> hardware, but I believe ffmpeg's GPU accelerations don't take advantage
> of this. Perhaps its OpenCL and OpenGL uses do, but I don't believe
> there are many. (Or is there any ffmpeg software encoder assisted by
> hardware, except for the MMX/SSE/AVX type CPU accelerations?)

I haven't used them, but I see from posts to this list that some folks 
are applying CUDA-specific filters.

>> Of course, the ffmpeg crew works to exploit the capabilities that are
>> thereby created. This is all to be celebrated. What is it that you
>> object to? What in what I wrote is wrong or that industry chip providers
>> would find offensive?
> 
> If someone took the ffmpeg sources to "reimplement" them in hardware,
> selling that service without open-sourcing their efforts. I believe
> that is what is meant.

That is not what I meant. What I'm getting at is that it appears that 
the latest streaming protocols & methods are being developed via ffmpeg, 
then being tested-debugged-retested-etc. via ffmpeg users, then 
implemented in FPGAs by software-in-silicon vendors like NGCodec, 
MainConcept, Beamr, etc., and finally cast into FSMs in GPUs. Good for 
all: Open source developers get support & methods to 'talk' to chips and 
GPU makers get well developed, proven protocols & methods.

> Sorry if I drifted off, used some incorrect terms, or have a
> misunderstanding of this. It's just that my understanding of "FPGA
> based" seems to be different than yours.

Well, I'm very familiar with FPGAs and have done many FPGA designs.