[FFmpeg-devel] [PATCH]avfilter: add Intel IPP library based x86 optimized video scaling filter

[Jan Ekström]

On Mon, Jun 7, 2021 at 11:44 PM Soft Works <softworkz at hotmail.com> wrote:
> > -----Original Message-----
> > From: ffmpeg-devel <ffmpeg-devel-bounces at ffmpeg.org> On Behalf Of
> > Nicolas George
> > Sent: Monday, June 7, 2021 6:57 PM
> > To: FFmpeg development discussions and patches <ffmpeg-
> > devel at ffmpeg.org>
> > Subject: Re: [FFmpeg-devel] [PATCH]avfilter: add Intel IPP library based x86
> > optimized video scaling filter
> >
> > Soft Works (12021-06-05):
> > > And I agree to that disagreement. Also we shouldn't start acting as if
> > > the nonfree category wouldn't exist at all and everything that would
> > > fall into that category would suddenly no longer be acceptable.
> >
> > This is exactly what we should do. We should not have accepted code that
> > links to non-free libraries in the first place.
>
> While I respect and even understand your opinion, I don't share it.  When there's a way to achieve so much better results, then I'd want to use it, just like hardware accelerations that are not always open source either. Surely, it makes sense to decide on a case-by-case basis.
> Though, just blindly rejecting everything that isn't completely open, might cause some significant(!) fork to emerge that covers all these things, which gladly hasn't happened yet.
>

If you just go without any rhetoric and just look at what "nonfree"
(which IMHO is an awful name for the configuration option, it's just
"non-distributable")

We can start with the history of the option - originally added in
3fe142e2555ec8b527f2ff4cc517c015a114e91a (Jan, 2008) - to denote that
due to the libamrnb/libamrwb wrappers being based on reference code,
them being effectively not compatibly licensed even if someone was
nice enough to make an open source wrapper around them. So it was all
about open source, just licensing incompatibilities that were found
out (just like some AAC library was later found to re-utilize
reference code which was not under a friendly license).

Then if we look at what the nonfree option is currently utilized for:

capture card hardware integration (1):
    decklink

open source that just happens to have incompatible licensing with
regards to GPL (3):
    libfdk_aac
    openssl
    libtls

(old) CUDA SDK things (3):
    cuda_nvcc
    cuda_sdk
    libnpp

Thus looking at both historical point of that option, as well as its
current uses, I don't see there's any reason to include a wrapper for
a closed source scaling library that does not benefit the project (it
re-implements already existing capability).

For CPU based video scaling and color space conversions we already have:
1. swscale (which is problematic due to its early 2000s design, but
recently got slice threading by Anton, so kudos to him)
2. zscale filter which is zimg-based (which also has intra picture
threading if someone just cares to enable it in the filter)

So I unfortunately disagree with adding this library wrapper for a
closed source scaling/color space conversion library.

> > > If there's an open source or native implementation available, this
> > > should always receive precedence, but otherwise it's not a valid
> > > argument IMO to reject based on some fictional future (= someday,
> > > somebody might contribute a native implementation).
> >
> > The Libre software movement has achieved so much thanks to people who
> > have rejected this kind of compromise. But now people are taking it for
> > granted, and it loses ground to Open Source, where software giants get free
> > work without giving anything in return beyond token contribution and quasi-
> > proprietary code.
>
> I don't think you could say that about Intel. They have open-sourced a huge amount of code in the recent years, they have more than a dozen developers that are making contributions to ffmpeg, so what you are saying may apply to others but not in this case.
> IPP is huge and exists for a long time. It has formerly been a commercial product, and now it's free, hopefully open-sourced at some time - I'd second that.
>
> What makes this a special case, is that IPP includes functions, each with many implementations, optimized for individual CPU models. This is something that a single or few open source developers couldn't easily replicate in a reasonable amount of time, at least not in the same detail and distinction between models, simply because nobody has access to that wide range of models to verify all these implementation on each model. Plus, there's has surely been a flow of internal knowledge into those implementations.
>
> Nothing is impossible, but in this case I think that it's just not realistic to expect that this could ever be paralleled by another implementation.
>
> That's why I'm all for including this.

I don't disagree with swscale or zimg being not necessarily the best
optimized (although to be honest, IMHO they are very much optimized on
the general level), but instead of talking about this proprietary
library as a mythical power-beast, maybe it would be more productive
to see which things (in a proper benchmark) are on which level when
compared to each other. Figure out the playing field and if you
profile you could even figure out where your bottlenecks are, and what
you *actually* wish for.

I think that would be much more useful than the theoretical discussion
about how much greater this thing would be compared to the open source
alternatives.

Best regards,
Jan