[FFmpeg-devel] [PATCH v1] libavfi/dnn: add Paddle Inference as one of DNN backend

[Guo, Yejun]

> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces at ffmpeg.org> On Behalf Of
> "zhilizhao(赵志立)"
> Sent: Wednesday, May 10, 2023 12:09 PM
> To: FFmpeg development discussions and patches <ffmpeg-
> devel at ffmpeg.org>
> Subject: Re: [FFmpeg-devel] [PATCH v1] libavfi/dnn: add Paddle Inference as
> one of DNN backend
> 
> 
> 
> > On May 10, 2023, at 10:25, WenzheWang <wongwwz at foxmail.com> wrote:
> >
> > Dear Madam or Sir,
> >
> >
> > Hope this email finds you well.
> >
> >
> > I am writing this email since i recently found FFmepg remove DNN native
> backend, and i will be really grateful if you let me know if there is  any new
> plan on libavfilter/dnn.
> >
> >
> > I would like to explain to you again about the addition of dnn paddle
> backend.
> >
> > At  present, ffmpeg only supports openvino and tensorflow backend.
> Among  the current deep learning frameworks, TensorFlow is the most active
> in  development. TensorFlow has 174k stars and pytorch has 66.5k. openvino
> is 4.2k, and the models that openvino can implement are relatively few.  But
> in terms of attention on GitHub, there's no doubt that TensorFlow  and
> pytorch are more promising. Currently, the paddle framework has  reached
> 20.2k stars on github, which is much more widely used and active  than
> frameworks such as mxnet and caffe.
> 
> Stars don't matter much here.
> 
> Just for reference, there is a thread before:
> 
> https://patchwork.ffmpeg.org/project/ffmpeg/patch/20220523092918.9548-
> 2-ting.fu at intel.com/
> 
> >
> > Tensoflow has a very  rich ecosystem. The TensorFlow models library
> updates very quickly and  has existing examples of deep learning applications
> for image  classification, object detection, image generation text, and
> generation  of adversus-network models. The dnn libavfilter module is
> undoubtedly very necessary for tensorflow  backend to support. But the
> complexity of the TensorFlow API and the  complexity of the training are
> almost prohibitive, making it a love-hate  framework.
> >
> > PyTorch framework tends to be applied to academic  fast implementation,
> and its industrial application performance is not  good. For example, Pytorch
> framework makes a model to run on a server,  Android phone or embedded
> system, and its performance is poor compared  with other deep learning
> frameworks.
> >
> >
> > PaddlePadddle  is an open source framework of Baidu, which is also used
> by many people  in China. It is very consistent with the usage habits of
> developers,  but the practicability of the API still needs to be further
> strengthened. However, Paddle is the only deep learning framework I  have
> ever used, which does not configure any third-party libraries and  can be
> used directly by cloning make. Besides, Paddle occupies a small  amount of
> memory and is fast. It also serves a considerable number of  projects inside
> Baidu, which is very strong in industrial application.  And PaddlePaddle
> supports multiple machine and multiple card training.

Imo, my idea is that we can add 1 or 2 dnn backends as discussed at 
http://ffmpeg.org/pipermail/ffmpeg-devel/2022-December/304534.html

The background is that we see different good models from different deep learning
frameworks, and most framework does not support models developed with other 
frameworks due to different model format. imo, we'd support several popular frameworks.