[FFmpeg-devel] [PATCH v3 01/10] avfilter/dnn: Refactor DNN parameter configuration system

[Zhao Zhili]

On 2024/5/6 21:42, Zhao Zhili wrote:
>
>
>> On May 6, 2024, at 19:48, Guo, Yejun 
>> <yejun.guo-at-intel.com at ffmpeg.org> wrote:
>>
>>> -----Original Message-----
>>> From: ffmpeg-devel <ffmpeg-devel-bounces at ffmpeg.org> On Behalf Of Zhao
>>> Zhili
>>> Sent: Tuesday, April 30, 2024 3:12 PM
>>> To: ffmpeg-devel at ffmpeg.org
>>> Cc: Zhao Zhili <zhilizhao at tencent.com>
>>> Subject: [FFmpeg-devel] [PATCH v3 01/10] avfilter/dnn: Refactor DNN
>>> parameter configuration system
>>>
>>> From: Zhao Zhili <zhilizhao at tencent.com>
>>>
>>> This patch trying to resolve mulitiple issues related to parameter
>>> configuration:
>>>
>>> Firstly, each DNN filters duplicate DNN_COMMON_OPTIONS, which should
>>> be the common options of backend.
>>>
>>> Secondly, backend options are hidden behind the scene. It's a
>>> AV_OPT_TYPE_STRING backend_configs for user, and parsed by each
>>> backend. We don't know each backend support what kind of options
>>> from the help message.
>>>
>>> Third, DNN backends duplicate DNN_BACKEND_COMMON_OPTIONS.
>>>
>>> Last but not the least, pass backend options via AV_OPT_TYPE_STRING
>>> makes it hard to pass AV_OPT_TYPE_BINARY to backend, if not impossible.
>>>
>>> This patch puts backend common options and each backend options inside
>>> DnnContext to reduce code duplication, make options user friendly, and
>>> easy to extend for future usecase.
>>>
>>> There is a known issue that, for a filter which only support one or two
>>> of the backends, the help message still show the option of all three
>>> backends. Each DNN filter should be able to run on any backend. Current
>>> issue is mostly due to incomplete implementation (e.g., libtorch only
>>> support DFT_PROCESS_FRAME), and lack of maintenance on the filters.
>>
>> This patch 01 basically looks good, two comments:
>> - it is possible that we add one dnn filter with one backend support 
>> first, and then
>> other backends one by one some-time later. So, please adjust the help 
>> message
>> accordingly with only the supported backends.
>
> I can’t figure out how to do that properly without duplicating some 
> code for each
> filter. Do you have any idea?

Fixed in patch v4.

>
>>
>> - is it possible to split this patch into small patches for an easier 
>> detail review?
>
> The configuration methods were scattered across filter common code, 
> filters, backend
> common code and backend implementation. I have written six version, 
> patch 1/10
> is hard to split to small patches like patch 2-10.
>
>>
>>>
>>> For example,
>>>
>>> ./ffmpeg -h filter=dnn_processing
>>>
>>> dnn_processing AVOptions:
>>>   dnn_backend       <int>        ..FV....... DNN backend (from 
>>> INT_MIN to
>>> INT_MAX) (default tensorflow)
>>>     tensorflow      1            ..FV....... tensorflow backend flag
>>>     openvino        2            ..FV....... openvino backend flag
>>>     torch           3            ..FV....... torch backend flag
>>>
>>> dnn_base AVOptions:
>>>   model             <string>     ..F........ path to model file
>>>   input             <string>     ..F........ input name of the model
>>>   output            <string>     ..F........ output name of the model
>>>   backend_configs   <string>     ..F.......P backend configs 
>>> (deprecated)
>>>   options           <string>     ..F.......P backend configs 
>>> (deprecated)
>>>   nireq             <int>        ..F........ number of request (from 
>>> 0 to INT_MAX)
>>> (default 0)
>>>   async             <boolean>    ..F........ use DNN async inference 
>>> (default true)
>>>   device            <string>     ..F........ device to run model
>>>
>>> dnn_tensorflow AVOptions:
>>>   sess_config       <string>     ..F........ config for SessionOptions
>>>
>>> dnn_openvino AVOptions:
>>>   batch_size        <int>        ..F........ batch size per request 
>>> (from 1 to 1000)
>>> (default 1)
>>>   input_resizable   <boolean>    ..F........ can input be resizable 
>>> or not (default
>>> false)
>>>   layout            <int>        ..F........ input layout of model 
>>> (from 0 to 2) (default
>>> none)
>>>     none            0            ..F........ none
>>>     nchw            1            ..F........ nchw
>>>     nhwc            2            ..F........ nhwc
>>>   scale             <float>      ..F........ Add scale preprocess 
>>> operation. Divide each
>>> element of input by specified value. (from INT_MIN to INT_MAX) 
>>> (default 0)
>>>   mean              <float>      ..F........ Add mean preprocess 
>>> operation. Subtract
>>> specified value from each element of input. (from INT_MIN to INT_MAX)
>>> (default 0)
>>>
>>> dnn_th AVOptions:
>>>   optimize          <int>        ..F........ turn on graph executor 
>>> optimization (from 0
>>> to 1) (default 0)
>>> ---
>>> libavfilter/dnn/dnn_backend_common.h   |  13 ++-
>>> libavfilter/dnn/dnn_backend_openvino.c | 146 ++++++++++---------------
>>> libavfilter/dnn/dnn_backend_tf.c       |  82 +++++---------
>>> libavfilter/dnn/dnn_backend_torch.cpp  |  67 ++++--------
>>> libavfilter/dnn/dnn_interface.c        |  89 +++++++++++++++
>>> libavfilter/dnn_filter_common.c        |  38 ++++++-
>>> libavfilter/dnn_filter_common.h        |  39 +++----
>>> libavfilter/dnn_interface.h            |  67 +++++++++++-
>>> libavfilter/vf_derain.c                |   6 +-
>>> libavfilter/vf_dnn_classify.c          |   4 +-
>>> libavfilter/vf_dnn_detect.c            |   4 +-
>>> libavfilter/vf_dnn_processing.c        |   4 +-
>>> libavfilter/vf_sr.c                    |   6 +-
>>> 13 files changed, 336 insertions(+), 229 deletions(-)
>> _______________________________________________
>> ffmpeg-devel mailing list
>> ffmpeg-devel at ffmpeg.org
>> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>>
>> To unsubscribe, visit link above, or email
>> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
>