[FFmpeg-devel] [GSOC] [PATCH] DNN module introduction and SRCNN filter update

[Pedro Arthur]

2018-05-29 5:14 GMT-03:00 Sergey Lavrushkin <dualfal at gmail.com>:
> 2018-05-29 4:08 GMT+03:00 Pedro Arthur <bygrandao at gmail.com>:
>>
>> 2018-05-28 19:52 GMT-03:00 Sergey Lavrushkin <dualfal at gmail.com>:
>> > 2018-05-28 9:32 GMT+03:00 Guo, Yejun <yejun.guo at intel.com>:
>> >
>> >> looks that no tensorflow dependency is introduced, a new model format
>> >> is
>> >> created together with some CPU implementation for inference.   With
>> >> this
>> >> idea, Android Neural Network would be a very good reference, see
>> >> https://developer.android.google.cn/ndk/guides/neuralnetworks/. It
>> >> defines how the model is organized, and also provided a CPU optimized
>> >> inference implementation (within the NNAPI runtime, it is open source).
>> >> It
>> >> is still under development but mature enough to run some popular dnn
>> >> models
>> >> with proper performance. We can absorb some basic design. Anyway, just
>> >> a
>> >> reference fyi.  (btw, I'm not sure about any IP issue)
>> >>
>> >
>> > The idea was to first introduce something to use when tensorflow is not
>> > available. Here is another patch, that introduces tensorflow backend.
>> I think it would be better for reviewing if you send the second patch
>> in a new email.
>
>
> Then we need to push the first patch, I think.
Not necessarily, 'git send-email' may give you a glimpse of how it is done.

>
>>
>> >
>> >
>> >> For this patch, I have two comments.
>> >>
>> >> 1. change from "DNNModel* (*load_default_model)(DNNDefaultModel
>> >> model_type);" to " DNNModel* (*load_builtin_model)(DNNBuiltinModel
>> >> model_type);"
>> >> The DNNModule can be invoked by many filters,  default model is a good
>> >> name at the filter level, while built-in model is better within the DNN
>> >> scope.
>> >>
>> >> typedef struct DNNModule{
>> >>     // Loads model and parameters from given file. Returns NULL if it
>> >> is
>> >> not possible.
>> >>     DNNModel* (*load_model)(const char* model_filename);
>> >>     // Loads one of the default models
>> >>     DNNModel* (*load_default_model)(DNNDefaultModel model_type);
>> >>     // Executes model with specified input and output. Returns
>> >> DNN_ERROR
>> >> otherwise.
>> >>     DNNReturnType (*execute_model)(const DNNModel* model);
>> >>     // Frees memory allocated for model.
>> >>     void (*free_model)(DNNModel** model);
>> >> } DNNModule;
>> >>
>> >>
>> >> 2. add a new variable 'number' for DNNData/InputParams
>> >> As a typical DNN concept, the data shape usually is: <number, height,
>> >> width, channel> or <number, channel, height, width>, the last component
>> >> denotes its index changes the fastest in the memory. We can add this
>> >> concept into the API, and decide to support <NHWC> or <NCHW> or both.
>> >
>> >
>> > I did not add number of elements in batch because I thought, that we
>> > would
>> > not feed more than one element at once to a network in a ffmpeg filter.
>> > But it can be easily added if necessary.
>> >
>> > So here is the patch that adds tensorflow backend with the previous
>> > patch.
>> > I forgot to change include guards from AVUTIL_* to AVFILTER_* in it.
>> You moved the files from libavutil to libavfilter while it was
>> proposed to move them to libavformat.
>
>
> Not only, it was also proposed to move it to libavfilter if it is going to
> be used only
> in filters. I do not know if this module is useful anywhere else besides
> libavfilter.