[FFmpeg-devel] native mode in FFmpeg DNN module

[Steven Liu]

> 在 2019年5月24日，20:34，Pedro Arthur <bygrandao at gmail.com> 写道：
> 
> Em qui, 23 de mai de 2019 às 00:06, Guo, Yejun <yejun.guo at intel.com> escreveu:
>> 
>> 
>> 
>>>>>>>> Option 2)
>>>>>>>> Write c code in FFmpeg to convert tensorflow file format (format 1)
>>>> directly
>>>>>>> into memory representation (format 3), and so we controls everything in
>>>>>>> ffmpeg community. And the conversion can be extended to import more
>>>> file
>>>>>>> formats such as torch, darknet, etc. One example is that OpenCV uses
>>>> this
>>>>>>> method.
>>>>>>>> 
>>>>>>>> The in memory representation (format 3) can still be current.
>>>>>>>> 
>>>>>>> 
>>>>>>> Option 2 would be ideal, as it does not introduce any dependency for
>>>>>>> using the native backend.
>>>>>>> Yet I'm not sure  how complex implementing the tf model reader can
>>> be,
>>>>>>> If I remember correctly the student said it was not trivial at the
>>>>>>> time.
>>>>>> 
>>>>>> yes, it is not easy, but I think it is worthy to do it. Here is a reference
>>>> example
>>>>>> for the complexity, see
>>>>>> 
>>>> 
>>> https://github.com/opencv/opencv/blob/master/modules/dnn/src/tensorflow/
>>>>>> tf_importer.cpp.
>>>>>> 
>>>>>>> 
>>>>>>> Is the tf model file stable? if not it will be a maintenance burden to
>>>>>>> keep it working whenever tf releases a new version. This point makes
>>>>>>> me think having control over our file format is good.
>>>>>> 
>>>>>> imho, this issue is always there, no matter which method used, unless our
>>>>>> format could be exported by tensorflow (it has little possibility).
>>>>>> 
>>>>>> Whenever tf releases a new version with a new file format, we still have
>>> to
>>>>>> change the python script in phase 1 (convert tf file model to our format)
>>>> which
>>>>>> is even an external dependency at
>>>>>> https://github.com/HighVoltageRocknRoll/sr,
>>>>>> 
>>>>>> As from effort perspective, the current implementation is better since
>>>> python
>>>>>> script is simpler. But I think we are still worth implementing option 2 as
>>> the
>>>>>> ideal technical direction.
>>>>> 
>>>>> I checked a bit more about https://github.com/HighVoltageRocknRoll/sr, it
>>> is
>>>> actually
>>>>> not an converter (from tf model to native model), but hard code for given
>>>> models.
>>>>> And the native model is not exactly the same as tf model, it even changes
>>> the
>>>> behavior
>>>>> of pad parameter of conv layer.
>>>>> 
>>>>> If community is open to option 2, I'll try it.
>>>>> 
>>>> Option 2 is fine for me.
>>> 
>>> that's great, :)
>> 
>> looks that option 2 is a bit complex, TF model file is in protocol buffers (protobuf) format and not easy to parse it with simple c code.
>> 
>> Since there is no official c support for protobuf, let's first image how the work can be done via official c++ support.
>> 
>> 1. get protobuf compiler protoc, .h header files and .so library files (download or build from https://github.com/protocolbuffers/protobuf/tree/master/src).
>> 2. get tensorflow model's .proto files from https://github.com/tensorflow/tensorflow/tree/master/tensorflow/core/framework.
>> 3. generate .cc/.h files from .proto files (see step 2) via protoc (see step 1).
>> 4. let the generated .cc/.h files be part of ffmpeg source tree, and build with protobuf header/library files.
>> 5. at run time, the protobuf libraries are invoked. It means that the system should have installed protobuf dev package.
>> 
>> furthermore, there is a compatible problem between the protobuf compiler, header files and library files.
>> So, as a practice to fix it, the method is to make the protobuf source code be part of ffmpeg source tree. (it is a common practice, so we can many other projects contain the protobuf source code).
>> 
>> I guess the above method is not acceptable in ffmpeg. I would be glad to continue if the community embrace this change. :)
> Indeed I think it is not acceptable.
> 
>> 
>> While the current implementation has external dependency, my new suggestion is:
>> -  add a python script under .../libavfilter/dnn/  (all other dnn source files will be also moved here later), so ffmpeg has the full control on it.
> I'm not sure about the policy on putting secondary scripts with the
> main code, but another option is to create a repo controlled by ffmpeg
> maybe?
> I think this option would also help GSoC students that work with dnn,
> so they don't have to depend on previous students maintaining
> independent repositories.

Yes, I agreed with you.
I think this is a better way.
maintaining the repositories at one repo controlled by ffmpeg.
> 
>> -  it is a script to convert tensorflow model file into native model file. (other formats such as caffe, torch can also be supported later if needed)
>> 
>> thanks.
>> 
>> _______________________________________________
>> ffmpeg-devel mailing list
>> ffmpeg-devel at ffmpeg.org
>> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>> 
>> To unsubscribe, visit link above, or email
>> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".

Thanks
Steven