[FFmpeg-devel] native mode in FFmpeg DNN module

[Guo, Yejun]

> > > > > > Option 2)
> > > > > > Write c code in FFmpeg to convert tensorflow file format (format 1)
> > directly
> > > > > into memory representation (format 3), and so we controls everything in
> > > > > ffmpeg community. And the conversion can be extended to import more
> > file
> > > > > formats such as torch, darknet, etc. One example is that OpenCV uses
> > this
> > > > > method.
> > > > > >
> > > > > > The in memory representation (format 3) can still be current.
> > > > > >
> > > > >
> > > > > Option 2 would be ideal, as it does not introduce any dependency for
> > > > > using the native backend.
> > > > > Yet I'm not sure  how complex implementing the tf model reader can
> be,
> > > > > If I remember correctly the student said it was not trivial at the
> > > > > time.
> > > >
> > > > yes, it is not easy, but I think it is worthy to do it. Here is a reference
> > example
> > > > for the complexity, see
> > > >
> >
> https://github.com/opencv/opencv/blob/master/modules/dnn/src/tensorflow/
> > > > tf_importer.cpp.
> > > >
> > > > >
> > > > > Is the tf model file stable? if not it will be a maintenance burden to
> > > > > keep it working whenever tf releases a new version. This point makes
> > > > > me think having control over our file format is good.
> > > >
> > > > imho, this issue is always there, no matter which method used, unless our
> > > > format could be exported by tensorflow (it has little possibility).
> > > >
> > > > Whenever tf releases a new version with a new file format, we still have
> to
> > > > change the python script in phase 1 (convert tf file model to our format)
> > which
> > > > is even an external dependency at
> > > > https://github.com/HighVoltageRocknRoll/sr,
> > > >
> > > > As from effort perspective, the current implementation is better since
> > python
> > > > script is simpler. But I think we are still worth implementing option 2 as
> the
> > > > ideal technical direction.
> > >
> > > I checked a bit more about https://github.com/HighVoltageRocknRoll/sr, it
> is
> > actually
> > > not an converter (from tf model to native model), but hard code for given
> > models.
> > > And the native model is not exactly the same as tf model, it even changes
> the
> > behavior
> > > of pad parameter of conv layer.
> > >
> > > If community is open to option 2, I'll try it.
> > >
> > Option 2 is fine for me.
> 
> that's great, :)

looks that option 2 is a bit complex, TF model file is in protocol buffers (protobuf) format and not easy to parse it with simple c code.

Since there is no official c support for protobuf, let's first image how the work can be done via official c++ support.

1. get protobuf compiler protoc, .h header files and .so library files (download or build from https://github.com/protocolbuffers/protobuf/tree/master/src).
2. get tensorflow model's .proto files from https://github.com/tensorflow/tensorflow/tree/master/tensorflow/core/framework.
3. generate .cc/.h files from .proto files (see step 2) via protoc (see step 1).
4. let the generated .cc/.h files be part of ffmpeg source tree, and build with protobuf header/library files.
5. at run time, the protobuf libraries are invoked. It means that the system should have installed protobuf dev package.

furthermore, there is a compatible problem between the protobuf compiler, header files and library files. 
So, as a practice to fix it, the method is to make the protobuf source code be part of ffmpeg source tree. (it is a common practice, so we can many other projects contain the protobuf source code).

I guess the above method is not acceptable in ffmpeg. I would be glad to continue if the community embrace this change. :)

While the current implementation has external dependency, my new suggestion is:
-  add a python script under .../libavfilter/dnn/  (all other dnn source files will be also moved here later), so ffmpeg has the full control on it.
-  it is a script to convert tensorflow model file into native model file. (other formats such as caffe, torch can also be supported later if needed)

thanks.