[FFmpeg-devel] native mode in FFmpeg DNN module

[Pedro Arthur]

Em qui, 23 de mai de 2019 às 00:06, Guo, Yejun <yejun.guo at intel.com> escreveu:
>
>
>
> > > > > > > Option 2)
> > > > > > > Write c code in FFmpeg to convert tensorflow file format (format 1)
> > > directly
> > > > > > into memory representation (format 3), and so we controls everything in
> > > > > > ffmpeg community. And the conversion can be extended to import more
> > > file
> > > > > > formats such as torch, darknet, etc. One example is that OpenCV uses
> > > this
> > > > > > method.
> > > > > > >
> > > > > > > The in memory representation (format 3) can still be current.
> > > > > > >
> > > > > >
> > > > > > Option 2 would be ideal, as it does not introduce any dependency for
> > > > > > using the native backend.
> > > > > > Yet I'm not sure  how complex implementing the tf model reader can
> > be,
> > > > > > If I remember correctly the student said it was not trivial at the
> > > > > > time.
> > > > >
> > > > > yes, it is not easy, but I think it is worthy to do it. Here is a reference
> > > example
> > > > > for the complexity, see
> > > > >
> > >
> > https://github.com/opencv/opencv/blob/master/modules/dnn/src/tensorflow/
> > > > > tf_importer.cpp.
> > > > >
> > > > > >
> > > > > > Is the tf model file stable? if not it will be a maintenance burden to
> > > > > > keep it working whenever tf releases a new version. This point makes
> > > > > > me think having control over our file format is good.
> > > > >
> > > > > imho, this issue is always there, no matter which method used, unless our
> > > > > format could be exported by tensorflow (it has little possibility).
> > > > >
> > > > > Whenever tf releases a new version with a new file format, we still have
> > to
> > > > > change the python script in phase 1 (convert tf file model to our format)
> > > which
> > > > > is even an external dependency at
> > > > > https://github.com/HighVoltageRocknRoll/sr,
> > > > >
> > > > > As from effort perspective, the current implementation is better since
> > > python
> > > > > script is simpler. But I think we are still worth implementing option 2 as
> > the
> > > > > ideal technical direction.
> > > >
> > > > I checked a bit more about https://github.com/HighVoltageRocknRoll/sr, it
> > is
> > > actually
> > > > not an converter (from tf model to native model), but hard code for given
> > > models.
> > > > And the native model is not exactly the same as tf model, it even changes
> > the
> > > behavior
> > > > of pad parameter of conv layer.
> > > >
> > > > If community is open to option 2, I'll try it.
> > > >
> > > Option 2 is fine for me.
> >
> > that's great, :)
>
> looks that option 2 is a bit complex, TF model file is in protocol buffers (protobuf) format and not easy to parse it with simple c code.
>
> Since there is no official c support for protobuf, let's first image how the work can be done via official c++ support.
>
> 1. get protobuf compiler protoc, .h header files and .so library files (download or build from https://github.com/protocolbuffers/protobuf/tree/master/src).
> 2. get tensorflow model's .proto files from https://github.com/tensorflow/tensorflow/tree/master/tensorflow/core/framework.
> 3. generate .cc/.h files from .proto files (see step 2) via protoc (see step 1).
> 4. let the generated .cc/.h files be part of ffmpeg source tree, and build with protobuf header/library files.
> 5. at run time, the protobuf libraries are invoked. It means that the system should have installed protobuf dev package.
>
> furthermore, there is a compatible problem between the protobuf compiler, header files and library files.
> So, as a practice to fix it, the method is to make the protobuf source code be part of ffmpeg source tree. (it is a common practice, so we can many other projects contain the protobuf source code).
>
> I guess the above method is not acceptable in ffmpeg. I would be glad to continue if the community embrace this change. :)
Indeed I think it is not acceptable.

>
> While the current implementation has external dependency, my new suggestion is:
> -  add a python script under .../libavfilter/dnn/  (all other dnn source files will be also moved here later), so ffmpeg has the full control on it.
I'm not sure about the policy on putting secondary scripts with the
main code, but another option is to create a repo controlled by ffmpeg
maybe?
I think this option would also help GSoC students that work with dnn,
so they don't have to depend on previous students maintaining
independent repositories.

> -  it is a script to convert tensorflow model file into native model file. (other formats such as caffe, torch can also be supported later if needed)
>
> thanks.
>
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".