[FFmpeg-devel] [PATCH V3 1/3] tools/python: add script to convert TensorFlow model (.pb) to native model (.model)

Guo, Yejun yejun.guo at intel.com
Mon Jun 24 18:24:00 EEST 2019



> -----Original Message-----
> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces at ffmpeg.org] On Behalf Of
> Pedro Arthur
> Sent: Monday, June 24, 2019 11:13 PM
> To: FFmpeg development discussions and patches <ffmpeg-devel at ffmpeg.org>
> Subject: Re: [FFmpeg-devel] [PATCH V3 1/3] tools/python: add script to convert
> TensorFlow model (.pb) to native model (.model)
> 
> LGTM.
> 
> BTW I think we should have an ffmpeg controlled repo hosting the
> scripts to train the network and also some pretrained files to easy
> testing.

yes, good idea. Do you happen to know how to apply such repo? thanks.

> 
> Em qua, 19 de jun de 2019 às 21:29, Guo, Yejun <yejun.guo at intel.com>
> escreveu:
> >
> >
> >
> > > -----Original Message-----
> > > From: Guo, Yejun
> > > Sent: Thursday, June 13, 2019 1:31 PM
> > > To: ffmpeg-devel at ffmpeg.org
> > > Cc: Guo, Yejun <yejun.guo at intel.com>
> > > Subject: [PATCH V3 1/3] tools/python: add script to convert TensorFlow
> model
> > > (.pb) to native model (.model)
> > >
> > > For example, given TensorFlow model file espcn.pb,
> > > to generate native model file espcn.model, just run:
> > > python convert.py espcn.pb
> > >
> > > In current implementation, the native model file is generated for
> > > specific dnn network with hard-code python scripts maintained out of
> ffmpeg.
> > > For example, srcnn network used by vf_sr is generated with
> > >
> https://github.com/HighVoltageRocknRoll/sr/blob/master/generate_header_a
> > > nd_model.py#L85
> > >
> > > In this patch, the script is designed as a general solution which
> > > converts general TensorFlow model .pb file into .model file. The script
> > > now has some tricky to be compatible with current implemention, will
> > > be refined step by step.
> > >
> > > The script is also added into ffmpeg source tree. It is expected there
> > > will be many more patches and community needs the ownership of it.
> > >
> > > Another technical direction is to do the conversion in c/c++ code within
> > > ffmpeg source tree. While .pb file is organized with protocol buffers,
> > > it is not easy to do such work with tiny c/c++ code, see more discussion
> > > at http://ffmpeg.org/pipermail/ffmpeg-devel/2019-May/244496.html. So,
> > > choose the python script.
> > >
> > > Signed-off-by: Guo, Yejun <yejun.guo at intel.com>
> > > ---
> > >  .gitignore                              |   1 +
> > >  tools/python/convert.py                 |  52 +++++++++
> > >  tools/python/convert_from_tensorflow.py | 201
> > > ++++++++++++++++++++++++++++++++
> > >  3 files changed, 254 insertions(+)
> > >  create mode 100644 tools/python/convert.py
> > >  create mode 100644 tools/python/convert_from_tensorflow.py
> >
> > this patch set ping for review, thanks.
> >
> > >
> > > diff --git a/.gitignore b/.gitignore
> > > index 0e57cb0..2450ee8 100644
> > > --- a/.gitignore
> > > +++ b/.gitignore
> > > @@ -36,3 +36,4 @@
> > >  /lcov/
> > >  /src
> > >  /mapfile
> > > +/tools/python/__pycache__/
> > > diff --git a/tools/python/convert.py b/tools/python/convert.py
> > > new file mode 100644
> > > index 0000000..662b429
> > > --- /dev/null
> > > +++ b/tools/python/convert.py
> > > @@ -0,0 +1,52 @@
> > > +# Copyright (c) 2019 Guo Yejun
> > > +#
> > > +# This file is part of FFmpeg.
> > > +#
> > > +# FFmpeg is free software; you can redistribute it and/or
> > > +# modify it under the terms of the GNU Lesser General Public
> > > +# License as published by the Free Software Foundation; either
> > > +# version 2.1 of the License, or (at your option) any later version.
> > > +#
> > > +# FFmpeg is distributed in the hope that it will be useful,
> > > +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> > > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> GNU
> > > +# Lesser General Public License for more details.
> > > +#
> > > +# You should have received a copy of the GNU Lesser General Public
> > > +# License along with FFmpeg; if not, write to the Free Software
> > > +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301
> USA
> > > +#
> > >
> ================================================================
> > > ==============
> > > +
> > > +# verified with Python 3.5.2 on Ubuntu 16.04
> > > +import argparse
> > > +import os
> > > +from convert_from_tensorflow import *
> > > +
> > > +def get_arguments():
> > > +    parser = argparse.ArgumentParser(description='generate native
> mode
> > > model with weights from deep learning model')
> > > +    parser.add_argument('--outdir', type=str, default='./', help='where to
> put
> > > generated files')
> > > +    parser.add_argument('--infmt', type=str, default='tensorflow',
> > > help='format of the deep learning model')
> > > +    parser.add_argument('infile', help='path to the deep learning model
> with
> > > weights')
> > > +
> > > +    return parser.parse_args()
> > > +
> > > +def main():
> > > +    args = get_arguments()
> > > +
> > > +    if not os.path.isfile(args.infile):
> > > +        print('the specified input file %s does not exist' % args.infile)
> > > +        exit(1)
> > > +
> > > +    if not os.path.exists(args.outdir):
> > > +        print('create output directory %s' % args.outdir)
> > > +        os.mkdir(args.outdir)
> > > +
> > > +    basefile = os.path.split(args.infile)[1]
> > > +    basefile = os.path.splitext(basefile)[0]
> > > +    outfile = os.path.join(args.outdir, basefile) + '.model'
> > > +
> > > +    if args.infmt == 'tensorflow':
> > > +        convert_from_tensorflow(args.infile, outfile)
> > > +
> > > +if __name__ == '__main__':
> > > +    main()
> > > diff --git a/tools/python/convert_from_tensorflow.py
> > > b/tools/python/convert_from_tensorflow.py
> > > new file mode 100644
> > > index 0000000..37049e5
> > > --- /dev/null
> > > +++ b/tools/python/convert_from_tensorflow.py
> > > @@ -0,0 +1,201 @@
> > > +# Copyright (c) 2019 Guo Yejun
> > > +#
> > > +# This file is part of FFmpeg.
> > > +#
> > > +# FFmpeg is free software; you can redistribute it and/or
> > > +# modify it under the terms of the GNU Lesser General Public
> > > +# License as published by the Free Software Foundation; either
> > > +# version 2.1 of the License, or (at your option) any later version.
> > > +#
> > > +# FFmpeg is distributed in the hope that it will be useful,
> > > +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> > > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> GNU
> > > +# Lesser General Public License for more details.
> > > +#
> > > +# You should have received a copy of the GNU Lesser General Public
> > > +# License along with FFmpeg; if not, write to the Free Software
> > > +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301
> USA
> > > +#
> > >
> ================================================================
> > > ==============
> > > +
> > > +import tensorflow as tf
> > > +import numpy as np
> > > +import sys, struct
> > > +
> > > +__all__ = ['convert_from_tensorflow']
> > > +
> > > +# as the first step to be compatible with vf_sr, it is not general.
> > > +# it will be refined step by step.
> > > +
> > > +class TFConverter:
> > > +    def __init__(self, graph_def, nodes, outfile):
> > > +        self.graph_def = graph_def
> > > +        self.nodes = nodes
> > > +        self.outfile = outfile
> > > +        self.layer_number = 0
> > > +        self.output_names = []
> > > +        self.name_node_dict = {}
> > > +        self.edges = {}
> > > +        self.conv_activations = {'Relu':0, 'Tanh':1, 'Sigmoid':2,
> 'LeakyRelu':4}
> > > +        self.conv_paddings = {'VALID':2, 'SAME':1}
> > > +        self.converted_nodes = set()
> > > +        self.op2code = {'Conv2D':1, 'DepthToSpace':2}
> > > +
> > > +
> > > +    def dump_for_tensorboard(self):
> > > +        graph = tf.get_default_graph()
> > > +        tf.import_graph_def(self.graph_def, name="")
> > > +        # tensorboard --logdir=/tmp/graph
> > > +        tf.summary.FileWriter('/tmp/graph', graph)
> > > +
> > > +
> > > +    def get_conv2d_params(self, node):
> > > +        knode = self.name_node_dict[node.input[1]]
> > > +        bnode = None
> > > +        activation = 'None'
> > > +        next = self.edges[node.name][0]
> > > +        if next.op == 'BiasAdd':
> > > +            self.converted_nodes.add(next.name)
> > > +            bnode = self.name_node_dict[next.input[1]]
> > > +            next = self.edges[next.name][0]
> > > +        if next.op in self.conv_activations:
> > > +            self.converted_nodes.add(next.name)
> > > +            activation = next.op
> > > +        return knode, bnode, activation
> > > +
> > > +
> > > +    def dump_conv2d_to_file(self, node, f):
> > > +        assert(node.op == 'Conv2D')
> > > +        self.layer_number = self.layer_number + 1
> > > +        self.converted_nodes.add(node.name)
> > > +        knode, bnode, activation = self.get_conv2d_params(node)
> > > +
> > > +        dilation = node.attr['dilations'].list.i[0]
> > > +        padding = node.attr['padding'].s
> > > +        padding = self.conv_paddings[padding.decode("utf-8")]
> > > +
> > > +        ktensor = knode.attr['value'].tensor
> > > +        filter_height = ktensor.tensor_shape.dim[0].size
> > > +        filter_width = ktensor.tensor_shape.dim[1].size
> > > +        in_channels = ktensor.tensor_shape.dim[2].size
> > > +        out_channels = ktensor.tensor_shape.dim[3].size
> > > +        kernel = np.frombuffer(ktensor.tensor_content,
> dtype=np.float32)
> > > +        kernel = kernel.reshape(filter_height, filter_width, in_channels,
> > > out_channels)
> > > +        kernel = np.transpose(kernel, [3, 0, 1, 2])
> > > +
> > > +        np.array([self.op2code[node.op], dilation, padding,
> > > self.conv_activations[activation], in_channels, out_channels, filter_height],
> > > dtype=np.uint32).tofile(f)
> > > +        kernel.tofile(f)
> > > +
> > > +        btensor = bnode.attr['value'].tensor
> > > +        if btensor.tensor_shape.dim[0].size == 1:
> > > +            bias = struct.pack("f", btensor.float_val[0])
> > > +        else:
> > > +            bias = btensor.tensor_content
> > > +        f.write(bias)
> > > +
> > > +
> > > +    def dump_depth2space_to_file(self, node, f):
> > > +        assert(node.op == 'DepthToSpace')
> > > +        self.layer_number = self.layer_number + 1
> > > +        block_size = node.attr['block_size'].i
> > > +        np.array([self.op2code[node.op], block_size],
> > > dtype=np.uint32).tofile(f)
> > > +        self.converted_nodes.add(node.name)
> > > +
> > > +
> > > +    def generate_layer_number(self):
> > > +        # in current hard code implementation, the layer number is the
> first
> > > data written to the native model file
> > > +        # it is not easy to know it at the beginning time in the general
> > > converter, so first do a dry run for compatibility
> > > +        # will be refined later.
> > > +        with open('/tmp/tmp.model', 'wb') as f:
> > > +            self.dump_layers_to_file(f)
> > > +        self.converted_nodes.clear()
> > > +
> > > +
> > > +    def dump_layers_to_file(self, f):
> > > +        for node in self.nodes:
> > > +            if node.name in self.converted_nodes:
> > > +                continue
> > > +            if node.op == 'Conv2D':
> > > +                self.dump_conv2d_to_file(node, f)
> > > +            elif node.op == 'DepthToSpace':
> > > +                self.dump_depth2space_to_file(node, f)
> > > +
> > > +
> > > +    def dump_to_file(self):
> > > +        self.generate_layer_number()
> > > +        with open(self.outfile, 'wb') as f:
> > > +            np.array([self.layer_number], dtype=np.uint32).tofile(f)
> > > +            self.dump_layers_to_file(f)
> > > +
> > > +
> > > +    def generate_name_node_dict(self):
> > > +        for node in self.nodes:
> > > +            self.name_node_dict[node.name] = node
> > > +
> > > +
> > > +    def generate_output_names(self):
> > > +        used_names = []
> > > +        for node in self.nodes:
> > > +            for input in node.input:
> > > +                used_names.append(input)
> > > +
> > > +        for node in self.nodes:
> > > +            if node.name not in used_names:
> > > +                self.output_names.append(node.name)
> > > +
> > > +
> > > +    def remove_identity(self):
> > > +        id_nodes = []
> > > +        id_dict = {}
> > > +        for node in self.nodes:
> > > +            if node.op == 'Identity':
> > > +                name = node.name
> > > +                input = node.input[0]
> > > +                id_nodes.append(node)
> > > +                # do not change the output name
> > > +                if name in self.output_names:
> > > +                    self.name_node_dict[input].name = name
> > > +                    self.name_node_dict[name] =
> > > self.name_node_dict[input]
> > > +                    del self.name_node_dict[input]
> > > +                else:
> > > +                    id_dict[name] = input
> > > +
> > > +        for idnode in id_nodes:
> > > +            self.nodes.remove(idnode)
> > > +
> > > +        for node in self.nodes:
> > > +            for i in range(len(node.input)):
> > > +                input = node.input[i]
> > > +                if input in id_dict:
> > > +                    node.input[i] = id_dict[input]
> > > +
> > > +
> > > +    def generate_edges(self):
> > > +        for node in self.nodes:
> > > +            for input in node.input:
> > > +                if input in self.edges:
> > > +                    self.edges[input].append(node)
> > > +                else:
> > > +                    self.edges[input] = [node]
> > > +
> > > +
> > > +    def run(self):
> > > +        self.generate_name_node_dict()
> > > +        self.generate_output_names()
> > > +        self.remove_identity()
> > > +        self.generate_edges()
> > > +
> > > +        #check the graph with tensorboard with human eyes
> > > +        #self.dump_for_tensorboard()
> > > +
> > > +        self.dump_to_file()
> > > +
> > > +
> > > +def convert_from_tensorflow(infile, outfile):
> > > +    with open(infile, 'rb') as f:
> > > +        # read the file in .proto format
> > > +        graph_def = tf.GraphDef()
> > > +        graph_def.ParseFromString(f.read())
> > > +        nodes = graph_def.node
> > > +
> > > +    converter = TFConverter(graph_def, nodes, outfile)
> > > +    converter.run()
> > > --
> > > 2.7.4
> >
> > _______________________________________________
> > ffmpeg-devel mailing list
> > ffmpeg-devel at ffmpeg.org
> > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> >
> > To unsubscribe, visit link above, or email
> > ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".


More information about the ffmpeg-devel mailing list