[FFmpeg-devel] [PATCH V7 2/3] lavfi/dnn: Modified DNN native backend related tools and docs.
Ting Fu
ting.fu at intel.com
Thu Apr 27 12:43:45 EEST 2023
Will remove native backend, so change the default backend in filters,
and also remove the python scripts which generate native model file.
Signed-off-by: Ting Fu <ting.fu at intel.com>
---
doc/filters.texi | 39 +-
libavfilter/vf_derain.c | 2 +-
libavfilter/vf_dnn_processing.c | 2 +-
libavfilter/vf_sr.c | 2 +-
tools/python/convert.py | 56 ---
tools/python/convert_from_tensorflow.py | 607 ------------------------
tools/python/convert_header.py | 26 -
7 files changed, 7 insertions(+), 727 deletions(-)
delete mode 100644 tools/python/convert.py
delete mode 100644 tools/python/convert_from_tensorflow.py
delete mode 100644 tools/python/convert_header.py
diff --git a/doc/filters.texi b/doc/filters.texi
index 5dde79919a..f1f87a24fd 100644
--- a/doc/filters.texi
+++ b/doc/filters.texi
@@ -11338,9 +11338,6 @@ See @url{http://openaccess.thecvf.com/content_ECCV_2018/papers/Xia_Li_Recurrent_
Training as well as model generation scripts are provided in
the repository at @url{https://github.com/XueweiMeng/derain_filter.git}.
-Native model files (.model) can be generated from TensorFlow model
-files (.pb) by using tools/python/convert.py
-
The filter accepts the following options:
@table @option
@@ -11361,21 +11358,16 @@ Specify which DNN backend to use for model loading and execution. This option ac
the following values:
@table @samp
- at item native
-Native implementation of DNN loading and execution.
-
@item tensorflow
TensorFlow backend. To enable this backend you
need to install the TensorFlow for C library (see
@url{https://www.tensorflow.org/install/lang_c}) and configure FFmpeg with
@code{--enable-libtensorflow}
@end table
-Default value is @samp{native}.
@item model
Set path to model file specifying network architecture and its parameters.
-Note that different backends use different file formats. TensorFlow and native
-backend can load files for only its format.
+Note that different backends use different file formats. TensorFlow can load files for only its format.
@end table
To get full functionality (such as async execution), please use the @ref{dnn_processing} filter.
@@ -11699,9 +11691,6 @@ Specify which DNN backend to use for model loading and execution. This option ac
the following values:
@table @samp
- at item native
-Native implementation of DNN loading and execution.
-
@item tensorflow
TensorFlow backend. To enable this backend you
need to install the TensorFlow for C library (see
@@ -11717,14 +11706,9 @@ be needed if the header files and libraries are not installed into system path)
@end table
-Default value is @samp{native}.
-
@item model
Set path to model file specifying network architecture and its parameters.
-Note that different backends use different file formats. TensorFlow, OpenVINO and native
-backend can load files for only its format.
-
-Native model file (.model) can be generated from TensorFlow model file (.pb) by using tools/python/convert.py
+Note that different backends use different file formats. TensorFlow, OpenVINO backend can load files for only its format.
@item input
Set the input name of the dnn network.
@@ -11750,12 +11734,6 @@ Remove rain in rgb24 frame with can.pb (see @ref{derain} filter):
./ffmpeg -i rain.jpg -vf format=rgb24,dnn_processing=dnn_backend=tensorflow:model=can.pb:input=x:output=y derain.jpg
@end example
- at item
-Halve the pixel value of the frame with format gray32f:
- at example
-ffmpeg -i input.jpg -vf format=grayf32,dnn_processing=model=halve_gray_float.model:input=dnn_in:output=dnn_out:dnn_backend=native -y out.native.png
- at end example
-
@item
Handle the Y channel with srcnn.pb (see @ref{sr} filter) for frame with yuv420p (planar YUV formats supported):
@example
@@ -21813,9 +21791,6 @@ Training scripts as well as scripts for model file (.pb) saving can be found at
@url{https://github.com/XueweiMeng/sr/tree/sr_dnn_native}. Original repository
is at @url{https://github.com/HighVoltageRocknRoll/sr.git}.
-Native model files (.model) can be generated from TensorFlow model
-files (.pb) by using tools/python/convert.py
-
The filter accepts the following options:
@table @option
@@ -21824,9 +21799,6 @@ Specify which DNN backend to use for model loading and execution. This option ac
the following values:
@table @samp
- at item native
-Native implementation of DNN loading and execution.
-
@item tensorflow
TensorFlow backend. To enable this backend you
need to install the TensorFlow for C library (see
@@ -21834,13 +21806,10 @@ need to install the TensorFlow for C library (see
@code{--enable-libtensorflow}
@end table
-Default value is @samp{native}.
-
@item model
Set path to model file specifying network architecture and its parameters.
-Note that different backends use different file formats. TensorFlow backend
-can load files for both formats, while native backend can load files for only
-its format.
+Note that different backends use different file formats. TensorFlow, OpenVINO backend
+can load files for only its format.
@item scale_factor
Set scale factor for SRCNN model. Allowed values are @code{2}, @code{3} and @code{4}.
diff --git a/libavfilter/vf_derain.c b/libavfilter/vf_derain.c
index 86e9eb8752..7e84cd65a3 100644
--- a/libavfilter/vf_derain.c
+++ b/libavfilter/vf_derain.c
@@ -43,7 +43,7 @@ static const AVOption derain_options[] = {
{ "filter_type", "filter type(derain/dehaze)", OFFSET(filter_type), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, 1, FLAGS, "type" },
{ "derain", "derain filter flag", 0, AV_OPT_TYPE_CONST, { .i64 = 0 }, 0, 0, FLAGS, "type" },
{ "dehaze", "dehaze filter flag", 0, AV_OPT_TYPE_CONST, { .i64 = 1 }, 0, 0, FLAGS, "type" },
- { "dnn_backend", "DNN backend", OFFSET(dnnctx.backend_type), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, 1, FLAGS, "backend" },
+ { "dnn_backend", "DNN backend", OFFSET(dnnctx.backend_type), AV_OPT_TYPE_INT, { .i64 = 1 }, 0, 1, FLAGS, "backend" },
{ "native", "native backend flag", 0, AV_OPT_TYPE_CONST, { .i64 = 0 }, 0, 0, FLAGS, "backend" },
#if (CONFIG_LIBTENSORFLOW == 1)
{ "tensorflow", "tensorflow backend flag", 0, AV_OPT_TYPE_CONST, { .i64 = 1 }, 0, 0, FLAGS, "backend" },
diff --git a/libavfilter/vf_dnn_processing.c b/libavfilter/vf_dnn_processing.c
index 4462915073..968df666fc 100644
--- a/libavfilter/vf_dnn_processing.c
+++ b/libavfilter/vf_dnn_processing.c
@@ -45,7 +45,7 @@ typedef struct DnnProcessingContext {
#define OFFSET(x) offsetof(DnnProcessingContext, dnnctx.x)
#define FLAGS AV_OPT_FLAG_FILTERING_PARAM | AV_OPT_FLAG_VIDEO_PARAM
static const AVOption dnn_processing_options[] = {
- { "dnn_backend", "DNN backend", OFFSET(backend_type), AV_OPT_TYPE_INT, { .i64 = 0 }, INT_MIN, INT_MAX, FLAGS, "backend" },
+ { "dnn_backend", "DNN backend", OFFSET(backend_type), AV_OPT_TYPE_INT, { .i64 = 1 }, INT_MIN, INT_MAX, FLAGS, "backend" },
{ "native", "native backend flag", 0, AV_OPT_TYPE_CONST, { .i64 = 0 }, 0, 0, FLAGS, "backend" },
#if (CONFIG_LIBTENSORFLOW == 1)
{ "tensorflow", "tensorflow backend flag", 0, AV_OPT_TYPE_CONST, { .i64 = 1 }, 0, 0, FLAGS, "backend" },
diff --git a/libavfilter/vf_sr.c b/libavfilter/vf_sr.c
index cb24c096ce..e9fe746bae 100644
--- a/libavfilter/vf_sr.c
+++ b/libavfilter/vf_sr.c
@@ -46,7 +46,7 @@ typedef struct SRContext {
#define OFFSET(x) offsetof(SRContext, x)
#define FLAGS AV_OPT_FLAG_FILTERING_PARAM | AV_OPT_FLAG_VIDEO_PARAM
static const AVOption sr_options[] = {
- { "dnn_backend", "DNN backend used for model execution", OFFSET(dnnctx.backend_type), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, 1, FLAGS, "backend" },
+ { "dnn_backend", "DNN backend used for model execution", OFFSET(dnnctx.backend_type), AV_OPT_TYPE_INT, { .i64 = 1 }, 0, 1, FLAGS, "backend" },
{ "native", "native backend flag", 0, AV_OPT_TYPE_CONST, { .i64 = 0 }, 0, 0, FLAGS, "backend" },
#if (CONFIG_LIBTENSORFLOW == 1)
{ "tensorflow", "tensorflow backend flag", 0, AV_OPT_TYPE_CONST, { .i64 = 1 }, 0, 0, FLAGS, "backend" },
diff --git a/tools/python/convert.py b/tools/python/convert.py
deleted file mode 100644
index 64cf76b2d8..0000000000
--- a/tools/python/convert.py
+++ /dev/null
@@ -1,56 +0,0 @@
-# Copyright (c) 2019 Guo Yejun
-#
-# This file is part of FFmpeg.
-#
-# FFmpeg is free software; you can redistribute it and/or
-# modify it under the terms of the GNU Lesser General Public
-# License as published by the Free Software Foundation; either
-# version 2.1 of the License, or (at your option) any later version.
-#
-# FFmpeg is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
-# Lesser General Public License for more details.
-#
-# You should have received a copy of the GNU Lesser General Public
-# License along with FFmpeg; if not, write to the Free Software
-# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
-# ==============================================================================
-
-# verified with Python 3.5.2 on Ubuntu 16.04
-import argparse
-import os
-from convert_from_tensorflow import *
-
-def get_arguments():
- parser = argparse.ArgumentParser(description='generate native mode model with weights from deep learning model')
- parser.add_argument('--outdir', type=str, default='./', help='where to put generated files')
- parser.add_argument('--infmt', type=str, default='tensorflow', help='format of the deep learning model')
- parser.add_argument('infile', help='path to the deep learning model with weights')
- parser.add_argument('--dump4tb', type=str, default='no', help='dump file for visualization in tensorboard')
-
- return parser.parse_args()
-
-def main():
- args = get_arguments()
-
- if not os.path.isfile(args.infile):
- print('the specified input file %s does not exist' % args.infile)
- exit(1)
-
- if not os.path.exists(args.outdir):
- print('create output directory %s' % args.outdir)
- os.mkdir(args.outdir)
-
- basefile = os.path.split(args.infile)[1]
- basefile = os.path.splitext(basefile)[0]
- outfile = os.path.join(args.outdir, basefile) + '.model'
- dump4tb = False
- if args.dump4tb.lower() in ('yes', 'true', 't', 'y', '1'):
- dump4tb = True
-
- if args.infmt == 'tensorflow':
- convert_from_tensorflow(args.infile, outfile, dump4tb)
-
-if __name__ == '__main__':
- main()
diff --git a/tools/python/convert_from_tensorflow.py b/tools/python/convert_from_tensorflow.py
deleted file mode 100644
index 38e64c1c94..0000000000
--- a/tools/python/convert_from_tensorflow.py
+++ /dev/null
@@ -1,607 +0,0 @@
-# Copyright (c) 2019 Guo Yejun
-#
-# This file is part of FFmpeg.
-#
-# FFmpeg is free software; you can redistribute it and/or
-# modify it under the terms of the GNU Lesser General Public
-# License as published by the Free Software Foundation; either
-# version 2.1 of the License, or (at your option) any later version.
-#
-# FFmpeg is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
-# Lesser General Public License for more details.
-#
-# You should have received a copy of the GNU Lesser General Public
-# License along with FFmpeg; if not, write to the Free Software
-# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
-# ==============================================================================
-
-import tensorflow as tf
-import numpy as np
-import sys, struct
-import convert_header as header
-
-__all__ = ['convert_from_tensorflow']
-
-class Operand(object):
- IOTYPE_INPUT = 1
- IOTYPE_OUTPUT = 2
- IOTYPE_INTERMEDIATE = IOTYPE_INPUT | IOTYPE_OUTPUT
- DTYPE_FLOAT = 1
- DTYPE_UINT8 = 4
- index = 0
- def __init__(self, name, dtype, dims):
- self.name = name
- self.dtype = dtype
- self.dims = dims
- self.iotype = 0
- self.used_count = 0
- self.index = Operand.index
- Operand.index = Operand.index + 1
- self.iotype2str = {Operand.IOTYPE_INPUT: 'in', Operand.IOTYPE_OUTPUT: 'out', Operand.IOTYPE_INTERMEDIATE: 'inout'}
- self.dtype2str = {Operand.DTYPE_FLOAT: 'DT_FLOAT', Operand.DTYPE_UINT8: 'DT_UINT8'}
-
- def add_iotype(self, iotype):
- self.iotype = self.iotype | iotype
- if iotype == Operand.IOTYPE_INPUT:
- self.used_count = self.used_count + 1
-
- def __str__(self):
- return "{}: (name: {}, iotype: {}, dtype: {}, dims: {}, used_count: {})".format(self.index,
- self.name, self.iotype2str[self.iotype], self.dtype2str[self.dtype],
- self.dims, self.used_count)
-
- def __lt__(self, other):
- return self.index < other.index
-
-class TFConverter:
- def __init__(self, graph_def, nodes, outfile, dump4tb):
- self.graph_def = graph_def
- self.nodes = nodes
- self.outfile = outfile
- self.dump4tb = dump4tb
- self.layer_number = 0
- self.output_names = []
- self.name_node_dict = {}
- self.edges = {}
- self.conv_activations = {'Relu':0, 'Tanh':1, 'Sigmoid':2, 'None':3, 'LeakyRelu':4}
- self.conv_paddings = {'VALID':0, 'SAME':1}
- self.pool_paddings = {'VALID':0, 'SAME':1}
- self.converted_nodes = set()
- self.conv2d_scope_names = set()
- self.conv2d_scopename_inputname_dict = {}
- self.dense_scope_names = set()
- self.dense_scopename_inputname_dict = {}
- self.op2code = {'Conv2D':1, 'DepthToSpace':2, 'MirrorPad':3, 'Maximum':4,
- 'MathBinary':5, 'MathUnary':6, 'AvgPool':7, 'MatMul':8}
- self.mathbin2code = {'Sub':0, 'Add':1, 'Mul':2, 'RealDiv':3, 'Minimum':4, 'FloorMod':5}
- self.mathun2code = {'Abs':0, 'Sin':1, 'Cos':2, 'Tan':3, 'Asin':4,
- 'Acos':5, 'Atan':6, 'Sinh':7, 'Cosh':8, 'Tanh':9, 'Asinh':10,
- 'Acosh':11, 'Atanh':12, 'Ceil':13, 'Floor':14, 'Round':15,
- 'Exp':16}
- self.mirrorpad_mode = {'CONSTANT':0, 'REFLECT':1, 'SYMMETRIC':2}
- self.name_operand_dict = {}
-
-
- def add_operand(self, name, type):
- node = self.name_node_dict[name]
- if name not in self.name_operand_dict:
- dtype = node.attr['dtype'].type
- if dtype == 0:
- dtype = node.attr['T'].type
- dims = [-1,-1,-1,-1]
- if 'shape' in node.attr:
- dims[0] = node.attr['shape'].shape.dim[0].size
- dims[1] = node.attr['shape'].shape.dim[1].size
- dims[2] = node.attr['shape'].shape.dim[2].size
- dims[3] = node.attr['shape'].shape.dim[3].size
- operand = Operand(name, dtype, dims)
- self.name_operand_dict[name] = operand;
- self.name_operand_dict[name].add_iotype(type)
- return self.name_operand_dict[name].index
-
-
- def dump_for_tensorboard(self):
- graph = tf.get_default_graph()
- tf.import_graph_def(self.graph_def, name="")
- tf.summary.FileWriter('/tmp/graph', graph)
- print('graph saved, run "tensorboard --logdir=/tmp/graph" to see it')
-
-
- def get_conv2d_params(self, conv2d_scope_name):
- knode = self.name_node_dict[conv2d_scope_name + '/kernel']
- bnode = self.name_node_dict[conv2d_scope_name + '/bias']
-
- if conv2d_scope_name + '/dilation_rate' in self.name_node_dict:
- dnode = self.name_node_dict[conv2d_scope_name + '/dilation_rate']
- else:
- dnode = None
-
- # the BiasAdd name is possible be changed into the output name,
- # if activation is None, and BiasAdd.next is the last op which is Identity
- if conv2d_scope_name + '/BiasAdd' in self.edges:
- anode = self.edges[conv2d_scope_name + '/BiasAdd'][0]
- if anode.op not in self.conv_activations:
- anode = None
- else:
- anode = None
- return knode, bnode, dnode, anode
-
-
- def get_dense_params(self, dense_scope_name):
- knode = self.name_node_dict[dense_scope_name + '/kernel']
- bnode = self.name_node_dict.get(dense_scope_name + '/bias')
- # the BiasAdd name is possible be changed into the output name,
- # if activation is None, and BiasAdd.next is the last op which is Identity
- anode = None
- if bnode:
- if dense_scope_name + '/BiasAdd' in self.edges:
- anode = self.edges[dense_scope_name + '/BiasAdd'][0]
- if anode.op not in self.conv_activations:
- anode = None
- else:
- anode = None
- return knode, bnode, anode
-
-
- def dump_complex_conv2d_to_file(self, node, f):
- assert(node.op == 'Conv2D')
- self.layer_number = self.layer_number + 1
- self.converted_nodes.add(node.name)
-
- scope_name = TFConverter.get_scope_name(node.name)
- #knode for kernel, bnode for bias, dnode for dilation, anode for activation
- knode, bnode, dnode, anode = self.get_conv2d_params(scope_name)
-
- if dnode is not None:
- dilation = struct.unpack('i', dnode.attr['value'].tensor.tensor_content[0:4])[0]
- else:
- dilation = 1
-
- if anode is not None:
- activation = anode.op
- else:
- activation = 'None'
-
- padding = node.attr['padding'].s.decode("utf-8")
- # conv2d with dilation > 1 generates tens of nodes, not easy to parse them, so use this tricky method.
- if dilation > 1 and scope_name + '/stack' in self.name_node_dict:
- if self.name_node_dict[scope_name + '/stack'].op == "Const":
- padding = 'SAME'
- padding = self.conv_paddings[padding]
-
- ktensor = knode.attr['value'].tensor
- filter_height = ktensor.tensor_shape.dim[0].size
- filter_width = ktensor.tensor_shape.dim[1].size
- in_channels = ktensor.tensor_shape.dim[2].size
- out_channels = ktensor.tensor_shape.dim[3].size
- kernel = np.frombuffer(ktensor.tensor_content, dtype=np.float32)
- kernel = kernel.reshape(filter_height, filter_width, in_channels, out_channels)
- kernel = np.transpose(kernel, [3, 0, 1, 2])
-
- has_bias = 1
- np.array([self.op2code[node.op], dilation, padding, self.conv_activations[activation], in_channels, out_channels, filter_height, has_bias], dtype=np.uint32).tofile(f)
- kernel.tofile(f)
-
- btensor = bnode.attr['value'].tensor
- if btensor.tensor_shape.dim[0].size == 1:
- bias = struct.pack("f", btensor.float_val[0])
- else:
- bias = btensor.tensor_content
- f.write(bias)
-
- input_name = self.conv2d_scopename_inputname_dict[scope_name]
- input_operand_index = self.add_operand(input_name, Operand.IOTYPE_INPUT)
-
- if anode is not None:
- output_operand_index = self.add_operand(anode.name, Operand.IOTYPE_OUTPUT)
- else:
- output_operand_index = self.add_operand(self.edges[bnode.name][0].name, Operand.IOTYPE_OUTPUT)
- np.array([input_operand_index, output_operand_index], dtype=np.uint32).tofile(f)
-
- def dump_dense_to_file(self, node, f):
- assert(node.op == 'MatMul')
- self.layer_number = self.layer_number + 1
- self.converted_nodes.add(node.name)
-
- scope_name = TFConverter.get_scope_name(node.name)
- #knode for kernel, bnode for bias, anode for activation
- knode, bnode, anode = self.get_dense_params(scope_name.split('/')[0])
-
- if bnode is not None:
- has_bias = 1
- btensor = bnode.attr['value'].tensor
- if btensor.tensor_shape.dim[0].size == 1:
- bias = struct.pack("f", btensor.float_val[0])
- else:
- bias = btensor.tensor_content
- else:
- has_bias = 0
-
- if anode is not None:
- activation = anode.op
- else:
- activation = 'None'
-
- ktensor = knode.attr['value'].tensor
- in_channels = ktensor.tensor_shape.dim[0].size
- out_channels = ktensor.tensor_shape.dim[1].size
- if in_channels * out_channels == 1:
- kernel = np.float32(ktensor.float_val[0])
- else:
- kernel = np.frombuffer(ktensor.tensor_content, dtype=np.float32)
- kernel = kernel.reshape(in_channels, out_channels)
- kernel = np.transpose(kernel, [1, 0])
-
- np.array([self.op2code[node.op], self.conv_activations[activation], in_channels, out_channels, has_bias], dtype=np.uint32).tofile(f)
- kernel.tofile(f)
- if has_bias:
- f.write(bias)
-
- input_name = self.dense_scopename_inputname_dict[scope_name.split('/')[0]]
- input_operand_index = self.add_operand(input_name, Operand.IOTYPE_INPUT)
-
- if anode is not None:
- output_operand_index = self.add_operand(anode.name, Operand.IOTYPE_OUTPUT)
- else:
- if bnode is not None:
- output_operand_index = self.add_operand(self.edges[bnode.name][0].name, Operand.IOTYPE_OUTPUT)
- else:
- output_operand_index = self.add_operand(self.edges[scope_name+'/concat_1'][0].name, Operand.IOTYPE_OUTPUT)
- np.array([input_operand_index, output_operand_index], dtype=np.uint32).tofile(f)
-
-
- def dump_simple_conv2d_to_file(self, node, f):
- assert(node.op == 'Conv2D')
- self.layer_number = self.layer_number + 1
- self.converted_nodes.add(node.name)
-
- node0 = self.name_node_dict[node.input[0]]
- node1 = self.name_node_dict[node.input[1]]
- if node0.op == 'Const':
- knode = node0
- input_name = node.input[1]
- else:
- knode = node1
- input_name = node.input[0]
-
- ktensor = knode.attr['value'].tensor
- filter_height = ktensor.tensor_shape.dim[0].size
- filter_width = ktensor.tensor_shape.dim[1].size
- in_channels = ktensor.tensor_shape.dim[2].size
- out_channels = ktensor.tensor_shape.dim[3].size
- if filter_height * filter_width * in_channels * out_channels == 1:
- kernel = np.float32(ktensor.float_val[0])
- else:
- kernel = np.frombuffer(ktensor.tensor_content, dtype=np.float32)
- kernel = kernel.reshape(filter_height, filter_width, in_channels, out_channels)
- kernel = np.transpose(kernel, [3, 0, 1, 2])
-
- has_bias = 0
- dilation = 1
- padding = node.attr['padding'].s.decode("utf-8")
- np.array([self.op2code[node.op], dilation, self.conv_paddings[padding], self.conv_activations['None'],
- in_channels, out_channels, filter_height, has_bias], dtype=np.uint32).tofile(f)
- kernel.tofile(f)
-
- input_operand_index = self.add_operand(input_name, Operand.IOTYPE_INPUT)
- output_operand_index = self.add_operand(node.name, Operand.IOTYPE_OUTPUT)
- np.array([input_operand_index, output_operand_index], dtype=np.uint32).tofile(f)
-
-
- def dump_depth2space_to_file(self, node, f):
- assert(node.op == 'DepthToSpace')
- self.layer_number = self.layer_number + 1
- block_size = node.attr['block_size'].i
- np.array([self.op2code[node.op], block_size], dtype=np.uint32).tofile(f)
- self.converted_nodes.add(node.name)
- input_operand_index = self.add_operand(node.input[0], Operand.IOTYPE_INPUT)
- output_operand_index = self.add_operand(node.name, Operand.IOTYPE_OUTPUT)
- np.array([input_operand_index, output_operand_index], dtype=np.uint32).tofile(f)
-
-
- def dump_mirrorpad_to_file(self, node, f):
- assert(node.op == 'MirrorPad')
- self.layer_number = self.layer_number + 1
- mode = node.attr['mode'].s
- mode = self.mirrorpad_mode[mode.decode("utf-8")]
- np.array([self.op2code[node.op], mode], dtype=np.uint32).tofile(f)
- pnode = self.name_node_dict[node.input[1]]
- self.converted_nodes.add(pnode.name)
- paddings = pnode.attr['value'].tensor.tensor_content
- f.write(paddings)
- self.converted_nodes.add(node.name)
- input_operand_index = self.add_operand(node.input[0], Operand.IOTYPE_INPUT)
- output_operand_index = self.add_operand(node.name, Operand.IOTYPE_OUTPUT)
- np.array([input_operand_index, output_operand_index], dtype=np.uint32).tofile(f)
-
-
- def dump_maximum_to_file(self, node, f):
- assert(node.op == 'Maximum')
- self.layer_number = self.layer_number + 1
- ynode = self.name_node_dict[node.input[1]]
- y = ynode.attr['value'].tensor.float_val[0]
- np.array([self.op2code[node.op]], dtype=np.uint32).tofile(f)
- np.array([y], dtype=np.float32).tofile(f)
- self.converted_nodes.add(node.name)
- input_operand_index = self.add_operand(node.input[0], Operand.IOTYPE_INPUT)
- output_operand_index = self.add_operand(node.name, Operand.IOTYPE_OUTPUT)
- np.array([input_operand_index, output_operand_index], dtype=np.uint32).tofile(f)
-
-
- def dump_mathbinary_to_file(self, node, f):
- self.layer_number = self.layer_number + 1
- self.converted_nodes.add(node.name)
- i0_node = self.name_node_dict[node.input[0]]
- i1_node = self.name_node_dict[node.input[1]]
- np.array([self.op2code['MathBinary'], self.mathbin2code[node.op]], dtype=np.uint32).tofile(f)
- if i0_node.op == 'Const':
- scalar = i0_node.attr['value'].tensor.float_val[0]
- np.array([1], dtype=np.uint32).tofile(f) # broadcast: 1
- np.array([scalar], dtype=np.float32).tofile(f)
- np.array([0], dtype=np.uint32).tofile(f) # broadcast: 0
- input_operand_index = self.add_operand(i1_node.name, Operand.IOTYPE_INPUT)
- np.array([input_operand_index], dtype=np.uint32).tofile(f)
- elif i1_node.op == 'Const':
- scalar = i1_node.attr['value'].tensor.float_val[0]
- np.array([0], dtype=np.uint32).tofile(f)
- input_operand_index = self.add_operand(i0_node.name, Operand.IOTYPE_INPUT)
- np.array([input_operand_index], dtype=np.uint32).tofile(f)
- np.array([1], dtype=np.uint32).tofile(f)
- np.array([scalar], dtype=np.float32).tofile(f)
- else:
- np.array([0], dtype=np.uint32).tofile(f)
- input_operand_index = self.add_operand(i0_node.name, Operand.IOTYPE_INPUT)
- np.array([input_operand_index], dtype=np.uint32).tofile(f)
- np.array([0], dtype=np.uint32).tofile(f)
- input_operand_index = self.add_operand(i1_node.name, Operand.IOTYPE_INPUT)
- np.array([input_operand_index], dtype=np.uint32).tofile(f)
- output_operand_index = self.add_operand(node.name, Operand.IOTYPE_OUTPUT)
- np.array([output_operand_index], dtype=np.uint32).tofile(f)
-
-
- def dump_mathunary_to_file(self, node, f):
- self.layer_number = self.layer_number + 1
- self.converted_nodes.add(node.name)
- i0_node = self.name_node_dict[node.input[0]]
- np.array([self.op2code['MathUnary'], self.mathun2code[node.op]], dtype=np.uint32).tofile(f)
- input_operand_index = self.add_operand(i0_node.name, Operand.IOTYPE_INPUT)
- np.array([input_operand_index], dtype=np.uint32).tofile(f)
- output_operand_index = self.add_operand(node.name, Operand.IOTYPE_OUTPUT)
- np.array([output_operand_index],dtype=np.uint32).tofile(f)
-
-
- def dump_avg_pool_to_file(self, node, f):
- assert(node.op == 'AvgPool')
- self.layer_number = self.layer_number + 1
- self.converted_nodes.add(node.name)
- node0 = self.name_node_dict[node.input[0]]
- strides = node.attr['strides']
-
- # Tensorflow do not support pooling strides in batch dimension and
- # current native NN do not support pooling strides in channel dimension, added assert() here.
- assert(strides.list.i[1]==strides.list.i[2])
- assert(strides.list.i[0]==1)
- assert(strides.list.i[3]==1)
- strides = strides.list.i[1]
- filter_node = node.attr['ksize']
- input_name = node.input[0]
-
- # Tensorflow do not support pooling ksize in batch dimension and channel dimension.
- assert(filter_node.list.i[0]==1)
- assert(filter_node.list.i[3]==1)
- filter_height = filter_node.list.i[1]
- filter_width = filter_node.list.i[2]
-
- padding = node.attr['padding'].s.decode("utf-8")
- np.array([self.op2code[node.op], strides, self.pool_paddings[padding], filter_height],
- dtype=np.uint32).tofile(f)
-
- input_operand_index = self.add_operand(input_name, Operand.IOTYPE_INPUT)
- output_operand_index = self.add_operand(node.name, Operand.IOTYPE_OUTPUT)
- np.array([input_operand_index, output_operand_index],dtype=np.uint32).tofile(f)
-
-
- def dump_layers_to_file(self, f):
- for node in self.nodes:
- if node.name in self.converted_nodes:
- continue
-
- # conv2d with dilation generates very complex nodes, so handle it in special
- if self.in_conv2d_scope(node.name):
- if node.op == 'Conv2D':
- self.dump_complex_conv2d_to_file(node, f)
- continue
- if self.in_dense_scope(node.name):
- if node.op == 'MatMul':
- self.dump_dense_to_file(node, f)
- continue
-
-
- if node.op == 'Conv2D':
- self.dump_simple_conv2d_to_file(node, f)
- continue
- if node.name in self.output_names:
- input_name = self.id_different_scope_dict[node.name]
- if TFConverter.get_scope_name(input_name)!=TFConverter.get_scope_name(node.name):
- continue
- if node.op == 'AvgPool':
- self.dump_avg_pool_to_file(node, f)
- elif node.op == 'DepthToSpace':
- self.dump_depth2space_to_file(node, f)
- elif node.op == 'MirrorPad':
- self.dump_mirrorpad_to_file(node, f)
- elif node.op == 'Maximum':
- self.dump_maximum_to_file(node, f)
- elif node.op in self.mathbin2code:
- self.dump_mathbinary_to_file(node, f)
- elif node.op in self.mathun2code:
- self.dump_mathunary_to_file(node, f)
-
-
- def dump_operands_to_file(self, f):
- operands = sorted(self.name_operand_dict.values())
- for operand in operands:
- #print('{}'.format(operand))
- np.array([operand.index, len(operand.name)], dtype=np.uint32).tofile(f)
- f.write(operand.name.encode('utf-8'))
- np.array([operand.iotype, operand.dtype], dtype=np.uint32).tofile(f)
- np.array(operand.dims, dtype=np.uint32).tofile(f)
-
-
- def dump_to_file(self):
- with open(self.outfile, 'wb') as f:
- f.write(header.str.encode('utf-8'))
- np.array([header.major, header.minor], dtype=np.uint32).tofile(f)
- self.dump_layers_to_file(f)
- self.dump_operands_to_file(f)
- np.array([self.layer_number, len(self.name_operand_dict)], dtype=np.uint32).tofile(f)
-
-
- def generate_name_node_dict(self):
- for node in self.nodes:
- self.name_node_dict[node.name] = node
-
-
- def generate_output_names(self):
- used_names = []
- for node in self.nodes:
- for input in node.input:
- used_names.append(input)
-
- for node in self.nodes:
- if node.name not in used_names:
- self.output_names.append(node.name)
-
-
- def remove_identity(self):
- self.id_different_scope_dict = {}
- id_nodes = []
- id_dict = {}
- for node in self.nodes:
- if node.op == 'Identity':
- name = node.name
- input = node.input[0]
- id_nodes.append(node)
- # do not change the output name
- if name in self.output_names:
- self.name_node_dict[input].name = name
- self.name_node_dict[name] = self.name_node_dict[input]
- del self.name_node_dict[input]
- self.id_different_scope_dict[name] = input
- else:
- id_dict[name] = input
-
- for idnode in id_nodes:
- self.nodes.remove(idnode)
-
- for node in self.nodes:
- for i in range(len(node.input)):
- input = node.input[i]
- if input in id_dict:
- node.input[i] = id_dict[input]
-
-
- def generate_edges(self):
- for node in self.nodes:
- for input in node.input:
- if input in self.edges:
- self.edges[input].append(node)
- else:
- self.edges[input] = [node]
-
-
- @staticmethod
- def get_scope_name(name):
- index = name.rfind('/')
- if index == -1:
- return ""
- return name[0:index]
-
-
- def in_conv2d_scope(self, name):
- inner_scope = TFConverter.get_scope_name(name)
- if inner_scope == "":
- return False;
- for scope in self.conv2d_scope_names:
- index = inner_scope.find(scope)
- if index == 0:
- return True
- return False
-
-
- def in_dense_scope(self, name):
- inner_scope = TFConverter.get_scope_name(name)
- if inner_scope == "":
- return False;
- for scope in self.dense_scope_names:
- index = inner_scope.find(scope)
- if index == 0:
- return True
- return False
-
- def generate_sub_block_op_scope_info(self):
- # mostly, conv2d/dense is a sub block in graph, get the scope name
- for node in self.nodes:
- if node.op == 'Conv2D':
- scope = TFConverter.get_scope_name(node.name)
- # for the case tf.nn.conv2d is called directly
- if scope == '':
- continue
- # for the case tf.nn.conv2d is called within a scope
- if scope + '/kernel' not in self.name_node_dict:
- continue
- self.conv2d_scope_names.add(scope)
- elif node.op == 'MatMul':
- scope = TFConverter.get_scope_name(node.name)
- # for the case tf.nn.dense is called directly
- if scope == '':
- continue
- # for the case tf.nn.dense is called within a scope
- if scope + '/kernel' not in self.name_node_dict and scope.split('/Tensordot')[0] + '/kernel' not in self.name_node_dict:
- continue
- self.dense_scope_names.add(scope.split('/Tensordot')[0])
-
- # get the input name to the conv2d/dense sub block
- for node in self.nodes:
- scope = TFConverter.get_scope_name(node.name)
- if scope in self.conv2d_scope_names:
- if node.op == 'Conv2D' or node.op == 'Shape':
- for inp in node.input:
- if TFConverter.get_scope_name(inp) != scope:
- self.conv2d_scopename_inputname_dict[scope] = inp
- elif scope in self.dense_scope_names:
- if node.op == 'MatMul' or node.op == 'Shape':
- for inp in node.input:
- if TFConverter.get_scope_name(inp) != scope:
- self.dense_scopename_inputname_dict[scope] = inp
- elif scope.split('/Tensordot')[0] in self.dense_scope_names:
- if node.op == 'Transpose':
- for inp in node.input:
- if TFConverter.get_scope_name(inp).find(scope)<0 and TFConverter.get_scope_name(inp).find(scope.split('/')[0])<0:
- self.dense_scopename_inputname_dict[scope.split('/Tensordot')[0]] = inp
-
-
- def run(self):
- self.generate_name_node_dict()
- self.generate_output_names()
- self.remove_identity()
- self.generate_edges()
- self.generate_sub_block_op_scope_info()
-
- if self.dump4tb:
- self.dump_for_tensorboard()
-
- self.dump_to_file()
-
-
-def convert_from_tensorflow(infile, outfile, dump4tb):
- with open(infile, 'rb') as f:
- # read the file in .proto format
- graph_def = tf.GraphDef()
- graph_def.ParseFromString(f.read())
- nodes = graph_def.node
-
- converter = TFConverter(graph_def, nodes, outfile, dump4tb)
- converter.run()
diff --git a/tools/python/convert_header.py b/tools/python/convert_header.py
deleted file mode 100644
index 143f92c42e..0000000000
--- a/tools/python/convert_header.py
+++ /dev/null
@@ -1,26 +0,0 @@
-# Copyright (c) 2019
-#
-# This file is part of FFmpeg.
-#
-# FFmpeg is free software; you can redistribute it and/or
-# modify it under the terms of the GNU Lesser General Public
-# License as published by the Free Software Foundation; either
-# version 2.1 of the License, or (at your option) any later version.
-#
-# FFmpeg is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
-# Lesser General Public License for more details.
-#
-# You should have received a copy of the GNU Lesser General Public
-# License along with FFmpeg; if not, write to the Free Software
-# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
-# ==============================================================================
-
-str = 'FFMPEGDNNNATIVE'
-
-# increase major and reset minor when we have to re-convert the model file
-major = 1
-
-# increase minor when we don't have to re-convert the model file
-minor = 23
--
2.17.1
More information about the ffmpeg-devel
mailing list