[FFmpeg-devel] [PATCH V2 4/4] avfilter/vf_dnn_processing: add a generic filter for image proccessing with dnn networks

[Guo, Yejun]

> -----Original Message-----
> From: Guo, Yejun
> Sent: Monday, October 21, 2019 8:39 PM
> To: ffmpeg-devel at ffmpeg.org
> Cc: Guo, Yejun <yejun.guo at intel.com>
> Subject: [PATCH V2 4/4] avfilter/vf_dnn_processing: add a generic filter for
> image proccessing with dnn networks
> 
> This filter accepts all the dnn networks which do image processing.
> Currently, frame with formats rgb24 and bgr24 are supported. Other
> formats such as gray and YUV will be supported next. The dnn network
> can accept data in float32 or uint8 format. And the dnn network can
> change frame size.
> 
> Let's take an example with the following python script. This script
> halves the value of the first channel of the pixel.
> import tensorflow as tf
> import numpy as np
> import scipy.misc
> in_img = scipy.misc.imread('in.bmp')
> in_img = in_img.astype(np.float32)/255.0
> in_data = in_img[np.newaxis, :]
> filter_data = np.array([0.5, 0, 0, 0, 1., 0, 0, 0,
> 1.]).reshape(1,1,3,3).astype(np.float32)
> filter = tf.Variable(filter_data)
> x = tf.placeholder(tf.float32, shape=[1, None, None, 3], name='dnn_in')
> y = tf.nn.conv2d(x, filter, strides=[1, 1, 1, 1], padding='VALID', name='dnn_out')
> sess=tf.Session()
> sess.run(tf.global_variables_initializer())
> output = sess.run(y, feed_dict={x: in_data})
> graph_def = tf.graph_util.convert_variables_to_constants(sess,
> sess.graph_def, ['dnn_out'])
> tf.train.write_graph(graph_def, '.', 'halve_first_channel.pb', as_text=False)
> output = output * 255.0
> output = output.astype(np.uint8)
> scipy.misc.imsave("out.bmp", np.squeeze(output))
> 
> - generate halve_first_channel.pb with the above script
> - generate halve_first_channel.model with tools/python/convert.py
> - try with following commands
>   ./ffmpeg -i input.jpg -vf
> dnn_processing=model=halve_first_channel.model:input=dnn_in:output=dnn_
> out:fmt=rgb24:dnn_backend=native -y out.native.png
>   ./ffmpeg -i input.jpg -vf
> dnn_processing=model=halve_first_channel.pb:input=dnn_in:output=dnn_out:f
> mt=rgb24:dnn_backend=tensorflow -y out.tf.png
> 
> Signed-off-by: Guo, Yejun <yejun.guo at intel.com>
> ---
>  configure                       |   1 +
>  doc/filters.texi                |  44 ++++++
>  libavfilter/Makefile            |   1 +
>  libavfilter/allfilters.c        |   1 +
>  libavfilter/vf_dnn_processing.c | 333
> ++++++++++++++++++++++++++++++++++++++++
>  5 files changed, 380 insertions(+)


this patch set asks for review, thanks.

btw, the first 3 patches is to improve dnn module base and can be reviewed first.
The fourth patch is a filter based on dnn and can be reviewed secondly.
I put them together to explain why dnn module needs such change. thanks.