[FFmpeg-devel] [PATCH V3] avfilter/vf_dnn_processing: add a generic filter for image proccessing with dnn networks

[Guo, Yejun]

> -----Original Message-----
> From: Guo, Yejun
> Sent: Thursday, October 31, 2019 4:33 PM
> To: ffmpeg-devel at ffmpeg.org
> Cc: Guo, Yejun <yejun.guo at intel.com>
> Subject: [PATCH V3] avfilter/vf_dnn_processing: add a generic filter for image
> proccessing with dnn networks
> 
> This filter accepts all the dnn networks which do image processing.
> Currently, frame with formats rgb24 and bgr24 are supported. Other
> formats such as gray and YUV will be supported next. The dnn network
> can accept data in float32 or uint8 format. And the dnn network can
> change frame size.
> 
> The following is a python script to halve the value of the first
> channel of the pixel. It demos how to setup and execute dnn model
> with python+tensorflow. It also generates .pb file which will be
> used by ffmpeg.
> 
> import tensorflow as tf
> import numpy as np
> import scipy.misc
> in_img = scipy.misc.imread('in.bmp')
> in_img = in_img.astype(np.float32)/255.0
> in_data = in_img[np.newaxis, :]
> filter_data = np.array([0.5, 0, 0, 0, 1., 0, 0, 0,
> 1.]).reshape(1,1,3,3).astype(np.float32)
> filter = tf.Variable(filter_data)
> x = tf.placeholder(tf.float32, shape=[1, None, None, 3], name='dnn_in')
> y = tf.nn.conv2d(x, filter, strides=[1, 1, 1, 1], padding='VALID', name='dnn_out')
> sess=tf.Session()
> sess.run(tf.global_variables_initializer())
> output = sess.run(y, feed_dict={x: in_data})
> graph_def = tf.graph_util.convert_variables_to_constants(sess,
> sess.graph_def, ['dnn_out'])
> tf.train.write_graph(graph_def, '.', 'halve_first_channel.pb', as_text=False)
> output = output * 255.0
> output = output.astype(np.uint8)
> scipy.misc.imsave("out.bmp", np.squeeze(output))
> 
> To do the same thing with ffmpeg:
> - generate halve_first_channel.pb with the above script
> - generate halve_first_channel.model with tools/python/convert.py
> - try with following commands
>   ./ffmpeg -i input.jpg -vf
> dnn_processing=model=halve_first_channel.model:input=dnn_in:output=dnn_
> out:fmt=rgb24:dnn_backend=native -y out.native.png
>   ./ffmpeg -i input.jpg -vf
> dnn_processing=model=halve_first_channel.pb:input=dnn_in:output=dnn_out:f
> mt=rgb24:dnn_backend=tensorflow -y out.tf.png
> 
> Signed-off-by: Guo, Yejun <yejun.guo at intel.com>
> ---
>  configure                       |   1 +
>  doc/filters.texi                |  44 ++++++
>  libavfilter/Makefile            |   1 +
>  libavfilter/allfilters.c        |   1 +
>  libavfilter/vf_dnn_processing.c | 331
> ++++++++++++++++++++++++++++++++++++++++

this patch ask for review, thanks.