[FFmpeg-devel] GSoC mentored project: derain filter

[Guo, Yejun]

> -----Original Message-----
> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces at ffmpeg.org] On Behalf
> Of Liu Steven
> Sent: Wednesday, February 20, 2019 7:18 PM
> To: FFmpeg development discussions and patches <ffmpeg-
> devel at ffmpeg.org>
> Cc: Thilo Borgmann <thilo.borgmann at mail.de>; Liu Steven
> <lq at chinaffmpeg.org>
> Subject: Re: [FFmpeg-devel] GSoC mentored project: derain filter
> 
> 
> 
> > 在 2019年2月20日，下午6:35，孟学苇 <xwmeng at pku.edu.cn> 写道：
> >
> > Hi Dev-Community,
> >
> >
> >
> >
> > I am Iris Meng from China. I’m a PhD student in Institute of Digital Media,
> Peking University. I wish to contribute as a GSoC applicant this year.
> >
> > I am interested in Deep Learning. I want to add a derain filter in ffmpeg. If
> you have any suggestion or question, we can contact by email. My
> motivation and plans are as follows.
> >
> >
> >
> >
> >       Motivation
> >
> > Rain and fog are very common weather in actual life. However, it can affect
> the visibility. Especially in heavy rain, rain streaks from various directions
> accumulate and make the background scene misty, which will seriously
> influence the accuracy of many computer vision systems, including video
> surveillance, object detection and tracking in autonomous driving, etc.
> Therefore, it is an important task to remove the rain and fog, and recover the
> background from rain images. It can be used for image and video processing
> to make them clearer and it can be a preprocessing method for many
> computer vision systems.
> >
> >
> >
> >
> >       Proposed Idea
> >
> > We propose to implement this technology in ffmpeg. For video [1][2], we
> can utilize the relationship between frames to remove rain and fog. For
> single image [3], we can use traditional methods, such as discriminative
> sparse coding, low rank representation and the Gaussian mixture model. We
> can also use some deep learning methods. We should investigate these
> methods, and ultimately consider the effect of rain/fog removal and the
> complexity of the algorithm, and choose the optimal scheme.
> >
> >
> >
> >
> >       Practical application
> >
> > The derain and dehaze method can improve the subjective quality of
> videos and images.
> >
> >
> >
> >
> >      Development plan
> >
> > I would like to start working on my qualification task and try to solve my
> problems. Overall, I will follow the following steps to complete the project.
> >
> > (1)    Literature and algorithm investigation
> >
> > (2)    Data sets preparation
> >
> > (3)    Coding: Implement network, training code, inference code and so on
> >
> > (4)    Select the best method and transplantation it into ffmpeg
> >
> >
> >
> >
> >      Reference
> >
> >      [1] Zhang X, Li H, Qi Y, et al. Rain removal in video by combining temporal
> and chromatic properties[C]//2006 IEEE International Conference on
> Multimedia and Expo. IEEE, 2006: 461-464.
> >
> >      [2] Tripathi A K, Mukhopadhyay S. Removal of rain from videos: a
> review[J]. Signal, Image and Video Processing, 2014, 8(8): 1421-1430.
> >
> >      [3] Li X, Wu J, Lin Z, et al. Recurrent squeeze-and-excitation context
> aggregation net for single image deraining[C]//Proceedings of the European
> Conference on Computer Vision (ECCV). 2018: 254-269.
> >
> >
> >
> I think this can reference libavflter/sr.c to implementation, maybe you can
> try two ways to implement it, one is native and the other is model.
> 

and currently, only TensorFlow model is supported via tensorflow C API, 
you can easily save the model file in python with function tf.graph_util.convert_variables_to_constants and tf.train.write_graph.

For the native mode (executed with CPU), two operations (CONV and DEPTH_TO_SPACE) are supported now, you might add more.

> 
> Thanks
> Steven
> 
> >
> >
> >
> >
> > Thanks,
> >
> > Regards,
> >
> > Iris Meng
> > _______________________________________________
> > ffmpeg-devel mailing list
> > ffmpeg-devel at ffmpeg.org
> > http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> 
> 
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel