[FFmpeg-user] Passing three frames at once to the VESPCN and VSRNET models

[Vítek Novotný]

Hello all,

I am the maintainer of the miratmu/ffmpeg-tensorflow Docker image, which
provides up-to-date FFMpeg with --enable-libtensorflow and includes
super-resolution models from https://github.com/HighVoltageRocknRoll/sr.

 - https://github.com/MIR-MU/ffmpeg-tensorflow
 - https://hub.docker.com/r/miratmu/ffmpeg-tensorflow

The HighVoltageRocknRoll/sr repository provides two pre-trained
super-resolution models for the simpler ESPCN and SRCNN architectures,
which superscale the Y channel of a single frame at a time. Here is how
we can superscale a video using miratmu/ffmpeg-tensorflow and ESPCN:

    $ wget https://media.xiph.org/video/derf/y4m/flower_cif.y4m

    $ alias ffmpeg-tensorflow='
    >     docker run --rm --gpus all -u $(id -u):$(id -g)
    >     -v "$PWD":/data -w /data -it miratmu/ffmpeg-tensorflow
    > '

    $ ffmpeg-tensorflow -i flower_cif.y4m -filter_complex '
    >     [0:v] format=yuv420p, extractplanes=y+u+v [y][u][v];
    >     [y] sr=tensorflow:/models/espcn.pb:2 [Y];
    >     [u] scale=iw*2:ih*2 [U]; [v] scale=iw*2:ih*2 [V];
    >     [Y][U][V] mergeplanes=0x001020:yuv420p [merged]
    > ' -map [merged] -sws_flags lanczos -y flower_cif_2x.mp4

The HighVoltageRocknRoll/sr repository provides also two pre-trained
super-resolution models for the more advanced VSRNET and VESPCN
architectures, which superscale the Y channel of three frames at a
time---the previous frame, the current frame, and the following frame:

    https://github.com/HighVoltageRocknRoll/sr/blob/master/models/model_vespcn.py#L19

Using the above code with the /models/vespcn.pb model will produce a
video, where the Y channel contains three subsampled copies of the
single input frame, see the attachment produced with -frames:v 1
flower_cif_2x.tif.

How would you the pass the Y channel of three frames at once to the
sr filter in FFMpeg, so that we can use advanced super-resolution
architectures such as VSRNET and VESPCN?

Best,
Vítek Novotný
-------------- next part --------------
A non-text attachment was scrubbed...
Name: flower_cif_2x.tif
Type: image/tiff
Size: 541373 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-user/attachments/20210815/a0f6e925/attachment.tif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-user/attachments/20210815/a0f6e925/attachment.sig>