[FFmpeg-user] Uploading output of ffmpeg to AWS S3

[David Power]

> On Thu, Jun 07, 2018 at 17:39:06 +0530, tarun singhal wrote:
> >* But this command is not working.
*>
> So what is it doing? Are there any error messages?
>
> Without knowing aws, let me guess:
> >* ffmpeg -i <M3u8 Stream> -an -r 2 -t 60 -f image2 -vcodec mjpeg
*> >* pipe:a_%0d.jpg | aws s3 cp - s3://ffmpegbucket-1/
*>
> The documentation for the "cp" command says: "Copies a local file or S3
> object to another location locally or in S3." From those words, I would
> guess it cannot handle stdin ("-"). And even if it did, you didn't name
> a target file.
>
> You do realize that your ffmpeg command outputs a stream of JPEG
> images, and not a set of files? "pipe:a_%0d.jpg" most likely will not
> nearly do what you expect it to.
>
> >* The objective is read a live stream keep generating frames and as soon I
*> >* get 1st frame transfer it to AWS S3 bucket. So this will be continous
*> >* upload process
*>
> You can probably do this from outside the ffmpeg command, by letting
> ffmpeg create files, and checking the directory for the resulting files
> before copying.

Hi! I know this is an old thread however I'm curious to understand if
any ffmpeg users are still interested in this capability. I'm testing
a tool we created for the genomics space to understand how broadly
usable it may be. Its linux only at the moment and uses ld_preload to
allow ffmpeg (or any tool) write to or read from any s3 storage
environment (aws in the cloud or ceph on prem). I'd like to understand
if this works with more complex ffmpeg usage examples as well. Any
willing tyre kickers welcome. Works for input and output files to
directly stream to / read from object stores.


[Example just cover input files / works for output fine also]

[root at deploy-ext ~]# LD_PRELOAD=/usr/lib/cuno.so bash
[root at deploy-ext ~]# ls s3://bucket1/
file_example_WAV_2MG.mp3  file_example_WAV_2MG.wav  file_output2.mp3

[root at deploy-ext ~]# ffmpeg -i s3://bucket1/file_example_WAV_2MG.wav
./output-dp.mp3
ffmpeg version 4.2.4 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 8 (GCC)
  configuration: --prefix=/usr --bindir=/usr/bin
--datadir=/usr/share/ffmpeg --docdir=/usr/share/doc/ffmpeg
--incdir=/usr/include/ffmpeg --libdir=/usr/lib64
--mandir=/usr/share/man --arch=x86_64 --optflags='-O2 -g -pipe -Wall
-Werror=format-security -Wp,-D_FORTIFY_SOURCE=2
-Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong
-grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1
-specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic
-fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection'
--extra-ldflags='-Wl,-z,relro -Wl,-z,now
-specs=/usr/lib/rpm/redhat/redhat-hardened-ld ' --extra-cflags=' '
--enable-libopencore-amrnb --enable-libopencore-amrwb
--enable-libvo-amrwbenc --enable-version3 --enable-bzlib
--disable-crystalhd --enable-fontconfig --enable-frei0r
--enable-gcrypt --enable-gnutls --enable-ladspa --enable-libaom
--enable-libdav1d --enable-libass --enable-libbluray --enable-libcdio
--enable-libdrm --enable-libjack --enable-libfreetype
--enable-libfribidi --enable-libgsm --enable-libmp3lame --enable-nvenc
--enable-openal --enable-opencl --enable-opengl --enable-libopenjpeg
--enable-libopus --enable-libpulse --enable-librsvg --enable-libsrt
--enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora
--enable-libvorbis --enable-libv4l2 --enable-libvidstab
--enable-libvmaf --enable-version3 --enable-vapoursynth
--enable-libvpx --enable-libx264 --enable-libx265 --enable-libxvid
--enable-libzimg --enable-libzvbi --enable-avfilter
--enable-avresample --enable-libmodplug --enable-postproc
--enable-pthreads --disable-static --enable-shared --enable-gpl
--disable-debug --disable-stripping --shlibdir=/usr/lib64
--enable-libmfx --enable-runtime-cpudetect
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
[wav @ 0x5614e297a000] Discarding ID3 tags because more suitable tags
were found.
Guessed Channel Layout for Input Stream #0.0 : stereo
Input #0, wav, from '/s3:/bucket1/file_example_WAV_2MG.wav':
  Metadata:
    title           : Impact Moderato
    album           : YouTube Audio Library
    artist          : Kevin MacLeod
    genre           : Cinematic
  Duration: 00:00:33.53, bitrate: 512 kb/s
    Stream #0:0: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 16000 Hz,
stereo, s16, 512 kb/s
File './output-dp.mp3' already exists. Overwrite ? [y/N] y
Stream mapping:
  Stream #0:0 -> #0:0 (pcm_s16le (native) -> mp3 (libmp3lame))
Press [q] to stop, [?] for help
Output #0, mp3, to './output-dp.mp3':
  Metadata:
    TIT2            : Impact Moderato
    TALB            : YouTube Audio Library
    TPE1            : Kevin MacLeod
    TCON            : Cinematic
    TSSE            : Lavf58.29.100
    Stream #0:0: Audio: mp3 (libmp3lame), 16000 Hz, stereo, s16p
    Metadata:
      encoder         : Lavc58.54.100 libmp3lame
size=     197kB time=00:00:33.55 bitrate=  48.2kbits/s speed=11.8x
video:0kB audio:197kB subtitle:0kB other streams:0kB global
headers:0kB muxing overhead: 0.164069%


Thanks

Dave