[FFmpeg-user] Chunked request when using "-progress"

Thu Dec 13 18:16:38 CET 2012

I've just installed nginx 1.3.9 with chunked input support.
And still it behaves not as I am expecting.

As described in manpage:

       -progress url (global)
           Send program-friendly progress information to url.

           Progress information is written approximately every second and
at the end of the encoding process. It is made of "key=value" lines. key
consists of only alphanumeric characters. The last key of
           a sequence of progress information is always "progress".

I captured HTTP request that ffmpeg performs for progress:

POST /progress.php HTTP/1.1
Transfer-Encoding: chunked
User-Agent: Lavf54.29.104
Accept: */*
Connection: close
Host: converter.citylink-rk.ru

8f
frame=91
fps=0.0
stream_0_0_q=23.0
total_size=205437
out_time_ms=4388571
out_time=00:00:04.388571
dup_frames=1
drop_frames=0
progress=continue

92
frame=210
fps=209.5
stream_0_0_q=11.0
total_size=569264
out_time_ms=9357642
out_time=00:00:09.357642
dup_frames=1
drop_frames=0
progress=continue

92
frame=368
fps=244.8
stream_0_0_q=4.0
total_size=814739
out_time_ms=15952109
out_time=00:00:15.952109
dup_frames=1
drop_frames=0
progress=continue

94
frame=440
fps=219.0
stream_0_0_q=10.0
total_size=1921815
out_time_ms=18924263
out_time=00:00:18.924263
dup_frames=1
drop_frames=0
progress=continue

94
frame=499
fps=198.4
stream_0_0_q=15.0
total_size=2448688
out_time_ms=21408798
out_time=00:00:21.408798
dup_frames=1
drop_frames=0
progress=continue

94
frame=559
fps=185.4
stream_0_0_q=16.0
total_size=2809055
out_time_ms=23870113
out_time=00:00:23.870113
dup_frames=1
drop_frames=0
progress=continue

92
frame=594
fps=164.6
stream_0_0_q=32766.0
total_size=3312591
out_time_ms=25379410
out_time=00:00:25.379410
dup_frames=1
drop_frames=0
progress=end

0

FFmpeg sends POST-request headers at start and after that it sends a chunk
with current progress each second.
*It is not possible to use this chunks in realtime because web server waits
for full request body (terminated by last zero-length chunk),*
*so web-server passes this request after the process of convertation
(ffmpeg process) completed.*
This information is non-realtime and useless.

I expect ffmpeg making separate POST-request with current progress
information. Ex.:

POST /progress.php HTTP/1.1
User-Agent: Lavf54.29.104
Accept: */*
Host: conv.example.com
Content-length: NN

frame=91
fps=0.0
stream_0_0_q=23.0
total_size=205437
out_time_ms=4388571
out_time=00:00:04.388571
dup_frames=1
drop_frames=0
progress=continue

POST /progress.php HTTP/1.1
User-Agent: Lavf54.29.104
Accept: */*
Host: conv.example.com
Content-length: NN

frame=210
fps=209.5
stream_0_0_q=11.0
total_size=569264
out_time_ms=9357642
out_time=00:00:09.357642
dup_frames=1
drop_frames=0
progress=continue

and so on.

All these requests can be made in the sope of one keepalive connection.
If web-server does not support keepalive or closes the connection for some
reason, ffmpeg should re-establish the connection for a new request.

Requests could be identified by input name / output name / process id /
etc. and some unique key. ex:

input=/tmp/in.avi
key=ad24d5qw

This pairs must match to identify continious progress.

2012/12/13 Nicolas George <nicolas.george at normalesup.org>

> Le tridi 23 frimaire, an CCXXI, Вадим Лазовский a écrit :
> > I have encountered a problem when using new "-progress" option.
> > FFmpeg uses chunked request that is not supported by nginx web server up
> to
> > the last version 1.3.9 released on 2012-11-27 (Feature: support for
> chunked
> > transfer encoding while reading client request body).
> > All other versions of nginx are unable to handle chunked input and
> respond
> > with "411 Length Required".
> >
> > command line:
> >
> > ffmpeg -i 329635 -progress http://conv.example.com/progress.php -codec:v
> > libx264 -strict -2 -codec:a aac -ar 44100 -b:v 800k -f mp4 -y
> /tmp/123.mp4
>
> So exact problem is that you want to post the progress to an HTTP server.
> Got it.
>
> Note: you provided the exact command line, but not the corresponding full
> output. In this particular it is not necessary, but it is usually, and it
> is
> better to include it anyway.
>
> > client request headers (from tcpdump):
> >
> > POST /progress.php HTTP/1.1
> > Transfer-Encoding: chunked
> > User-Agent: Lavf54.29.104
> > Accept: */*
> > Connection: close
> > Host: conv.example.com
> >
> > server response (from tcpdump):
> >
> > HTTP/1.1 411 Length Required Server: nginx/1.2.3 Date: Thu, 13 Dec 2012
> > 10:37:09 GMT Content-Type: text/html Content-Length: 180 Connection:
> close
> > <html> <head><title>411 Length Required</title></head> <body
> > bgcolor="white"> <center><h1>411 Length Required</h1></center>
> > <hr><center>nginx/1.2.3</center> </body> </html>
> >
> > web server log entry:
> > 10.0.1.1 - - [13/Dec/2012:14:30:57 +0400] "POST /progress.php HTTP/1.1"
> 411
> > 180 "-" "Lavf54.29.104"
> >
> > Is it possible to make ffmpeg do requests in a "traditional" way: not
> > chunked, specifying Content-Length header?
>
> This will not work, for two reasons. First, to use the "traditional" way,
> ffmpeg would need to know the exact size of the data when it starts writing
> the progress, and it can not know it, obviously. Second, if nginx is
> anything like the web servers I know, it will read the whole body before
> calling the CGI script.
>
> You could get the expected result using something like that:
>
> ffmpeg -progress /tmp/progress.txt ... &&
> curl -d @/tmp/progress.txt http://localhost:1234/
>
> but that defeats the purpose of the progress option.
>
> The other solution would be to implement a new protocol, maybe ffmultihttp,
> that would make a new request for each packet written (the progress option
> is already ready for that because it can work with UDP). That would not be
> too difficult a task, probably.
>
> Another solution: I notice that Debian package for nginx seems to include a
> patch for chunked POST data. Maybe you should look at it.
>
> Regards,
>
> --
>   Nicolas George
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.12 (GNU/Linux)
>
> iEYEARECAAYFAlDJ9SsACgkQsGPZlzblTJPKUgCfaOWTVfZ6ycF7qpI7ChQfy1fz
> aMgAnRuMnRKcalSeBST/dw1FhzwV4O/s
> =UMwY
> -----END PGP SIGNATURE-----
>
> _______________________________________________
> ffmpeg-user mailing list
> ffmpeg-user at ffmpeg.org
> http://ffmpeg.org/mailman/listinfo/ffmpeg-user
>
>