[FFmpeg-devel] [RFC] How to deal with libavfilter buffer overflows?

[Nicolas George]

Le decadi 30 vendémiaire, an CCXXIII, Stefano Sabatini a écrit :
> $ ffmpeg -f lavfi -i "testsrc=d=10, split[t0][t1]; [t0]select='gte(t,5)',setpts=PTS-STARTPTS[s0]; [t1] select='lt(t,5)', setpts=PTS-STARTPTS[s1]; [s0][s1] concat" -y outtest.mp4
> [...]
> Input #0, lavfi, from 'testsrc=d=10, split[t0][t1]; [t0]select='gte(t,5)',setpts=PTS-STARTPTS[s0]; [t1] select='lt(t,5)', setpts=PTS-STARTPTS[s1]; [s0][s1] concat':
>   Duration: N/A, start: 0.000000, bitrate: N/A
>     Stream #0:0: Video: rawvideo (RGB[24] / 0x18424752), rgb24, 320x240 [SAR 1:1 DAR 4:3], 25 fps, 25 tbr, 1000k tbn, 1000k tbc
> [...]
> Output #0, mp4, to 'outtest.mp4':
>   Metadata:
>     encoder         : Lavf56.9.101
>     Stream #0:0: Video: h264 (libx264) ([33][0][0][0] / 0x0021), yuv444p, 320x240 [SAR 1:1 DAR 4:3], q=-1--1, 25 fps, 12800 tbn, 25 tbc
>     Metadata:
>       encoder         : Lavc56.8.102 libx264
> Stream mapping:
>   Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (libx264))
> Press [q] to stop, [?] for help
> [out @ 0x3140560] 100 buffers queued in out, something may be wrong.
> frame=  125 fps=0.0 q=-1.0 Lsize=      25kB time=00:00:04.92 bitrate=  42.4kbits/s    
> video:23kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 9.620967%
> 
> I frequently stumble upon situations when buffer overflows happen, and
> they are usually a showstopper in many applications. For example when
> recombining streams with the concat filter, or when overlaying. The
> output is usually playable but misses audio or video frames.  For
> example in the case above, the command is trying to switch order of
> the two input parts, but only the second part is displayed and the
> user is left with the vague message "something may be wrong".
> 
> So the current buffering system sometimes fails, and the user has no
> clue to understand where the problem lies. I want at least to improve
> diagnostics, but I want to listen about other developers experience
> first, and if you have ideas to tackle such issues.

Hi.

I am well aware of the problem too, and actually I have recently started
writing a summary to try address it, along with the other problems of the
current lavfi API.

There are several underlying issues that are raised by your example:

* You have AB, and you want to split and merge it into BA, but you only get
  B: it looks like it is "just" a bug. The issue disappears if I test with
  "ffmpeg -lavfi filter" instead of "ffmpeg -f lavfi -i filter". I have yet
  to fully understand if the bug is in lavd/lavfi or in lavfi itself, but
  even in the former case, there is still the problem that the API is almost
  impossible to use correctly.

* Apart from this bug, this command works, but that is a bit of luck.
  Someone raised the input buffer queue for concat to 256, and your graph
  requires 128. The warning you see is just a warning. But if you were to
  raise the duration of the first segment to something above 256 frames, it
  would indeed fail. The message to look for is "Buffer queue overflow,
  dropping.", not "100 buffers queued in output stream".

* You have AB from a non-replayable input and you want BA. That can not work
  reliably. If your input were replayable, you could replay it as A1B1 and
  A2B2 and produce B2A1. But since it is not replayable, you need to buffer
  all A as decoded frames in memory. In this particular case, that is
  5×25×320×240×3 = 27 Mo, still acceptable. But imagine the same thing with
  a full-HD credits sequence: 5×90×1920×1080×3/2 = 1.3 Go.

  When it has already eaten half a gigaoctet of memory, ffmpeg/lavfi can
  only guess whether this is expected and the memory consumption will soon
  stop or if the graph has a bug (arithmetic mistake in setpts, links
  connected to the wrong pad) that makes it eat memory indefinitely.

  Since the second case is likely to lead to memory and swap exhaustion,
  making the host machine unusable, ffmpeg should not attempt it without
  explicit user consent.

  You can usually give that user consent by inserting the fifo filter
  somewhere. In this particular case, inserting fifo at the s1 link does the
  trick.

  We could enhance the "Buffer queue overflow" message to print the exact
  link name where it happens, so the user can know where the fifo is needed.
  But that is not a very good solution because fifos are dangerous too.

The solution I consider to propose involves moving the FIFO service directly
into the framework. That would allow a few goodies, such as global
allocation of the queue and statistics.

Regards,

-- 
  Nicolas George
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20141022/d51ea11c/attachment.asc>