[FFmpeg-user] Out of virtual memory (swap, I assume)

[Mark Filipak (ffmpeg)]

Here is the original command that fails:

ffmpeg -i "source part5.mkv" -map 0 -filter_complex 
"minterpolate=fps=48000/1001:mi_mode=mci:mc_mode=obmc:scd=fdiff:scd_threshold=10:vsbmc=1:search_param=32, 
telecine=pattern=3322,pp=linblenddeint, setpts=N*1001/60000/TB" -codec:v libx265 -x265-params 
crf=20:qcomp=0.60 -codec:a copy -codec:s copy "part5.mkv"

Here is the test command for this run:

ffmpeg -i "source part5 (44561 frames).mkv" -map 0 -filter_complex 
"telecine=pattern=3322,pp=linblenddeint, setpts=N*1001/30000/TB" -codec:v libx265 -x265-params 
crf=20:qcomp=0.60 -codec:a copy -codec:s copy "part5 without minterpolate.mkv"

Test results:

Memory (of 32GB): 13.2GB to 25.4GB in 25 minutes, 31GB 15 minutes later, then hover between 31GB & 
31.6GB thereafter -- ramp up then drop, ramp up then drop: I imagine that's normal malloc-free 
cycles. But why would ffmpeg defer free'ing until it had filled memory?

Swap commit (of 128GB): 14.2GB to 27.3 in 25 minutes, 33GB 15 minutes later, 46GB 45 minites later 
-- rising steadily... Aha! Then leveling off at 46.3GB. But why would swap commits begin immediately 
instead of when memory has been filled?

A great many more "Starting new cluster due to timestamp", a continuous stream -- coming so fast & 
furious that I can't read what the current frame number is before "Starting new cluster due to 
timestamp" overwrites it.

Ah! The "Starting new cluster due to timestamp" stream has ended, and it ended at the same time that 
swap commit leveled off.

Is that a clue?

The run just completed at 3:07AM. It started at 1:37AM, so took 1 hr. 40 min. That contrasts with 
the original run (that included minterpolate) which ran all night before dying.

You know what I think? From the behavior of memory & swap commits, it looks to me that it's a 
combination of "Starting new cluster due to timestamp" persisting for so many hours longer during 
the original run.

Actually, there were 3 original runs because I tried it 3 times; all died.

=====

For what it's worth, here's my notes made while planning the source video cutting:

Divide transcoding the 222545 input frames between 5 concurrent tasks.
(222545-0)/5 = 44509 input frames per task

Define the bounds of the 5 segments.
0-44509, 44510-89018, 89019-133527, 133528-178036, 178037-222545

Adjust the segments so that each starts with the nearest key frame having an even frame number.
0-44503, 44504-88999, 89000-133447, 133448-177983, 177984-222545

Add 1 because MKVToolNIX expects frame numbers to start from 1, not zero.
1-44504, 44505-89000, 89001-133448, 133449-177984, 177985-222546

minterpolate silently drops the final 2 input frames, so add 2 frame overlaps to compensate.
1-44506, 44505-89002, 89001-133450, 133449-177986, 177985-222548

Via MKVToolNix, segment the input per the compensated bounds.
source-part1.mkv source-part2.mkv source-part3.mkv source-part4.mkv source-part5.mkv

Create 5 scripts and run them concurrently.
source part1.mkv.cmd
   ffmpeg -i source-part1.mkv -map 0 -filter_complex 
"minterpolate=fps=48000/1001:mi_mode=mci:vsbmc=1, telecine=pattern=3322,pp=linblenddeint, 
setpts=N*1001/60000/TB" -codec:v libx265 -x265-params crf=20:qcomp=0.60 -codec:a copy -codec:s copy 
part1.mkv
   pause
source part2.mkv.cmd
   ffmpeg -i source-part2.mkv ... part2.mkv
   pause
source part3.mkv.cmd
   ffmpeg -i source-part3.mkv ... part3.mkv
   pause
source part4.mkv.cmd
   ffmpeg -i source-part4.mkv ... part4.mkv
   pause
source part5.mkv.cmd
   ffmpeg -i source-part5.mkv ... part5.mkv
   pause

Hope that the resulting motion vectors at segment joins will be close enough that they're not too 
apparent.

-- 
I don't have a dog.
And furthermore, my dog doesn't bite.
And furthermore, you provoked him.