[FFmpeg-user] Sync video and audio after using complex filters to blend two files

[luke harrison]

Hi,

I have an application where I am using FFMPEG to automatically edit and add to a highly predictably daily program. I start with a raw file which I then run an analysis on and determine where the start and end points are. I then cut that file using a simple copy command. I then want to merge the cut file with an introduction by blending the one into the other. The introduction does not have any audio but the cut file does. After blending them I need to get the audio from the cut file (which has been pushed forward by 11s in the blended file) and re-sync it with the blended file. For additional clarity the steps and commands that I use are below. My ultimate question is listed at the bottom. Thanks to anyone who can help in advance.


Once my algorithm has identified the start and stop points I cut it using the following command.

ffmpeg -i $(ls -t /Users/broadcastingza/Desktop/MorningWorshipRaw/*.MP4 | head -1) -ss ${opentime} -to ${closetime} -c:v copy -c:a copy main_file.mp4 -y

Next I want to add an introduction to the file. The introduction is made up of images which are turned into videos. The following commands create the introduction.

ffmpeg -framerate 1/10 -i frames/temp3.jpg -c:v libx264 -profile:v main -preset slow -b:v 1200k -vsync 1 -r 25 -pix_fmt yuv420p -s 1280x960 intro.mp4 -y

ffmpeg -framerate 1/3 -i frames/black.jpeg -c:v libx264 -profile:v main -preset slow -b:v 1200k -vsync 1 -r 25 -pix_fmt yuv420p -s 1280x960 black.mp4 -y

ffmpeg -i black.mp4 -i intro.mp4 -f lavfi -i color=black -filter_complex "[0:v]format=pix_fmts=yuva420p,fade=t=out:st=2:d=1:alpha=1,setpts=PTS-STARTPTS[va0];[1:v]format=pix_fmts=yuva420p,fade=t=in:st=0:d=1:alpha=1,setpts=PTS-STARTPTS+2/TB[va1];[2:v]scale=1280x960,trim=duration=12[over];[over][va0]overlay[over1];[over1][va1]overlay=format=yuv420[outv]" -vcodec libx264 -s 1280x960 -r 25 -vsync 1 -crf 18 -map [outv] introblend.mp4 -y

The introduction is a blended composite made up of a 3s black intro overplayed on top of another 10s video. It changes dynamically each day and therefore has to be created each day. There is no audio at all in the introduction video.

Once the introduction is complete I blend it and the main file together using the following command:

trim=$(echo $closetime - $opentime | bc -l)

trim=$(echo $trim + 11 | bc -l)

ffmpeg -i introblend.mp4 -i main_file.mp4 -f lavfi -i color=black -filter_complex "[0:v]format=pix_fmts=yuva420p,fade=t=out:st=11:d=1:alpha=1,setpts=PTS-STARTPTS[va0];[1:v]format=pix_fmts=yuva420p,fade=t=in:st=0:d=1:alpha=1,setpts=PTS-STARTPTS+11/TB[va1];[2:v]scale=1280x960,trim=duration=$trim,setpts=PTS-STARTPTS[over];[over][va0]overlay[over1];[over1][va1]overlay=format=yuv420,setpts=PTS-STARTPTS[outv]" -c:v libx264 -crf 18 -r 25 -profile:v main -s 1280x960 -preset ultrafast -vsync 1 -map [outv] blended.mp4 -y

This produces a file without any audio (I’ve tried mapping the audio to the output but it results in lost frames at the output for some reason and it still does not solve my audio sync issue). To produce a final file with audio I run the following command:

ffmpeg -itsoffset 11 -i main_file.mp4 -i blended.mp4 -map 1:0 -map 0:1 -vcodec copy -acodec copy "final.mp4" -y


This command aims to shift the STARTPTS of the main file out by the 11s that the main file is shifted in the blended video and then use the video track from the blended video and the audio track from the shifted video. All good in theory but not so good in practice.

I have found that with some videos the sync is out by almost up to 1s throughout the file and with other videos the sync is almost spot on at the end of the process. I thought that it could be due to the various files having different PTS offsets at the start so I checked them all using ffprobe and found that the largest offset was that of the edited “main_file” and it was in the region of 0.009.

Please could some kind soul with deeper insight into ffmpeg help me understand why I am not able to achieve sync using this technique and perhaps even suggest a fix/another technique which would allow me to blend two files while still maintaining the audio sync in the second one?

Thanks,
Luke.