[FFmpeg-user] Unexpected Delay when overlaying two RTSP-Streams – Depending on input order

[Roland M. Eppelt]

Hello together,

I am discovering a strange behaviour when overlaying two RTSP-Streams: 
One of the streams is always delayed by 5 seconds – which one depends on 
the input order of the two streams in ffmpeg: always the second input 
stream is displayed with a delay of 4-5 seconds.

What is the cause of this behaviour? How can I circumvent it, so that 
both streams are in sync?

Thank you very much!

For demonstration purposes I've written a small linux shell script using
• ffmpeg version 4.4.4 (SUSE Linux)
• mediamtx v1.0.3 https://github.com/bluenviron/mediamtx
• vlc

This script creates
• stream1 displaying "1. %{localtime}"
• stream2 displaying "2. %{localtime}"
• stream3 merging 1+2
• stream4 merging 2+1

Please notice that the stream3/4-filter always overlays stream2 over the 
colorkeyed stream1.

Finally I get this result:
-> Stream3 shows a delayed stream2.
-> Stream4 shows a delayed stream1.

Regards,
Roland.

------ 8< ------ 8< ------ 8< ------ 8< ------ 8< ------ 8< ------

#!/bin/sh
# Start RTSP-Server MediaMTX (formerly: rtsp-simple-server)
./mediamtx &
RTSP=rtsp://127.0.0.1:8554
# Start first stream
ffmpeg -re -nostdin -f lavfi -i color=c=white:r=5 -vf 
"drawtext=fontsize=20:text='1. %{localtime}'" -c:v libx264 -g 5 
-rtsp_transport tcp -f rtsp $RTSP/stream1 2>/dev/null &
# Start second stream
ffmpeg -re -nostdin -f lavfi -i color=c=white:r=5 -vf 
"drawtext=fontsize=20:y=30:text='2. %{localtime}'" -c:v libx264 -g 5 
-rtsp_transport tcp -f rtsp $RTSP/stream2  2>/dev/null &
sleep 1
# Start Stream 3 = 1+2
/usr/bin/ffmpeg -nostdin -rtsp_transport tcp -i $RTSP/stream1 
-rtsp_transport tcp -i $RTSP/stream2 -filter_complex 
"[0:v]colorkey=white:0.01:0.5[ck];[1:v][ck]overlay" -c:v libx264 -g 10 
-preset ultrafast -crf 25 -rtsp_transport tcp -f rtsp $RTSP/stream3 
2>/dev/null &
# Start Stream 4 = 2+1
/usr/bin/ffmpeg -nostdin -rtsp_transport tcp -i $RTSP/stream2 
-rtsp_transport tcp -i $RTSP/stream1 -filter_complex 
"[1:v]colorkey=white:0.01:0.5[ck];[0:v][ck]overlay" -c:v libx264 -g 10 
-preset ultrafast -crf 25 -rtsp_transport tcp -f rtsp $RTSP/stream4 
2>/dev/null &
sleep 20
# Display Stream 3 and  4
vlc $RTSP/stream3 &
vlc $RTSP/stream4 &
# Wait for user
echo "Press enter to stop"
read
# Cleanup
killall ffmpeg vlc mediamtx

------ 8< ------ 8< ------ 8< ------ 8< ------ 8< ------ 8< ------