[FFmpeg-user] Syncing Multiple Audio Sources To One Video Source

[Gabriel Balaich]

Hello all,


I'd like to start by saying thank you for your time and any help

you can send my way, I really appreciate it. I'm trying to keep three

outputs in-sync, 3 audio and 1 video. I'll start with my block of

code and then I'll try to explain exactly what I'm trying to do. Here

is my current block:


ffmpeg -guess_layout_max 0 -y -f dshow -video_size 3440x1440 -rtbufsize
2147M -pixel_format nv12 -r 100.00 ^

-i video="Video (00 Pro Capture HDMI 4K+)":audio="Analog (1+2) (RME
Fireface UC)" -map 0:0 -map 0:1 ^

-codec:v h264_nvenc -pix_fmt nv12 -b:v 250M -maxrate 250M -bufsize 250M
-b:a 320k -ac 2 ^

-segment_time 600 -segment_wrap 9 -f segment
C:\Users\djcim\Videos\PC\PC%02d.mp4 ^

-guess_layout_max 0 -f dshow -rtbufsize 100M -i audio="Analog (3+4) (RME
Fireface UC)" -map 1:0 -b:a 320k -ac 2 ^

-segment_time 600 -segment_wrap 9 -f segment
C:\Users\djcim\Videos\PC\Voices\Theirs\TheirsPC%02d.wav ^

-guess_layout_max 0 -f dshow -rtbufsize 100M -i audio="Analog (5+6) (RME
Fireface UC)" -map 2:0 -b:a 320k -ac 2 ^

-segment_time 600 -segment_wrap 9 -f segment
C:\Users\djcim\Videos\PC\Voices\Mine\MinePC%02d.wav


I have two PC's, one that I'm gaming on, one that I'm recording on. I

also have two RME audio cards (one in each respective PC) hooked up

to each other to share audio between the two PC's. My main display

(3440x1440 @ 100Hz) is hooked up to a displayport splitter that

splits to my recording PC's capture card. I am also routing my game

audio from the sound card in my gaming PC to the capture PC's sound

card. On top of this I have a mic that I'm talking into and a discord

(voice chat) that I'm recording simultaneously with the game / game

audio. All three desired audio sources are on their own input channels

on the RME audio card hooked up to the capture PC. I'm trying to keep

these files separated so I can mix the volumes of the game audio, my

mic, and the discord to make sure one audio source isn't overwhelming

the others. Or so I can choose to entirely omit the voices altogether

leaving only game audio. Also from what I understand I couldn't link

multiple audio channels to one video channel with dshow per the

documentation anyway. Also as seen in the block I am taking advantage

of segmenting, it's much nicer to deal with 9 10 minute clips than it is to

deal with a 1 130 minute clip. This way the  parts are constantly

overwriting each other allowing me to record continuously non-stop

without filling up my SSD and giving me the last 130 minutes of content

at any given time.


I'm facing two problems with this setup currently: 1) the audio from

my mic and the discord audio is starting before the game video/audio

creating sync issues right off the bat for the rest of the

recordings. 2) The game video and audio are in-sync at first but as

I drop frames throughout the recordings they slowly become unsynced.

I drop frames every few hours (sometimes more, sometimes less) do to

the real-time buffer of the capture card getting too full. It would seem

like increasing the buffer size or reducing the bitrate would fix this
issue

but sadly it does not. Regardless of what I do I drop a few frames every

few hours do to an overfilled buffer.


Basically what I am looking for is a way to start all three streams

at the same time, and then when a frame is dropped just duplicate the

last one to maintain the 100 fps goal keeping everything in-sync. Is

this possible with the current state of FFMPEG? At first I thought

-vsync was the key but after I couldn't get it working from what I

gathered it doesn't work if frames drop before ingestion. I

understand that this is ambitious but it has been from the start,

finding hardware that allowed me to record at such a high resolution

and framerate took months (and wasn't cheap). Even after obtaining the

necessary hardware every software I tried to utilize it with has failed me

time and time again. Until I found FFMPEG, it is so close, a truly

amazing program.


Thanks again for any suggestions or help.