[Libav-user] Multithreaded Asynchronous Encoding/Muxing

[BIGLER Don (Framatome)]

>>From: BIGLER Don (Framatome)
>>Sent: 27 February 2019 14:55
>>To: libav-user at ffmpeg.org
>>Subject: [Libav-user] Multithreaded Asynchronous Encoding/Muxing
>>
>>All,
>>
>>I am writing an application that displays, encodes, and muxes live video using libav as the backend.  The audio and video encoding occurs asynchronously in the background, each in its own thread, such that the encoded packets arrive to the 
>>av_interleaved_write_frame() function call at different times.  H.264 video encoding by itself works fine.  However when I add audio, the audio is out of sync from the video even though the audio and video pts are in sync (from avdevice 
>>using >the matroska muxer).  The cause of the problem is not clear to me.  Specifically, here are my questions:
>>
>>1.          Can the av_interleaved_write_frame() function handle multithreaded asynchronous calls for audio and video streams?  
>>2.          The transcoding example uses a filter graph using buffer/abuffer filters.  My current implementation does not use the buffer/abuffer filters because I am not applying any filters before encoding.  Are they required for my situation?
>>3.          The encoding happens randomly within the stream such that the first pts received by the muxer is not zero.  Is this required?
>>
>>I will greatly appreciate any assistance!
>>
>>Regards,
>>Don Bigler

>Hi Don,

>I don’t think that av_interleaved_write_frame() is thread safe. What you could do is gather video and audio in a thread safe queue, possibly order the >packets, and then in separate thread send from that queue to the av_interleaved_write_frame(). For me this works fine. 

>Kind regards,
>Strahinja

Hi  Strahinja,

Thank you for the prompt reply.  Your recommendation is fairly easy to implement.  That said, it leads to additional questions.  
   1.  How many audio and video packets should I queue before sending them to  av_interleaved_write_frame()?  This is not obvious because as far as I can tell the single-threaded examples don't perform any queueing (unless this happens behind the scenes using the buffer/abuffer filters).  Otherwise it seems that the packets can just be fed to av_interleaved_write_frame() as they arrive.
   2.  What do you mean by packet ordering?  By PTS or DTS?  Are you suggesting that the order matters?  It seems that the order should just be first-out first-in as that is what occurs in the single-threaded case.   
   3.   What if I pause the recording and then start the recording?  Do I need to adjust PTS/DTS for the pause before sending them to av_interleaved_write_frame()?
   4.   Are you able to point me to some open-source code where this approach is implemented?

Regards,
Don