[Libav-user] Multithreaded Asynchronous Encoding/Muxing

[Strahinja Radman]

>From: BIGLER Don (Framatome)
>Sent: 27 February 2019 16:25
>To: This list is about using libavcodec, libavformat, libavutil, libavdevice and libavfilter.
>Subject: Re: [Libav-user] Multithreaded Asynchronous Encoding/Muxing
>
>>>From: BIGLER Don (Framatome)
>>>Sent: 27 February 2019 14:55
>>>To: libav-user at ffmpeg.org
>>>Subject: [Libav-user] Multithreaded Asynchronous Encoding/Muxing
>>>
>>>All,
>>>
>>>I am writing an application that displays, encodes, and muxes live video using libav as the backend.  The audio and video encoding occurs asynchronously in the background, each in its own thread, such that the encoded packets arrive to the 
>>>av_interleaved_write_frame() function call at different times.  H.264 video encoding by itself works fine.  However when I add audio, the audio is out of sync from the video even though the audio and video pts are in sync (from avdevice 
>>>using >the matroska muxer).  The cause of the problem is not clear to me.  Specifically, here are my questions:
>>>
>>>1.          Can the av_interleaved_write_frame() function handle multithreaded asynchronous calls for audio and video streams?  
>>>2.          The transcoding example uses a filter graph using buffer/abuffer filters.  My current implementation does not use the buffer/abuffer filters because I am not applying any filters before encoding.  Are they required for my 
>>>situation?
>>>3.          The encoding happens randomly within the stream such that the first pts received by the muxer is not zero.  Is this required?
>>>
>>>I will greatly appreciate any assistance!
>>>
>>>Regards,
>>>Don Bigler
>
>>Hi Don,
>
>>I don’t think that av_interleaved_write_frame() is thread safe. What you could do is gather video and audio in a thread safe queue, possibly order the >packets, and then in separate thread send from that queue to the
>>av_interleaved_write_frame(). For me this works fine. 
>
>>Kind regards,
>>Strahinja
>
>Hi  Strahinja,
>
>Thank you for the prompt reply.  Your recommendation is fairly easy to implement.  That said, it leads to additional questions.  
>   1.  How many audio and video packets should I queue before sending them to  av_interleaved_write_frame()?  This is not obvious because as far as I can tell the single-threaded examples don't perform any queueing (unless this happens behind >the scenes using the buffer/abuffer filters).  Otherwise it seems that the packets can just be fed to av_interleaved_write_frame() as they arrive.
>   2.  What do you mean by packet ordering?  By PTS or DTS?  Are you suggesting that the order matters?  It seems that the order should just be first-out first-in as that is what occurs in the single-threaded case.   
>   3.   What if I pause the recording and then start the recording?  Do I need to adjust PTS/DTS for the pause before sending them to av_interleaved_write_frame()?
>   4.   Are you able to point me to some open-source code where this approach is implemented?
>
>Regards,
>Don

Hi Don,

Your queue serves only as a place where both threads can access without colliding. So its size is only determined by the speed of the encoding process and how fast you write to the output.
In my case I had to pause between calls to av_interleaved_write_frame() so that it appears that my encoding process is real time. Maybe you wont need to do this but its nice to keep in 
mind. Av_interleaved_write_frame() will do its own internal buffering also, the default is couple of seconds iirc. 

I don’t know if this is done automatically now, but I order my packets by PTS before calling the av_interleaved_write_frame(). Maybe they are ordered by DTS inside of the av_interleaved_write_frame(). Speed of your threads will be different so will the number of packets that come out of  them. This means your queue will be full of packets that may not be ordered (100 audio packets and then 2 video). Without reordering I am experiencing errors, with ordering by PTS it works fine.  

Output muxer expects timestamps that are increasing monotonically, meaning in case of pause you continue where you left, you don’t include a pause. 

Unfortunatelly no. 

Kind regards,
Strahinja
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://ffmpeg.org/pipermail/libav-user/attachments/20190227/233a90cc/attachment.html>