[FFmpeg-user] Multithreaded multioutput problem

[Deron]

On 4/21/15 2:51 AM, Marton Balint wrote:
>
>
> On Mon, 20 Apr 2015, Deron wrote:
>
>> On 4/20/15 4:22 PM, Marton Balint wrote:
>>>
>>> On Mon, 20 Apr 2015, Deron wrote:
>>>
>>>> On 4/20/15 1:48 PM, Marton Balint wrote:
>>>>> On Mon, 20 Apr 2015, Deron wrote:
>>>>>
>>>>>> Another user has contacted me with the exact same problem hoping 
>>>>>> that I stumbled on a solution for the below problem. I did not, 
>>>>>> so I am adding some more extensive log output in hopes that 
>>>>>> someone might recongize the source of the problem.
>>>>>>
>>>>>
>>>>> I have also experienced this problem. Depending on your use case 
>>>>> you can face a similar situation with only one input and one 
>>>>> output as well and a few cores. There is no buffering between the 
>>>>> various stages of the ffmpeg encoder/filter/decoder pipeline, so 
>>>>> even if the stages by themselves are multi threaded, you won't be 
>>>>> able to scale up, because passing data between the stages is done 
>>>>> in a single thread.
>>>>>
>>>>> Or at least that is what I think is going on. So as far as I know 
>>>>> you can only scale up properly by running multiple ffmpeg 
>>>>> instances. E.g: create a multicast and encode that.
>>>>>
>>>>> Regards,
>>>>> Marton
>>>>
>>>>
>>>> What would be the best way to solve this then? I'm not sure I 
>>>> understand what you mean by multicast. Having multiple ffmpeg's 
>>>> decoding the same original mpegts input would be pretty slow.
>>>
>>> If that overhead is unacceptable, then I don't see any other way. 
>>> You wrote you had plenty of free CPU :)
>>
>> Speaking of CPU and having been away from the problem for many weeks, 
>> I forget about the important part. The encoding process for the data 
>> runs at better than real time (nearly 2x) when pulling from a file 
>> instead of the /dvb device. So I don't think that this is thread 
>> bound for me but something else.
>>
>> I'm still missing a piece of the puzzle. Something else is causing me 
>> grief.
>>
>
> In that case have you tried using the threaded input read support of 
> ffmpeg? You have to specify more than one inputs (add an extra dummy 
> input source) and play with -thread_queue_size option.
>
> Regards,
> Ma 


I've resolved this problem and would like to report my solution. The 
error evidently was coming from some part of azap/dvb device. When I 
increased the buffer size as per http://panteltje.com/panteltje/dvd/ the 
problem went away. I can now capture and encode 6 channels 
simultaneously, outputting 2 video,1 audio, and images for each on this 
computer. I presume the problem was simply that setup time in ffmpeg 
increased enough that it (temporarily) exceeded real time encoding and 
overflowed the buffer between azap and ffmpeg.

I've considering writing an azap-like input module for ffmpeg. I'm sure 
this is generally frowned on, but in this case I see a couple of 
advantages. One is obviously this problem with too small a buffer. The 
other is when interference causes the stream to be degraded, ffmpeg will 
start consuming huge amounts of cpu until it actually freezes up the 
computer. Since azap knows when the signal tanks, it would appear to me 
it could deal with the encoding issue if it had access into the encoder 
itself. Maybe some way exists already to deal with it, or ffmpeg should 
simply be made more robust to deal with this problem?

Anyway, thanks for the help!

Deron