[FFmpeg-devel] high cpu usage and frame dropping

[Don Moir]

> On Wed, May 02, 2012 at 02:31:34PM -0400, Don Moir wrote:
>> All players I tested fail badly when I push them. They try to drop 
>> frames, but they can stop displaying all together and never recover, 
>> audio is no longer serviced, and worse.
>>
>> A sad state of affairs for players that have been out for awhile.
>
> I don't think you can have tested many. MPlayer will just desync, or you
> can tell it to skip even decoding, though that has a good chance of
> crashing.
> Anyway I can't see why it is a sad state, in 90% of cases the video will
> be unwatchable anyway, why putting a lot of effort into what just ends
> up prolonging the users' pain.

I tested WMP, QuickTime, VLC, SMPlayer, and player built into firefox. I 
would say the most used players on windows are WMP, QuickTime, and VLC. 
SMPlayer uses mplayer.

All these players failed badly when pushed. When I say badly, I mean 
ridiculously bad.

It's a sad state mostly because it's handled poorly to begin with and then 
they can't even recover from it. Again stop playing altogether, infinite 
short audio loops (audio stops getting serviced), and more. WMP seemed to 
handle it better but only because it did nothing to deal with it and audio 
and video totally out of sync.

>> I have tried various flags etc to speed things up and that helps a little 
>> but not enough and they don't really work anyway. There could be 
>> something in the flags or callbacks that would help alot but I have not 
>> found them yet.
>
> You're too vague. skiploopfilter always helps a lot for files using the
> feature.
> For high level H.264 skipping nonref frames will cut the processing time
> by at least half.

I have tried skip_frame and skip_loop_filter. These do help. The thing is 
you need to turn these on and off on the fly depending on CPU usage etc. and 
that is ok but still in some cases not enough. Also, you can't use these 
blindly as you may not get any frame at all depending on demuxer and if 
there are going to be future keyframes etc. I tried AVDISCARD_NOREF and 
AVDISCARD_NONKEY with some improvement.

I think using skip_frame and skip_loop_filter can have some side effects 
such as distortion also, when changing on the fly, so that has to be dealt 
with.

In our applications, we have the abilitily to play any number of videos at 
the same time. This is not to say we do this in an abitrary way. The most 
use case of this is to fade in one video while the other is fading out. 
There was a request that I do this and that led me down the path of beating 
the hell out of things related to CPU usage. On my fast machine I played 7 
high res videos all at the same time in one container with no real impact 
and no frame drops.

In my test app, I can control when I need to start dropping frames depending 
on desired CPU usage. I can set this to allow anywhere from 1 to 100 percent 
usage. For much of testing with many different files, I set it to 1 or 2 
percent so I can see the effect it might have.

So the thing that works absolutely the best so far is to stop decoding and 
wait but I will continue to dig.

Is there any possible way to know if a future keyframe will exist on a 
general basis when index entries may not exist ? If it means I have to look 
as every possible codec and make a best guess that would probably be ok.