[FFmpeg-devel] high cpu usage and frame dropping

[Don Moir]

I think this looks like a user issue and maybe I should ask this elsewhere, but I think it is probably more of an enhancement and I don't think I will get an acceptable answer anywhere else.

When things get behind or cpu usage is high, it's easy enough not to display a frame but that doesn't improve things much.

All players I tested fail badly when I push them. They try to drop frames, but they can stop displaying all together and never recover, audio is no longer serviced, and worse.

A sad state of affairs for players that have been out for awhile.

For me, the slowest thing is avcodec_decode_video2. Nothing else even comes close. Just saying it's slow because it's an intense process so don't get me wrong on that.

I have tried various flags etc to speed things up and that helps a little but not enough and they don't really work anyway. There could be something in the flags or callbacks that would help alot but I have not found them yet.

The thing that works best for me so far, is to peek ahead and wait for a keyframe. This will drop cpu usage considerably. Note that you cannot always know when they will be a next keyframe as there might not be any index entries. When there are no index entries, the only thing I have been successful with so far is to look ahead and see if any of the packets are marked with AV_PKT_FLAG_KEY and wait for that if it exist.

If I determine I can wait for a keyframe, I call avcodec_flush_buffers on the video stream and wait. In most cases this works very well. That is, cpu usage has dropped a great deal and I get a nice clean display. In some cases I get a little distortion but it is much better than the behavior I see with any other player tested.

The above logic has been experimental and it seems to me that more has to be done. For example, I am not sure what state the demuxer might be in and depending on AV_PKT_FLAG_KEY is not an absolute as it depends on the demuxer as to what this actually means.

So ideally, there would be way to reset the demuxer (avcodec_flush_buffers?), and wait for a future keyframe before calling avcodec_decode_video2 again. You also need to know If a future keyframe is not going to be available or the timestamp of that future keyframe. You know this when there are index entries but it depends on the demuxer when this might be available if at all.

I hope you all might have some feedback on this and there may be built in ways to deal with this better. After testing many files, the only thing I can tell you is that it works pretty good but I know I have a ways to go with it yet.

The real problem is when there are no index entries or it hasn;t been build yet and the inconsistency of the index entries table in general.