[FFmpeg-devel] Fixing av_seek

[Don Moir]

>>> Finnally getting around to looking more closely at seek problems in 
>>> ffmpeg.
>>>
>>> This is a follow up to ticket #504. 
>>> https://ffmpeg.org/trac/ffmpeg/ticket/504
>>>
>>> My goal is to write a higher level seek function that depends on the 
>>> lower level ffmpeg seek functions. In some cases, patches to ffmpeg may 
>>> need to be applied.
>>>
>>> The higher level function will only deal with a timestamp that might 
>>> correspond to a slider or similiar, and will seek very close to that 
>>> timestamp without image distortion.
>>>
>>> There are several categories of success or failure when seeking. This 
>>> first group (Group A), deals with things that work or are easy to deal 
>>> with.
>>>
>>> o - everything works and works well. There are plently of index_entries 
>>> and the format in question does the right thing. CODEC_ID_FLV1 is an 
>>> example that works well but some other formats work just as well.
>>>
>>> o - there are no index_entries or none of the index_entries are marked 
>>> as AVINDEX_KEYFRAME. This will cause avformat_seek_file or av_seek_frame 
>>> to fail. If there are index_entries and the first entry represents the 
>>> first frame, then I have found that just marking this first index_entry 
>>> with AVINDEX_KEYFRAME fixes the ffmpeg seek failure problem. If there 
>>> are no index_entries then you can add one using av_add_index_entry using 
>>> the correct parameters for the first frame and marking it as 
>>> AVINDEX_KEYFRAME. This assumes the first frame must be a keyframe and 
>>> easier to deal with this at startup rather then trying to deal with it 
>>> when doing a seek.
>>>
>>> o - lower level seek functions mostly deal with keyframes and thats all 
>>> good, but thats not good enough for an end user trying to use a slider 
>>> to seek. So get as close as possible using one of the ffmpeg seek 
>>> functions and then proceed forward to the timestamp of interest. This 
>>> works pretty good and is fast enough. Mostly, good files will have 
>>> enough AVINDEX_KEYFRAME index_entries to make this behave quite well. 
>>> Even when there are not enough entries, it works good. You probably want 
>>> to do this in a separate thread. Make sure you first seek to a timestamp 
>>> that is less than or equal to the timestamp of interest so you can 
>>> proceed forward. There are probably ways to improve on this.
>>>
>>> The second group (Group B) falls into the class of failures or bad 
>>> behavior that is not easy to deal with. e.g. an inability to make it 
>>> work as expected for various reasons.
>>>
>>> I am still dealing with trying to find out Group B problems. I have had 
>>> issues with MPEG1, THEORA, and a few others, and I will know more about 
>>> these problems as I get into it further.
>>>
>>> I use VLC and SMPlayer to cross check and both of these have their share 
>>> of seek problems. I use my own player to test with.
>>
>> Using this h264 file, the index_entries are built on the fly and appear 
>> to be incorrect.
>>
>> http://sms.pangolin.com/temp/bad_seek_index_h264.zip  (2.5mb)
>>
>> codebase: git-67f5650 - after fixes with h264 seeking but I don't see any 
>> difference with older codebase versions than this one.
>>
>> Just after av_open_input_file, there are 2 index_entries. The first one 
>> has timestamp of 0 and 2nd has timestamp of 0x48f. The odd thing is they 
>> both have the same pos which 0x122F. So seems strange from the start. As 
>> playback progresses, the number of index_entries increases and this is 
>> normal.
>>
>> When you try to seek on this file to a supposed keyframe, you don't get a 
>> finished frame until much later (~4 seconds). What should happen is if 
>> you have seeked to a keyframe, you should be getting a finished frame 
>> immediately. This is the way I see it at any rate. So trying to seek into 
>> this file breaks the playback, meaning you will get a long pause before 
>> normal playback resumes because you are not getting a finsihed frame when 
>> you should be. If I just seek to zero on this file, then all is good and 
>> I get the finished frames as expected but of course it's slower to reach 
>> the requested timestamp.
>>
>> I bring this up because I think this represents a key problem with some 
>> of the codec types. That is index_entries are added that don't click with 
>> actual keyframes. I asked a question about this relationship on the 
>> original ticket but there was no response.
>>
>> I think getting this fixed will go a long way in fixing several seek 
>> problems. But I need information from you if you have any on this. If you 
>> don't have a answer, maybe "I or we don't know" would be better than no 
>> answer. Also not to just focus on h264 here because I believe there are 
>> other formats with this very same problem.
>
> This file identifies some problems with the ogg decoder when seeking:
>
> http://sms.pangolin.com/temp/bad_seek_ogg_BuckBunny.zip (46 mb)
>
> I am not making any attempt to cut some of these samples down to make sure 
> all is relevant.
>
> This file plays fine in my own player without any problems. SMPlayer 
> chokes on it abit, then a reload and it plays ok. ffplay plays it fine 
> with maybe a few artifacts.
>
> The problems come in when you try to seek on it. The ogg decoder has just 
> a basic seek that calls ff_seek_frame_binary. This can be found in 
> libavformat\oggdec.c in the function ogg_read_seek. Seems seek was kind of 
> an after thought and patched in to the orignial code.
>
> o - unlike many of the other decoders, the ogg decoder makes no attempt to 
> build it's index_entries when you do a seek. This is the first cause of 
> failure. Since it does not have these entries, it will fail in 
> ff_seek_frame_binary if the file has not been read sufficiently to create 
> it's index. Some ogg files work ok in this regard but BuckBunny will fail. 
> I currently go ahead and build these entries if needbe before I call any 
> seek for an ogg decoder file and this is a start to get this working. 
> BuckBunny is THEORA but ogg decoder handles several CODEC id's.
>
> o - The ogg decoder keeps a private structure in AVFormatContext.priv_data 
> (struct ogg) and within this it keeps some private stream information 
> (struct ogg_stream). In the ogg_stream it keeps track of lastdts and 
> lastpts. When you do a seek everything seems to be ok like position 
> information etc. But these 2 variables are not reset to AV_NOTPTS_VALUE 
> and they contain the values that were in effect before you did the seek. 
> So on the first read after a seek, you will get the right packet but the 
> pts and dts values will be wrong for the packet. No amount of flushing etc 
> seems to fix it and this is just a bug. I believe the ogg_read_seek 
> functions just needs to set these 2 vairaibles to the proper values. With 
> the second read after the seek the dts and pts values seem ok.
>
> For my own code I fixed the above issues and now seeking for this file 
> works fine. I still have some painting artifacts but this is more of a 
> general problem and I will deal with that later.
>
> I develop under windows because thats what most of our users use. Some mac 
> too. My experinece with linux is somewhat limited but I used to run that 
> using vmware. The application I use to test ffmpeg is designed for windows 
> and so it's a bit of a pain to debug and trace thru ffmpeg under windows 
> but this is just busy work.
>
> I can idenify and pinpoint these various problems but I am just not setup 
> to provide proper patches. I have people yelling at me already about 
> spending too much time on this and I have to earn a living. So while I 
> will continue to do this and idenify the problems, I will have to leave it 
> up to you to do the patches. Finding the problems is the most work anyway.
>
> So the 2 things for the ogg decoder is to make sure the index_entries are 
> built before you do a seek. Most of the other decoders do this correctly. 
> Make sure the lastpts and lastdts variables (struct ogg_stream) are set to 
> proper values in the ogg_read_seek function. ogg_read_seek should also 
> make sure the index_entries are built up to the timestamp of interest at 
> least before calling ff_seek_frame_binary. Note: just reading the packets 
> with no decoding is quick and sufficient to build the index_entries. I 
> don't do it if sufficient entries are built alreadly and stop when I have 
> built up to the timestamp of interest. These entries are normally built on 
> the fly as you read packets when processing the file. I think some 
> decoders mght have more elegant ways to do it but this is a good fallback.
>
> ffplay complains about first frame not being key frame when you try to 
> seek with ffplay but that doesn't seem to be a problem and may be related 
> to the above bugs. Seeking fails badly in SMPlayer for this file and 
> generally seeking does not work well at all in ffplay for this file. For 
> my own app with the above fixes, it now works perfect outside of some 
> artifacts but thats a different issue.

The attached file contains the code fragments along with comments that I am 
using to fix seeking with the ogg decoder. Not a patch but details the 
basics on fixing seeking for ogg related files. I tested it with a couple 
THEORA files and so far so good. This would need to be applied to 
libavformat\oggdec.c. The attached file contains a function for creating the 
index_entries if needbe and sets the lastpts and lastdts to AV_NOPTS_VALUE 
after a successful seek. 
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: oggdec_changes.txt
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20120123/8923a336/attachment.txt>