[FFmpeg-devel] MPEG-PS demuxer index memory usage

[Paul Kelly]

On Fri, 4 Jan 2008, Michael Niedermayer wrote:

> On Fri, Jan 04, 2008 at 05:50:01PM +0100, Michel Bardiaux wrote:
>> Michael Niedermayer a ?crit :
>>> On Thu, Jan 03, 2008 at 10:48:49PM +0000, Paul Kelly wrote:
[...]
>>>> Might it be a good idea to add another flag to turn off the
>>>> demuxer-specific indexing, and make individual demuxers respect this? A
>>>> general catch-all way of disabling indexing (or specifying that seeking
>>>> isn't required) might be more elegant though.
>>>>
>>>> I can get over the immediate problem by simply commenting out the line
>>>> calling av_add_index_entry() in libavformat/mpeg.c, but would like to help
>>>> get a better solution into libavformat if I can.
>>>
>>> Disabling it with a flag is surely interresting. But i think there are better
>>> solutions.
>>> One for example would be a max_index_size. And when thats reached index
>>> entries would be pseudo randomly droped. That would limit the used memory and
>>> still speed up seeking.
>>>
>> Another possibility (not exclusive):
>>
>> if(!url_is_streamed(s->pb)) av_add_index_entry(...)
>>
>> After all, if the input is streamed, the index is unlikely to be of any use!
>
> patch welcome

Wouldn't it be best to make whatever changes are decided on to 
av_add_index_entry() directly, to avoid code duplication adding it to the 
individual demuxers that call av_add_index_entry() themselves?

I'm currently working on a patch for the max_index_size suggestion as I 
thought it sounded like a good versatile solution. Am I correct in 
thinking the max_index_size member should be added to the AVStream struct, 
seeing there is a separate index for each stream? I notice though in 
AVFormatContext there is "int index_built;" which looks vaguely relevant 
but doesn't seem to be used by anything at all.

While looking at av_add_index_entry() in depth I realised the current size 
*is* actually limited, to UINT_MAX bytes but in general I guess you'd 
probably run out of memory before hitting that limit. To keep the same 
behaviour by default though AVStream->max_index_size could be initialised 
to UINT_MAX. Perhaps it could also be automatically set to 0 if a streamed 
source is being used??

I also need to think of a good way of picking a random index entry to be 
removed, but will look into it.

Thanks for the suggestions.

Paul