[FFmpeg-devel] MPEG-PS demuxer index memory usage

[Michael Niedermayer]

On Fri, Jan 04, 2008 at 11:12:09PM +0000, Paul Kelly wrote:
> On Fri, 4 Jan 2008, Michael Niedermayer wrote:
>> On Fri, Jan 04, 2008 at 06:11:35PM +0000, Paul Kelly wrote:
>>>> On Fri, Jan 04, 2008 at 05:50:01PM +0100, Michel Bardiaux wrote:
>>>>> Michael Niedermayer a ?crit :
>>>>>> On Thu, Jan 03, 2008 at 10:48:49PM +0000, Paul Kelly wrote:
>>> [...]
>>>>>>> Might it be a good idea to add another flag to turn off the
>>>>>>> demuxer-specific indexing, and make individual demuxers respect this? 
>>>>>>> A
>>>>>>> general catch-all way of disabling indexing (or specifying that 
>>>>>>> seeking
>>>>>>> isn't required) might be more elegant though.
>>>>>>>
>>>>>> Disabling it with a flag is surely interresting. But i think there are
>>>>>> better
>>>>>> solutions.
>>>>>> One for example would be a max_index_size. And when thats reached 
>>>>>> index
>>>>>> entries would be pseudo randomly droped. That would limit the used
>>>>>> memory and
>>>>>> still speed up seeking.
>>>>>>
>>>>> Another possibility (not exclusive):
>>>>>
>>>>> if(!url_is_streamed(s->pb)) av_add_index_entry(...)
>>>>>
>>>
>>> I'm currently working on a patch for the max_index_size suggestion as I
>>> thought it sounded like a good versatile solution. Am I correct in 
>>> thinking
>>> the max_index_size member should be added to the AVStream struct, seeing
>>> there is a separate index for each stream? I notice though in
>>> AVFormatContext there is "int index_built;" which looks vaguely relevant
>>> but doesn't seem to be used by anything at all.
>>>
>>> While looking at av_add_index_entry() in depth I realised the current 
>>> size
>>> *is* actually limited, to UINT_MAX bytes but in general I guess you'd
>>> probably run out of memory before hitting that limit. To keep the same
>>> behaviour by default though AVStream->max_index_size could be initialised
>>> to UINT_MAX. Perhaps it could also be automatically set to 0 if a 
>>> streamed
>>> source is being used??
>>>
>>> I also need to think of a good way of picking a random index entry to be
>>> removed, but will look into it.
>>
>> One example, drop every 2nd entry if the limit is reached.
>
> OK, first attempt at a patch is attached. Is the general idea OK? I'm not 
> sure where the check for !url_is_streamed() should go so I haven't included 
> it - possibly in av_new_stream()?
>
> Paul

> Index: libavformat/avformat.h
> ===================================================================
> --- libavformat/avformat.h	(revision 11408)
> +++ libavformat/avformat.h	(working copy)
> @@ -342,6 +342,7 @@
>                                      support seeking natively */
>      int nb_index_entries;
>      unsigned int index_entries_allocated_size;
> +    unsigned int max_index_size; /**< max memory to use for index when demuxing */
>  
>      int64_t nb_frames;                 ///< number of frames in this stream if known or 0
>  

/**
 * Stream structure.
 * New fields can be added to the end with minor version bumps.
                                  ^^^
 * Removal, reordering and changes to existing fields require a major
 * version bump.
 * sizeof(AVStream) must not be used outside libav*.
 */
typedef struct AVStream {

also iam slightly thinking that this would belong more to AVFormatContext
or what is the use case of having max_index_size differ between streams
and the user should be able to set it from the command line
if it were in AVFormatContext you only would have to add a single line to
the AVOption array in libavformat/utils.c


[...]
> +        for(in= 0; in < st->nb_index_entries; in+= 2)
> +            memmove(&st->index_entries[out++], &st->index_entries[in], sizeof(AVIndexEntry));

for(in=0; in<st->nb_index_entries; in+= 2)
    st->index_entries[out++]= st->index_entries[in];


[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Freedom in capitalist society always remains about the same as it was in
ancient Greek republics: Freedom for slave owners. -- Vladimir Lenin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20080105/f3e8f700/attachment.pgp>