[FFmpeg-devel] MPEG-PS demuxer index memory usage[PATCH]

[Michael Niedermayer]

On Mon, Jan 07, 2008 at 11:37:46AM +0100, Michel Bardiaux wrote:
> Michael Niedermayer a ?crit :
>> On Fri, Jan 04, 2008 at 05:50:01PM +0100, Michel Bardiaux wrote:
>>> Michael Niedermayer a ?crit :
>>>> On Thu, Jan 03, 2008 at 10:48:49PM +0000, Paul Kelly wrote:
>>>>> Hello
>>>>> I'm using libavformat to demux a continuous stream of MPEG2-PS data and 
>>>>> am running into the problem that memory usage steadily increases over 
>>>>> time. The problem is not observed when demuxing an MPEG2-TS stream.
>>>>>
>>>>> After a bit of digging around I discovered the problem is caused by the 
>>>>> timestamp indexing in the PS demuxer - specifically, the calls to 
>>>>> av_add_index_entry() in mpegps_read_pes_header() in libavformat/mpeg.c. 
>>>>> All I'm doing is transcoding the stream to a different output format 
>>>>> and I don't need to be able to perform seeking but there doesn't seem 
>>>>> to be any way to disable the index. (I guess the memory occupied by the 
>>>>> index isn't a problem if a fixed-size file is being demuxed, but in my 
>>>>> case I am reading data from a hardware MPEG encoder card and splitting 
>>>>> the output into separate files and the process is required to run 
>>>>> indefinitely - the index quickly grows to an unwieldy size.)
>>>>>
>>>>> As far as I can see the flag AVFMT_GENERIC_INDEX can be turned off to 
>>>>> stop indexing if generic indexing is used (perhaps that's a 
>>>>> non-standard usage though) - but is there no way to turn off the 
>>>>> indexing in the MPEG-PS demuxer?
>>>>>
>>>>> Might it be a good idea to add another flag to turn off the 
>>>>> demuxer-specific indexing, and make individual demuxers respect this? A 
>>>>> general catch-all way of disabling indexing (or specifying that seeking 
>>>>> isn't required) might be more elegant though.
>>>>>
>>>>> I can get over the immediate problem by simply commenting out the line 
>>>>> calling av_add_index_entry() in libavformat/mpeg.c, but would like to 
>>>>> help get a better solution into libavformat if I can.
>>>> Disabling it with a flag is surely interresting. But i think there are 
>>>> better
>>>> solutions.
>>>> One for example would be a max_index_size. And when thats reached index
>>>> entries would be pseudo randomly droped. That would limit the used 
>>>> memory and
>>>> still speed up seeking.
>>>>
>>> Another possibility (not exclusive):
>>>
>>> if(!url_is_streamed(s->pb)) av_add_index_entry(...)
>>>
>>> After all, if the input is streamed, the index is unlikely to be of any 
>>> use!
>> patch welcome
> Here it is if it is still welcome after all the messages exchanged this 

patch o..., wait iam not mpeg maintainer, mans?


> week-end. It does not mean the max index size should not be implemented 
> too. And IMHO, yes, the same change should be made in most containers, case 
> by case. For AVI, certainly, since streaming an AVI wont work anyway!
>
> And this issue led me to read more carefully the code of 
> mpegps_read_pes_header, and I must say it looks quite strange to me, how 
> can it work for streamed (http) MPEG-PS when it relies so much on 
> url_fseek, ftell, fskip, which should fail for a streamed URL, but the 
> return value is never checked?

there is no reason why ftell or fskip or forward seeks should fail.
Backward seeks also wont fail if they are within the buffer.

[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Many things microsoft did are stupid, but not doing something just because
microsoft did it is even more stupid. If everything ms did were stupid they
would be bankrupt already.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20080107/fc926951/attachment.pgp>