Hi On Thu, Mar 02, 2006 at 01:25:38PM +0200, Oded Shimon wrote: [...]

...

...

...

EOR frames MUST be zero-length and must be set keyframe. All streams SHOULD end with EOR, where the pts of the EOR indicates the end presentation time of the final frame. An EOR set stream is unset by the first content frames. EOR can only be unset in streams with zero decode_delay . + has_checksum must be set if the frame is larger then 2*max_distance or its

I still feel this should be a seperate variable, the only reason you gave so far against it is that poor demuxers won't be able to decide... And IMO that's a very poor argument...

every additional field adds complexity, what do we gain here with that field? personally i would fix max_distance and not store it at all, as i fear people will set it to random values

? if (frame_size > 2*nut->max_distance)

if (frame_size > sc->max_size)

I fail to see the added complexity...

As for what do we gain - more freedom to the muxer, and the ability of limiting for ex. audio to a very small size and not increasing overhead, and limiting video to slightly higher to avoid additional overhead...

agree but only if we store all these max_size/distance stuff in u(16) so they cannot be arbitrary large [...] -- Michael

Hi On Thu, Mar 02, 2006 at 12:31:16PM -0500, Rich Felker wrote:

On Thu, Mar 02, 2006 at 04:29:50PM +0100, Michael Niedermayer wrote:

...

Hi

On Thu, Mar 02, 2006 at 01:25:38PM +0200, Oded Shimon wrote: [...]

...

...

...

...

EOR frames MUST be zero-length and must be set keyframe. All streams SHOULD end with EOR, where the pts of the EOR indicates the end presentation time of the final frame. An EOR set stream is unset by the first content frames. EOR can only be unset in streams with zero decode_delay . + has_checksum must be set if the frame is larger then 2*max_distance or its

I still feel this should be a seperate variable, the only reason you gave so far against it is that poor demuxers won't be able to decide... And IMO that's a very poor argument...

every additional field adds complexity, what do we gain here with that field? personally i would fix max_distance and not store it at all, as i fear people will set it to random values

? if (frame_size > 2*nut->max_distance)

if (frame_size > sc->max_size)

I fail to see the added complexity...

As for what do we gain - more freedom to the muxer, and the ability of limiting for ex. audio to a very small size and not increasing overhead, and limiting video to slightly higher to avoid additional overhead...

agree but only if we store all these max_size/distance stuff in u(16) so they cannot be arbitrary large

NO! They really need to be arbitrarily large! If you have a file where each frame is at least 5 megs, max_distance is useless unless it's well over 5 megs!

are you drunk? if all frames are 5megs then max_distance==5mb will give you exactly the same file as any smaller max_distance, now if max_distance is set to 10mb you reduce the overhead by 1 syncpoint every 10mb, if its set to infinite you gain 2 syncpoints per 10mb and loose the ability to seek 1 syncpoint needs approximatly 8(startcode)+1(forw_ptr)+3(back_ptr)+4(pts) +4(crc) = 20byte are you seriously insisting on allowing the user to set max_distance higher then 5mb?! thats a maximum gain of 40byte per 10mb or 0.0003815% now i hope that the average windows kid will be smarter then rich and not come to the same conclusion that larger average frames somehow need larger max_distance, or otherwise we can forget the error recovering capabilities of nut in practice sorry but i wont continue these disscussions, expect me to fork if this continues, proposals should be judged on their effects (like overhead, complexity, error robustness, amount of memory or computations or delay , ...) but not on philosophical ones, nonexisting codecs, nonexisting demuxer architectures, nonexisiting kernels and so on my proposed header compression, which has negligible complexity would reduce the overhead by ~1% and was rejected based on nonexistant kernel and demuxer architectures so either 0.0003815% is significnat that my header compression will go into the spec or arbirary sized max_distance leaves it, choose, but stop to change the rules for each thing depening upon if you like it or not [...] -- Michael

On Thu, Mar 02, 2006 at 05:16:23PM -0500, Rich Felker wrote:

...

are you drunk? if all frames are 5megs then max_distance==5mb will give you exactly the same file as any smaller max_distance, now if max_distance is

It's not about size or overhead, but usefulness. Distance between syncpoints being > max_distance should be a special case, not the general case. The demuxer implementation in principle has to do more to check for validity in this case. Maybe all that can be eliminated and it's not such a big deal, but I'm still against putting particular

To clarify: if it turns out that there's no demuxer issues with max_distance being significantly smaller than all frames, I'm sorry for making an issue out of this. If that's the case maybe we can make a rule that max_distance cannot be larger than some fixed upper limit, but IMO we should still leave it vlc-coded in case there's a need to support larger max_distance in NUT 2.0 or something (way in the future).

...

my proposed header compression, which has negligible complexity would reduce the overhead by ~1% and was rejected based on nonexistant kernel and demuxer architectures

Scratch kernel; the kernel architecture for it already exists. It's in POSIX and called posix_madvise. There is no demuxer to do zerocopy demuxing, but in the case where decoded frames fit in L2 cache easily, but the compressed frame is very large (i.e. high quality, high bitrate files -- the very ones where performance is a problem) zerocopy will make a significant improvement to performance. Sacrificing this to remove 1% codec overhead in crappy codecs is not a good tradeoff IMO. It would be easier to just make "MN custom MPEG4" codec that doesn't have the wasted bytes to begin with...

One other thing with this that I forgot to mention: it would be possible to support zerocopy for non-"header-compressed" files even if header compression were supported. My reason for not wanting to have this option was that it forces any demuxer with zerocopy support to also have a duplicate demuxing system for the other case. If this can be shown not to be a problem (i.e. a trivial way to support both without significant additional code or slowdown) I'm not entirely opposed to the idea. Rich

On Fri, Mar 03, 2006 at 01:12:55AM +0100, Michael Niedermayer wrote:

Hi

On Thu, Mar 02, 2006 at 06:11:17PM -0500, Rich Felker wrote: [...]

...

...

...

my proposed header compression, which has negligible complexity would reduce the overhead by ~1% and was rejected based on nonexistant kernel and demuxer architectures

Scratch kernel; the kernel architecture for it already exists. It's in POSIX and called posix_madvise. There is no demuxer to do zerocopy demuxing, but in the case where decoded frames fit in L2 cache easily, but the compressed frame is very large (i.e. high quality, high bitrate files -- the very ones where performance is a problem) zerocopy will make a significant improvement to performance. Sacrificing this to remove 1% codec overhead in crappy codecs is not a good tradeoff IMO. It would be easier to just make "MN custom MPEG4" codec that doesn't have the wasted bytes to begin with...

One other thing with this that I forgot to mention: it would be possible to support zerocopy for non-"header-compressed" files even if header compression were supported. My reason for not wanting to have this option was that it forces any demuxer with zerocopy support to also have a duplicate demuxing system for the other case. If this can be shown not to be a problem (i.e. a trivial way to support both without significant additional code or slowdown) I'm not entirely opposed to the idea.

here are a few random problems you will have with this zero copy demuxing all solvable sure but its alot of work for very questionable gain

IMO the gain is not very questionable. Cutting out 25-50k of data that's moving through the cache per frame could make a significant difference to performance. And for rawvideo it could be even more extreme. (Naturally some filters will require alignment/aligned stride and thus copying, but direct playback should not.)

* some bitstream readers in lavc have strict alignment requirements, frames cannot be aligned with zerocopy

With a nice component system expressing alignment requirements, stride requirements, etc. for all frames and not treating decoded frames differently, this would be handled automatically. In any case, high-efficiency codecs have no word alignment (sometimes not even byte alignment?) so I doubt this is an issue for the ones that matter.

* the vlc decoding of all mpeg and h26x codecs in lavc needs a bunch of zero bytes at the end to gurantee error detection before segfaulting

:(

* several (not few) codecs write into the bitstream buffer either to fix big-little endian stuff or in at least one case reverse some lame obfuscation of a few bytes

This is probably a bad approach, for many reasons..

* having the bitstream initially not in the L2 cache (i think that would be the case if you read by dma/busmastering) will mean that accesses to the uncompressed frame and bitstream will be interleaved, todays ram is optimized for sequential access this making the already slowest part even slower

You can use prefetch instructions if needed.

* and yeah the whole buffer management with zerocopy will be a nightmare especially for a generic codec-muxer architecture where codec and muxer could run with a delay or on different threads

There is no buffer management on a 64bit system. You just mmap the whole file. For 32bit you'll have to lock things and update the map when you hit the address space limit.

basically my oppinion on this is that its like the video filter architecture very strict idealistic goals which may or may not be all achievable at the same time but which almost certainly will never be implemented as the code is to complex and too many things depend on too many

IMO it's easy to implement (easier than an efficient onecopy system) -- it's just a single mmap. The strange (mis)behavior by various codecs is problematic, but it could possibly be solved too. BTW even if the source is not mmapped, readonly memory, there are still optimizations to be made by the demuxer exporting the same memory that was passed into it, the same as MPI_EXPORT stuff in mplayer. Whether we'll actually do any of this in the near future is of course doubtful, but in the long term it should be possible, especially as codecs get more and more performance-intensive and we need to work harder and harder to squeeze out maximum performance. I don't want to preclude this with NUT. However, again, like I said, if you believe that it would be possible to support both ways without a performance/complexity penalty over the zerocopy-only implementation, I'm willing to reconsider your 'header compression' idea. Rich

On Fri, Mar 03, 2006 at 03:09:15PM +0100, Michael Niedermayer wrote:

...

...

here are a few random problems you will have with this zero copy demuxing all solvable sure but its alot of work for very questionable gain

IMO the gain is not very questionable. Cutting out 25-50k of data

rich your oppinion on how much gain something has is about as much correlated with reality as (sign(gain + TINY_VAL*random()) * HUGE_VAL) :)

so i surely agree that there will be a gain in some cases, maybe most cases but i dont agree at all about its magnitude, IMHO its <1% which is not enough for the huge rewrite the world crussade for me

Critics say the same thing about my libc replacement, and then when I actually test, memory usage by typical apps drops by 75-90% and performance increases one-thousand-fold for some simple C functions. The glibc-lovers of course still won't shut up after the testing. They'll claim that 100k per process is "small" and does not matter even when it's 50-75% of the memory used, and that everyone should be using a 2GHz machine bought with ten-years'-wages if they want to be able to write in their own language.

not to mention the significantly higher complexity of the resulting architecture

In some ways the architecture is simpler than what we have now, which is full of hacks. In any case, a new architecture is NECESSARY for h264, since right now we're destroying performance by blitting a frame up to 5 or more frames after it was decoded, totally destroying the cache... :(

...

that's moving through the cache per frame could make a significant difference to performance. And for rawvideo it could be even more extreme. (Naturally some filters will require alignment/aligned stride and thus copying, but direct playback should not.)

iam still in favor of fread() into the hw video buffer for rawvideo ...

I hope you mean read(). fread() will inherently be very slow.

not to mention that rawvideo is a irrelevant and rare case where a few percent speed wont matter, if i seriously need fast rawvideo playback id write a small special propose player for it not rewrite a generic multimedia architecture to be able to handle it better

A good generic architecture will already support this as a consequence of other things it needs to support for performance.

current lavc will segfault with almost all codecs an some cpus if you feed unaligned buffers into it, this can be fixed in lavc for most relatvely easily but it nicely shows how many people do such weird things, IMHO the whole zerocopy thing is idiotic, its like the singlethread player is always supperior rule, thers no question that fewer copies, fewer threads and less synchronization between threads is better, but its not like that could be changed in isolation, other things depend on it and the 1% you gain here might cause a 50% loss somewhre else

Perhaps you'd like to demonstrate that mplayer is only 1% faster than the competition? Last I checked it was more like 10-200%, depending on which other player you're comparing to. Naturally this is not a result of being non-threaded by itself. It involves many factors, which include a reduction in the number of wasteful copies, lack of need for thread synchronization, and many other things I have no idea about. But you know as well as anyone else in ffmpeg development that many small things add up to huge performance advantage over the competition.

...

...

* having the bitstream initially not in the L2 cache (i think that would be the case if you read by dma/busmastering) will mean that accesses to the uncompressed frame and bitstream will be interleaved, todays ram is optimized for sequential access this making the already slowest part even slower

You can use prefetch instructions if needed.

wont help, and wont work (i tried this when playing with memcpy), one thing which would work is to do a dummy read pass over the bitstream buffer to force it into the cache, the difference to copying it into another spot then would be quite negligible, the code is limited by the mem speed, the writes wouldnt cost anything, only thing you loose is a little cache trashing, if that has any significance in practice is doubtfull IMO

IMO it can easily be tested. Just write 20-40k of random crap to some unused memory buffer while decoding a video that barely fits in cache and watch the change in performance.

...

...

* and yeah the whole buffer management with zerocopy will be a nightmare especially for a generic codec-muxer architecture where codec and muxer could run with a delay or on different threads

There is no buffer management on a 64bit system. You just mmap the whole file. For 32bit you'll have to lock things and update the map when you hit the address space limit.

you cant just update the map when you hit the end, some packets might still be in various buffers/queues, maybe a buffer in a muxer maybe a decoder, ...

All you need is a pointer keeping track of the earliest point in the stream still needed. You don't unmap the whole map, just the part before this point. It's a basic variable-size circular buffer implementation (which I was planning to support in a very generic way in a next-gen player), except with munmap/mmap instead of realloc. IIRC it's even possible to share this between threads/processes without any additional locking mechanisms.

then there are non interleaved files and seeking in which cases a pure mmap variant on 32bit seems problematic

No, non-interleaved case is easy. You simply treat it the same as -audiofile, i.e. open the file twice and treat the audio and video parts separately. This is needed anyway to make -cache work. The special-casing for non-interleaved AVI in MPlayer is a stupid hack.

but dont hesitate to implement it, after it exists, works and has been benchmarked and is faster i will happily demonstrate how header compression can be done without any speedloss i mean if we are already rewriting the whole demuxer architecture, fix 10 differnt "issues" in lavc whats the big problem with passing 2 bitstream buffes instead of one into the decoder? the first would be just the startcode and or header, so only the header parsing would need to use a slower bitstream reader ...

The intent was not to modify the codecs with hacks to support this specially, but to find a way to make sure 'onecopy on top of zerocopy' is as fast as ordinary demuxing with a single copy. The problem is that if a demuxer implements zerocopy, its input buffer might not actually be the filesystem cache buffer, depending on the player's implementation. It may have already been copied once. This does not incur a performance penalty if the demuxer guarantees that it will always output the same buffer that was given to it, but it is a problem if the demuxer might copy it into another third buffer. Maybe it's acceptable just to have the additional overhead in the demuxer for treating the two cases separately. Anyway, I don't want to argue and flame over this. If you really want the header compression, please either come up with a good solution or just say that you insist on deferring that question until someone implements zerocopy. If the latter is the case, then go ahead and do it. It's ok with me. Just please don't bash and flame the whole zerocopy concept which is not the subject at hand. Rich

Hi On Thu, Mar 02, 2006 at 05:16:23PM -0500, Rich Felker wrote: [...]

...

sorry but i wont continue these disscussions, expect me to fork if this continues, proposals should be judged on their effects (like overhead, complexity, error robustness, amount of memory or computations or delay , ...) but not on philosophical ones, nonexisting codecs, nonexisting demuxer architectures, nonexisiting kernels and so on

Michael, please have some patience. When you spring a bunch of new things on us all the sudden, there will be resistence, just like when our roles were reversed with the per-stream back_ptr/pts stuff. Forking and ignoring the concerns of the other people involved is possible but it does much less to improve the overall design. At that time I was adamantly against calling a vote (even though I probably could have 'won' the vote, as you said) or doing other things to polarize the situation. This has always been about making NUT as good as possible, not about people's personal egos, and where my ideas have turned out to be bad I've abandoned them. I hope we can reasonably discuss the remaining issues you've raised and reach a consensus on what the best design is rather than flaming and forking and such.

you ask someone else to stop flaming? ;)

...

my proposed header compression, which has negligible complexity would reduce the overhead by ~1% and was rejected based on nonexistant kernel and demuxer architectures

Scratch kernel; the kernel architecture for it already exists. It's in POSIX and called posix_madvise. There is no demuxer to do zerocopy demuxing, but in the case where decoded frames fit in L2 cache easily, but the compressed frame is very large (i.e. high quality, high bitrate files -- the very ones where performance is a problem) zerocopy will make a significant improvement to performance. Sacrificing this to remove 1% codec overhead in crappy codecs is not a good tradeoff IMO. It would be easier to just make "MN custom MPEG4" codec that doesn't have the wasted bytes to begin with...

this reminds me of my little mpeg1 experiment, replacing the entropy coder with an ac coder gave ~10% lower bitrate ...

...

so either 0.0003815% is significnat that my header compression will go into the spec or arbirary sized max_distance leaves it, choose, but stop to change the rules for each thing depening upon if you like it or not

My objection to upper bound on max_distance had nothing to do with size. I'm sorry I wasn't clear.

and what is the terrible thing which will/could happen if there where an upper limit? i mean from the user or developer perspecitve (complexity, speed, overhead, ...) not the idealist/philosopher perspective (its wrong, bad design, this is like <infamous container> does it, it will make <extreemly rare use case> slightly slower, ...) [...] -- Michael

Hi On Fri, Mar 03, 2006 at 01:11:50PM +0200, Oded Shimon wrote:

On Fri, Mar 03, 2006 at 12:30:22AM +0100, Michael Niedermayer wrote:

...

On Thu, Mar 02, 2006 at 05:16:23PM -0500, Rich Felker wrote:

...

My objection to upper bound on max_distance had nothing to do with size. I'm sorry I wasn't clear.

and what is the terrible thing which will/could happen if there where an upper limit? i mean from the user or developer perspecitve (complexity, speed, overhead, ...)

I could just as well ask - what does the user/developer GAIN from there being an upper bound?

yes you can and should ask, it makes it much harder to generate broken files

yes, if there is no upper bound you can make a broken file, but you can ALWAYS make a broken/inefficient file if you are being deliberately ignorant...

master we have 2 issues, lets call them A and B without fixing A the user can make a broken file without fixing B the user can make a broken file too now fixing A is a bad idea as the user still can ALWAYS make a broken/ inefficient file if he is deliberately ignorant (by using B) now fixing B is a bad idea as the user still can ALWAYS make a broken/ inefficient file if he is deliberately ignorant (by using A) so what do we do, yes we fix neither i like your arguments, they are amusing

...

not the idealist/philosopher perspective (its wrong, bad design, this is like <infamous container> does it, it will make <extreemly rare use case> slightly slower, ...)

To this same sentiment, I ask, what is gained from your new index system? You never replied to my rant.. Yes you did make it optional which makes it

error robustness and easier muxing for muxers which are not based on libnut

somewhat nicer, I am still annoyed by the index_ptr and the additional demuxer complexity for this index system (reallocing the syncpoint cache for each index chunk, reading several chunks...), mostly because I see

you need that for dynamic index building too if theres no index at all, and iam not against adding a syncpoint_count field which would provide you with the number of syncpoints in the whole index

absoloutely no gain. There's a pretty damn good chance that if the index is borked, the entire index is borked. not just a small piece of it. And the file still plays perfectly fine without the index... ("it will make <extreemly rare use case> slightly slower" ...)

its not slightly, and its not rare IMHO [...] -- Michael

On Fri, Mar 03, 2006 at 03:44:18PM +0100, Michael Niedermayer wrote:

...

...

absoloutely no gain. There's a pretty damn good chance that if the index is borked, the entire index is borked. not just a small piece of it. And the file still plays perfectly fine without the index... ("it will make <extreemly rare use case> slightly slower" ...)

its not slightly, and its not rare IMHO

but if you and rich prefer a single monolithic index then well lets return to that, its not the most critical issue ...

I prefer a monolithic index, but more importantly I want it to be writable in one pass without buffering. However, in order to write an index you need to have buffered all the data that goes into the index while writing the file, and if it was possible to buffer all that then it should be possible to reuse that memory (if nothing else) to store the whole nut-format index. Also, you could do a dummy-run first, writing the index to /dev/null to get its size, then once you know the size do the actual unbuffered write. This is similar to my recommended replacement for GNU libc's nonstandard asprintf: call snprintf once to get length, malloc, then snprintf again to store the string. Anyway, if there are good reasons that the broken-up index is better than monolithic, I'm happy to consider it. However, I believe the possibility of one index segment being broken and the rest being ok is extremely rare, like Oded said, and it's also (slightly) complicated for a demuxer to decide what to use the partial index for in this case anyway. Rich