hut... hut... is this thing on? OK, mailing list is working :) For all your anonymous enjoyment, you can get latest libnut and nututuils from: svn checkout svn://213.144.138.186/nut/trunk Make sure to read README... You'll also see all commits in this ml.. On to buisness... Spec: 1. Split index The only big problem I see is deciding where to split it, and I doubt if even libnut will support this. It's also weird in regards to index_ptr. I vote against this... If it's local storage, you probably have a big damage chunk, not a single byte one... 2. Frame repetition Well, is there anything to really do about this? I can't think of any way a flag would be useful to the demuxer or player, you can just repeat the frame as is, maybe let the codec layer deal with it. 3. convert_ts overflow Uhhh... 120 years otta be enough for everybody. 4. make 'N' implicit in frame codes (as in, no need to explicit specify it is invalid) Anything else? These are truely the last issues I'm aware of, as far as I'm concerned we can (finally) finalize spec... If anybody has anything, say it now... - ods15
[resent due to the 100l reply-to issue...] On Sat, Feb 11, 2006 at 09:41:41PM +0200, Oded Shimon wrote:
hut... hut... is this thing on? OK, mailing list is working :)
For all your anonymous enjoyment, you can get latest libnut and nututuils from: svn checkout svn://213.144.138.186/nut/trunk
Make sure to read README...
You'll also see all commits in this ml..
On to buisness... Spec:
1. Split index The only big problem I see is deciding where to split it, and I doubt if even libnut will support this. It's also weird in regards to index_ptr. I vote against this... If it's local storage, you probably have a big damage chunk, not a single byte one...
I'm against split-index. It's unnecessary complication to protect _redundant_ data. The index contains no information of it's own, it's all derived data stored in a common place for easy access. In the case of a file with corruption in the index, you can just build a new index if you really need one. IMO if just the index is damaged, you should be _glad_ that the damage was isolated to the index and either rebuild it or be happy with slightly slow seeking.
2. Frame repetition Well, is there anything to really do about this? I can't think of any way a flag would be useful to the demuxer or player, you can just repeat the frame as is, maybe let the codec layer deal with it.
I'd need to think about it some more. I really don't like frame repetition at all...
3. convert_ts overflow Uhhh... 120 years otta be enough for everybody.
So should 640kb :) Or, a more relevant example: 32bit time_t should be enough for everybody... Or, ... 2 digit years should be enough for everybody! However, this is NOT a bug in nut. The convert_ts code in the spec is _informative_, not normative, i.e. it's a recommended implementation that works for the range of numbers likely to be encountered. Implementations needing higher ts range can use different algorithms as long as the result is the same. However the spec should be changed slightly to make this clear.
4. make 'N' implicit in frame codes (as in, no need to explicit specify it is invalid)
Agree.
Anything else? These are truely the last issues I'm aware of, as far as I'm concerned we can (finally) finalize spec... If anybody has anything, say it now...
:) Rich
On Sat, Feb 11, 2006 at 11:23:09PM -0500, Rich Felker wrote:
On Sat, Feb 11, 2006 at 09:41:41PM +0200, Oded Shimon wrote:
On to buisness... Spec:
1. Split index The only big problem I see is deciding where to split it, and I doubt if even libnut will support this. It's also weird in regards to index_ptr. I vote against this... If it's local storage, you probably have a big damage chunk, not a single byte one...
I'm against split-index. It's unnecessary complication to protect _redundant_ data. The index contains no information of it's own, it's all derived data stored in a common place for easy access. In the case of a file with corruption in the index, you can just build a new index if you really need one. IMO if just the index is damaged, you should be _glad_ that the damage was isolated to the index and either rebuild it or be happy with slightly slow seeking.
My thoughts as well on the issue... Ugly complication.
4. make 'N' implicit in frame codes (as in, no need to explicit specify it is invalid)
Agree.
Patch. Can I commit... On Sun, Feb 12, 2006 at 12:51:51AM +0100, Michael Niedermayer wrote:
On Sat, Feb 11, 2006 at 09:41:41PM +0200, Oded Shimon wrote:
1. Split index The only big problem I see is deciding where to split it, and I doubt if even libnut will support this. It's also weird in regards to index_ptr. I vote against this... If it's local storage, you probably have a big damage chunk, not a single byte one...
you dont need a single byte error for this to be usefull, if you loose 10kb in one chunk of a 100kb index you really want to be able to recover as much of the 90kb as you can ...
10kb of damage without another 2mb of damage in that whole area is highly unlikely... Especially with local storage...
2. Frame repetition Well, is there anything to really do about this? I can't think of any way a flag would be useful to the demuxer or player, you can just repeat the frame as is, maybe let the codec layer deal with it.
well, i dont care strongly about that, if you are against it fine then repeated frame == (keyframe && binary identical to last), it means 1 compare pass more for the demuxer if it wants to discard them (like when storing a network stream on reliable media)
Ok, issue closed...
Anything else? These are truely the last issues I'm aware of, as far as I'm concerned we can (finally) finalize spec... If anybody has anything, say it now...
something crazy :) ive already suggested it long ago and its probably a stupid idea but it might mean big reduction of overhead and iam very curious how much space we could save with it ...
give each stream 2 byte sequences in the stream header (like codec_specific_data), one for keyframes one for non-key frames, then for each frame, put the correspong byte sequence before the frame before outputing it from the demuxer :)
It's weird.. It's the codec's job, not the container... Do we put it before ALL frames in the stream or add another stream flag (or nut flag?). Anyway, I suspect your savings will be about as much as your loss by adding checksums to syncpoints... I don't really consider it reduction in overhead, more like compression of data... I'm assuming you're refferring to silly mp3 and mpeg-4 startcodes?... - ods15
Hi On Sun, Feb 12, 2006 at 07:45:54AM +0200, Oded Shimon wrote: [...]
Anything else? These are truely the last issues I'm aware of, as far as I'm concerned we can (finally) finalize spec... If anybody has anything, say it now...
something crazy :) ive already suggested it long ago and its probably a stupid idea but it might mean big reduction of overhead and iam very curious how much space we could save with it ...
give each stream 2 byte sequences in the stream header (like codec_specific_data), one for keyframes one for non-key frames, then for each frame, put the correspong byte sequence before the frame before outputing it from the demuxer :)
It's weird.. It's the codec's job, not the container... Do we put it before ALL frames in the stream or add another stream flag (or nut flag?).
Anyway, I suspect your savings will be about as much as your loss by adding checksums to syncpoints... I don't really consider it reduction in overhead, more like compression of data... I'm assuming you're refferring to silly mp3 and mpeg-4 startcodes?...
yep, 4byte per frame for mpeg4 at least and 2 byte per mp3 frame at 30fps 44100khz thats ~1.6kbit/sec or 1.3 mb for a 2 hour movie for syncpoints its 4byte per 16384byte which for a 650mb movie is ~166kbyte the checksum per syncpoint overhead also is per filesize so lower bitrate lower overhead for the checksums, while the startcode overhead wont decrease, for example: for a 1hour 128kbit stereo mp3 in nut the syncpoint checksums need 14kbyte while the startcodes need 275kbyte [...] -- Michael
On Sun, Feb 12, 2006 at 12:52:34PM +0100, Michael Niedermayer wrote:
Hi
On Sun, Feb 12, 2006 at 07:45:54AM +0200, Oded Shimon wrote: [...]
Anything else? These are truely the last issues I'm aware of, as far as I'm concerned we can (finally) finalize spec... If anybody has anything, say it now...
something crazy :) ive already suggested it long ago and its probably a stupid idea but it might mean big reduction of overhead and iam very curious how much space we could save with it ...
give each stream 2 byte sequences in the stream header (like codec_specific_data), one for keyframes one for non-key frames, then for each frame, put the correspong byte sequence before the frame before outputing it from the demuxer :)
It's weird.. It's the codec's job, not the container... Do we put it before ALL frames in the stream or add another stream flag (or nut flag?).
Anyway, I suspect your savings will be about as much as your loss by adding checksums to syncpoints... I don't really consider it reduction in overhead, more like compression of data... I'm assuming you're refferring to silly mp3 and mpeg-4 startcodes?...
yep, 4byte per frame for mpeg4 at least and 2 byte per mp3 frame at 30fps 44100khz thats ~1.6kbit/sec or 1.3 mb for a 2 hour movie
for syncpoints its 4byte per 16384byte which for a 650mb movie is ~166kbyte
the checksum per syncpoint overhead also is per filesize so lower bitrate lower overhead for the checksums, while the startcode overhead wont decrease, for example: for a 1hour 128kbit stereo mp3 in nut the syncpoint checksums need 14kbyte while the startcodes need 275kbyte
Heh, wow, you're right. For a simple high bitrate file: before: TOTAL: 726098576 bytes data, 917174 bytes overhead, 0.13% overhead after: TOTAL: 726098576 bytes data, 187977 bytes overhead, 0.03% overhead And for the 620 low bitrate file: before: TOTAL: 623128520 bytes data, 1988245 bytes overhead, 0.32% overhead after: TOTAL: 623128520 bytes data, -1168114 bytes overhead, -0.19% overhead (negative overhead! hehe) I should note, the low bitrate file failed for me at first because some frames were zero bytes (stupid mencoder). This is not an issue because NUT is pts aware, and zero byte frames should never occur unless they really mean something to the decoder (like, black frame). Instead, they should be compensated by higher pts. Right? Anyway, the only thing I'm kind of worried about is how to put this in spec. Just have 2 fixed codes in the stream header, for keyframes and for non keyframes?... Also, do ALL frames get it, or can it be flagged off? BTW, is it considered a violation of NUT spec to put mpeg-4 startcodes inside nut frames, as spec says cannot have containers inside the NUT container? (ogg-in-avi) I guess MPEG-4 spec specifies the startcodes as literal parts of the frames for decoder, so ok... - ods15
Hi On Sun, Feb 12, 2006 at 07:37:55PM +0200, Oded Shimon wrote:
On Sun, Feb 12, 2006 at 12:52:34PM +0100, Michael Niedermayer wrote:
Hi
On Sun, Feb 12, 2006 at 07:45:54AM +0200, Oded Shimon wrote: [...]
Anything else? These are truely the last issues I'm aware of, as far as I'm concerned we can (finally) finalize spec... If anybody has anything, say it now...
something crazy :) ive already suggested it long ago and its probably a stupid idea but it might mean big reduction of overhead and iam very curious how much space we could save with it ...
give each stream 2 byte sequences in the stream header (like codec_specific_data), one for keyframes one for non-key frames, then for each frame, put the correspong byte sequence before the frame before outputing it from the demuxer :)
It's weird.. It's the codec's job, not the container... Do we put it before ALL frames in the stream or add another stream flag (or nut flag?).
Anyway, I suspect your savings will be about as much as your loss by adding checksums to syncpoints... I don't really consider it reduction in overhead, more like compression of data... I'm assuming you're refferring to silly mp3 and mpeg-4 startcodes?...
yep, 4byte per frame for mpeg4 at least and 2 byte per mp3 frame at 30fps 44100khz thats ~1.6kbit/sec or 1.3 mb for a 2 hour movie
for syncpoints its 4byte per 16384byte which for a 650mb movie is ~166kbyte
the checksum per syncpoint overhead also is per filesize so lower bitrate lower overhead for the checksums, while the startcode overhead wont decrease, for example: for a 1hour 128kbit stereo mp3 in nut the syncpoint checksums need 14kbyte while the startcodes need 275kbyte
Heh, wow, you're right. For a simple high bitrate file:
before: TOTAL: 726098576 bytes data, 917174 bytes overhead, 0.13% overhead after: TOTAL: 726098576 bytes data, 187977 bytes overhead, 0.03% overhead
And for the 620 low bitrate file:
before: TOTAL: 623128520 bytes data, 1988245 bytes overhead, 0.32% overhead after: TOTAL: 623128520 bytes data, -1168114 bytes overhead, -0.19% overhead
(negative overhead! hehe)
I should note, the low bitrate file failed for me at first because some frames were zero bytes (stupid mencoder). This is not an issue because NUT is pts aware, and zero byte frames should never occur unless they really mean something to the decoder (like, black frame). Instead, they should be compensated by higher pts. Right?
yes, AFAIK 0-byte - skip frames are a feature of AVI only
Anyway, the only thing I'm kind of worried about is how to put this in spec. Just have 2 fixed codes in the stream header, for keyframes and for non keyframes?... Also, do ALL frames get it, or can it be flagged off?
dunno, but we might even add it into the framecode table so each framecode could have a different "prefix" * no flag needed * fully optional on the muxer side * still as easy for the demuxer as if its in the stream header * theoretically lower overhead / more flexible
BTW, is it considered a violation of NUT spec to put mpeg-4 startcodes inside nut frames, as spec says cannot have containers inside the NUT container? (ogg-in-avi)
I guess MPEG-4 spec specifies the startcodes as literal parts of the frames for decoder, so ok...
yes, startcodes are ok, just think about keyframes, there are several startcodes and headers in a frame or quantization tables or several slices per frame with startcodes infront of them (mpeg1/2 case) [...] -- Michael
On Sun, Feb 12, 2006 at 07:04:31PM +0100, Michael Niedermayer wrote:
On Sun, Feb 12, 2006 at 07:37:55PM +0200, Oded Shimon wrote:
Anyway, the only thing I'm kind of worried about is how to put this in spec. Just have 2 fixed codes in the stream header, for keyframes and for non keyframes?... Also, do ALL frames get it, or can it be flagged off?
dunno, but we might even add it into the framecode table so each framecode could have a different "prefix" * no flag needed * fully optional on the muxer side * still as easy for the demuxer as if its in the stream header * theoretically lower overhead / more flexible
Yeah, I was thinking same thing, only reason I wanted to avoid it is that it's just slightly unfriendly to C - dynamic array for any possible prefixes in stream header.. But that's just a slight implementation issue. Ok, here's my proposed patch... Note that data_size does NOT include the length of prefix, because it would make all stream headers essential to demux nut file. - ods15
On Sun, Feb 12, 2006 at 07:45:54AM +0200, Oded Shimon wrote:
On Sat, Feb 11, 2006 at 11:23:09PM -0500, Rich Felker wrote:
On Sat, Feb 11, 2006 at 09:41:41PM +0200, Oded Shimon wrote:
4. make 'N' implicit in frame codes (as in, no need to explicit specify it is invalid)
Agree.
Patch. Can I commit...
Committed - ods15
Oded Shimon wrote:
On Sun, Feb 12, 2006 at 07:45:54AM +0200, Oded Shimon wrote:
On Sat, Feb 11, 2006 at 11:23:09PM -0500, Rich Felker wrote:
On Sat, Feb 11, 2006 at 09:41:41PM +0200, Oded Shimon wrote:
4. make 'N' implicit in frame codes (as in, no need to explicit specify it is invalid)
Agree.
Patch. Can I commit...
Committed
I don't understand this change. Why did you do it? Clarification on this matter would be welcome.. Alex (beastd)
On Wed, Feb 15, 2006 at 08:32:12PM +0100, Alexander Strasser wrote:
Oded Shimon wrote:
On Sun, Feb 12, 2006 at 07:45:54AM +0200, Oded Shimon wrote:
On Sat, Feb 11, 2006 at 11:23:09PM -0500, Rich Felker wrote:
On Sat, Feb 11, 2006 at 09:41:41PM +0200, Oded Shimon wrote:
4. make 'N' implicit in frame codes (as in, no need to explicit specify it is invalid)
Agree.
Patch. Can I commit...
Committed
I don't understand this change. Why did you do it? Clarification on this matter would be welcome..
It HAS to be invalid, requiring that it be said explicitly just wastes (little) space in the main header and is redundant... Making it implicit is better... - ods15
Oded Shimon wrote:
On Wed, Feb 15, 2006 at 08:32:12PM +0100, Alexander Strasser wrote:
Oded Shimon wrote:
On Sun, Feb 12, 2006 at 07:45:54AM +0200, Oded Shimon wrote:
On Sat, Feb 11, 2006 at 11:23:09PM -0500, Rich Felker wrote:
On Sat, Feb 11, 2006 at 09:41:41PM +0200, Oded Shimon wrote:
4. make 'N' implicit in frame codes (as in, no need to explicit specify it is invalid)
Agree.
Patch. Can I commit...
Committed
I don't understand this change. Why did you do it? Clarification on this matter would be welcome..
It HAS to be invalid, requiring that it be said explicitly just wastes (little) space in the main header and is redundant...
Making it implicit is better...
Ok, if i think about it again it seems better. Alex (beastd)
participants (4)
-
Alexander Strasser -
Michael Niedermayer -
Oded Shimon -
Rich Felker