- NUT-devel - lists.mplayerhq.hu

back_ptr_div8 -> 16
by Michael Niedermayer 10 Mar '06

10 Mar '06

Hi current syncpoints have 8(startcode)+1(forward_ptr)+1(back_ptr)+1(timestamp)+4(crc) = 15 so if we dissallow 2 consecutive syncpoints then we could drop 1 bit of the back ptr and still uniquely point to each comments? -- Michael

2 1

huge vs. damaged forward_ptrs in packets
by Michael Niedermayer 09 Mar '06

09 Mar '06

Hi $subj, and what about a change like: --- mpcf.txt 2006-03-08 13:36:02.000000000 +0100 +++ mncf.txt 2006-03-05 14:49:57.000000000 +0100 @@ -312,6 +306,8 @@ packet_header startcode f(64) forward_ptr v + if(forward_ptr>64k) + header_checksum u(32) packet_footer reserved_bytes @@ -591,6 +576,9 @@ for frame headers the checksum contains the framecode byte and all following bytes upto the checksum itself +header_checksum + checksum over the startcode and forward pointer + Syncpoint tags: --------------- -- Michael

3 11

Issues
by Oded Shimon 08 Mar '06

08 Mar '06

The controversies so far... 1. checksum in frame headers/syncpoints I did very much like the elegance with the syncpoints, however I did not like the hacks in code to accomplish this (frame header parser in find_syncpoint, put_frame using put_syncpoint to a temporary buffer, demuxing frame header is dependant of "saw syncpoint" context). So I have no strong feelings either way with this. I do have a small proposal for overhead reduction in new method (if it's stupid say so and I'll drop it) - has_checksum being implicit by the rules it is manditory for - so you can save frame codes or a byte for coded flags... Come to think of it, the frame header for a frame that needs has_checksum is large already anyway... 2. limiting max_distance, max_frame_size I say, keep them VLC, but add a MUST in spec regarding their value... Using 'u(16)' is as silly as using it for the 'tmp_mul' vars in the main header which also have a MUST restriction... 3. prefix/compression for frames Not sure if this is an actual controversy, possibly the flamewar going on has nothing to do with this (let's keep mplayer g5926 flames for another mailing list :). I'm quite against the prefix stuff, for a completely other reason - it's a silly complexity for something that does not belong in the container. If you're gonna do this, you might as well compress the frames with bzip2 while you're at it. underhead is nice, but not that nice, this will be nothing but a legacy feature in the future... 4. Info frames/streams/packets The first problem: limiting positions where info frames/packets should be Michael you said you're strongly against this, but you also said a muxer should place them sanely and the demuxer should not go crazy looking for them. IMO there should be rules that you can't just put the frames/packets all over the place as you please, and I'd like to find the best ones that stay fully featured while still allowing the demuxer to not have to search everywhere for them. Second problem: dump info streams? There is some advantage to this - info frames don't really have stream like behavior, unless you repeat them often. A NUT-aware streaming server could give the info frame relavent to current position just once, and not be wasteful and repeat often or at all... Info streams can work if you put an EOR closely after the frame, but that is not semantically correct... What I hate about dumping info streams though is some complication to my demuxer api (each call can give a frame or a packet, instead of always a frame), and, if we do dump info streams, I very much want a restriction that info packets that are global (chapter_id >=0 ?) can only come after main headers, and not wherever... Thinking again about my streaming example, the file isn't seekable anyway, and if you dump it, and you're still using the info frames instead of remuxing them to info packets, you deserve what you get of an unseekable/bad seekable file... (Which would nicely discourage using info frames in places where it is inappropiatte - not live streams..) - ods15

6 24

Suggestions [PATCH]
by Michael Niedermayer 05 Mar '06

05 Mar '06

Hi the Attached patch does the following: 1. cleanup packets so all have forward_ptr significantly improves extendability and simplifies stuff slightly larger overhead (~0.006% which is negligible) 2. split checksum into frame and syncpoint checksum simplification overhead changes by + 4byte for every packet >64kb (<0.006% worst case) overhead changes by - syncpoint size + 4 for every packet with dts_diff > 1sec overall this could even reduce the overhead, the worst case increase is also negligible 3. split index simplification for lavf muxer currently we just write the data out seek back and update the forward pointer, this is safe as long as the packet is < the io buffer size in lavf, if its larger we need to alloc&realloc a buffer and work within that or do 2 passes to find the forward ptr value, both are more messy index_ptr becomes unneccesary by this improves error resistance 4. always store chapter_start/len simplification 5. change file structure definion so extendability is restored 6. drop info streams i see no sense in them ... 7. dissalow width or height == 0 8. add leading coeff to the crc polynom -- Michael

3 19

[NUT-devel] Re: [MPlayer-cvslog] CVS: main/DOCS/tech mpcf.txt, 1.121, 1.122
by Oded Shimon 04 Mar '06

04 Mar '06

On Fri, Mar 03, 2006 at 11:33:33PM +0100, Michael Niedermayer CVS wrote: > CVS change done by Michael Niedermayer CVS > > Update of /cvsroot/mplayer/main/DOCS/tech > In directory mail:/var2/tmp/cvs-serv7350 > > Modified Files: > mpcf.txt > Log Message: > the return of the monolithic index May I further add... - ods15

2 1

[NUT] (ods15): r85 - /trunk/mplayer.diff
by syncmail＠mplayerhq.hu 04 Mar '06

04 Mar '06

Author: ods15 Date: Sat Mar 4 16:50:53 2006 New Revision: 85 Log: up to date patch... Modified: trunk/mplayer.diff Modified: trunk/mplayer.diff ============================================================================== --- trunk/mplayer.diff (original) +++ trunk/mplayer.diff Sat Mar 4 16:50:53 2006 @@ -1,26 +1,26 @@ Index: Makefile =================================================================== RCS file: /cvsroot/mplayer/main/Makefile,v -retrieving revision 1.330 -diff -u -r1.330 Makefile ---- Makefile 19 Aug 2005 19:24:30 -0000 1.330 -+++ Makefile 13 Sep 2005 19:28:19 -0000 -@@ -109,6 +109,7 @@ - $(W32_LIB) \ - $(DS_LIB) \ - libaf/libaf.a \ -+ ../nut/libnut/demuxer.o \ - libmpdemux/libmpdemux.a \ - input/libinput.a \ - postproc/libswscale.a \ +retrieving revision 1.341 +diff -u -r1.341 Makefile +--- Makefile 16 Feb 2006 20:45:24 -0000 1.341 ++++ Makefile 4 Mar 2006 15:46:04 -0000 +@@ -42,6 +42,7 @@ + sub_cc.c \ + subreader.c \ + vobsub.c \ ++ ../nut/libnut/demuxer.c \ + + SRCS_MENCODER = mencoder.c \ + mp_msg-mencoder.c \ Index: libmpdemux/Makefile =================================================================== RCS file: /cvsroot/mplayer/main/libmpdemux/Makefile,v -retrieving revision 1.96 -diff -u -r1.96 Makefile ---- libmpdemux/Makefile 1 Aug 2005 20:16:56 -0000 1.96 -+++ libmpdemux/Makefile 13 Sep 2005 19:28:42 -0000 -@@ -72,6 +72,7 @@ +retrieving revision 1.101 +diff -u -r1.101 Makefile +--- libmpdemux/Makefile 6 Feb 2006 16:04:17 -0000 1.101 ++++ libmpdemux/Makefile 4 Mar 2006 15:46:06 -0000 +@@ -70,6 +70,7 @@ demux_mpc.c \ demux_mpg.c \ demux_nsv.c \ @@ -28,7 +28,7 @@ demux_nuv.c \ demux_ogg.c \ demux_pva.c \ -@@ -148,7 +149,7 @@ +@@ -158,7 +159,7 @@ OBJS = $(SRCS:.c=.o) OBJS += $(CPLUSPLUSSRCS:.cpp=.o) @@ -40,19 +40,19 @@ Index: libmpdemux/demuxer.c =================================================================== RCS file: /cvsroot/mplayer/main/libmpdemux/demuxer.c,v -retrieving revision 1.199 -diff -u -r1.199 demuxer.c ---- libmpdemux/demuxer.c 26 Aug 2005 17:40:01 -0000 1.199 -+++ libmpdemux/demuxer.c 13 Sep 2005 19:28:53 -0000 -@@ -61,6 +61,7 @@ - extern demuxer_desc_t demuxer_desc_rtp; +retrieving revision 1.217 +diff -u -r1.217 demuxer.c +--- libmpdemux/demuxer.c 17 Feb 2006 01:57:41 -0000 1.217 ++++ libmpdemux/demuxer.c 4 Mar 2006 15:46:06 -0000 +@@ -66,6 +66,7 @@ extern demuxer_desc_t demuxer_desc_lavf; + #endif extern demuxer_desc_t demuxer_desc_aac; +extern demuxer_desc_t demuxer_desc_nut; demuxer_desc_t* demuxer_list[] = { &demuxer_desc_rawaudio, -@@ -119,6 +120,7 @@ +@@ -125,6 +126,7 @@ &demuxer_desc_lavf, #endif &demuxer_desc_aac, @@ -63,10 +63,10 @@ Index: libmpdemux/demuxer.h =================================================================== RCS file: /cvsroot/mplayer/main/libmpdemux/demuxer.h,v -retrieving revision 1.80 -diff -u -r1.80 demuxer.h ---- libmpdemux/demuxer.h 26 Aug 2005 17:40:02 -0000 1.80 -+++ libmpdemux/demuxer.h 13 Sep 2005 19:28:54 -0000 +retrieving revision 1.91 +diff -u -r1.91 demuxer.h +--- libmpdemux/demuxer.h 17 Feb 2006 01:57:41 -0000 1.91 ++++ libmpdemux/demuxer.h 4 Mar 2006 15:46:07 -0000 @@ -50,11 +50,12 @@ #define DEMUXER_TYPE_MPC 40 #define DEMUXER_TYPE_MPEG_PES 41

1 0

CVS: main/DOCS/tech mpcf.txt,1.117,1.118
by Oded Shimon 03 Mar '06

03 Mar '06

On Wed, Mar 01, 2006 at 03:19:40PM +0100, Michael Niedermayer CVS wrote: > CVS change done by Michael Niedermayer CVS > > Update of /cvsroot/mplayer/main/DOCS/tech > In directory mail:/var2/tmp/cvs-serv10365 > > Modified Files: > mpcf.txt > Log Message: > add forward_ptr to syncpoint (+0.006% overhead) > give syncpoint and frameheader their own checksums (worst case overhead increase <0.006%) > fix filestructure so that extendability is restored > move index_ptr to the fileend so that index packets arent a special case with their reserved_bytes position > -> all packets follow the same structure now > > remove "optional" word from info packets, they are not more optional then index packets > > split index packets > note, this is entirely optional and a muxer which has difficulty with it can always output a single index packet > > remove the index MUST be at the file end if anywher rule, its not needed anymore as index_ptr will always be at the end > > info frames must be keyframes > > last info frame is the most correct > > comments, flames? I'm not strongly against anything here, but it would've been better if you would have showed the patch (and resolved conflicts) before committing... :/ > index: > - index_startcode f(64) > - forward_ptr v > max_pts v > + syncpoint_start v Is syncpoint_start necessary? Seems redundant to me. BTW, it is non obvious from this spec that the index is split by syncpoints and how to use, it should be elaborated better (by an entry for syncpoint_start perhaps..) > +file: > + file_id_string > + while(bytes_left > 8){ I'm a bit weirded out by this, In a very extreme and silly example, in a truncated NUT file you could loose the last frame because it (and the frame header) was smaller than 8 bytes... > @@ -474,12 +508,15 @@ > 1 is_key if set, frame is keyframe > 2 end_of_relevance if set, stream has no relevance on > presentation. (EOR) > + 4 has_checksum if set then the frame header contains a checksum Like I said, this should be a NUT flag... > EOR frames MUST be zero-length and must be set keyframe. > All streams SHOULD end with EOR, where the pts of the EOR indicates the > end presentation time of the final frame. > An EOR set stream is unset by the first content frames. > EOR can only be unset in streams with zero decode_delay . > + has_checksum must be set if the frame is larger then 2*max_distance or its I still feel this should be a seperate variable, the only reason you gave so far against it is that poor demuxers won't be able to decide... And IMO that's a very poor argument... > @@ -612,11 +645,8 @@ > that EOR. EOR is unset by the first keyframe after it. > > index_ptr > - Length in bytes of the entire index, from the first byte of the > - startcode until the last byte of the checksum. > - Note: A demuxer can use this to find the index when it is written at > - EOF, as index_ptr will always be 12 bytes before the end of file if > - there is an index at all. > + absolute location in the file of the first byte of the startcode of the > + first index packet, or 0 if there is no index This would be a silly argument, but it does limit the filesize to 64 bits... Doesn't matter, it's not very different from limiting the index to 64 bits... > Info tags: > ---------- > @@ -630,6 +660,8 @@ > file. Positive chapter_id's are real chapters and MUST NOT overlap. > Negative chapter_id indicate a sub region of file and not a real > chapter. chapter_id MUST be unique to the region it represents. > + chapter_id n MUST not be used unless there are at least n chapters in the > + file Could you explain this, I'm confused... - ods15

3 19

flags cleanup
by Michael Niedermayer 03 Mar '06

03 Mar '06

Hi $subj, patch attached -- Michael

2 3

[NUT] (ods15): r84 - /trunk/libnut/demuxer.c
by syncmail＠mplayerhq.hu 01 Mar '06

01 Mar '06

Author: ods15 Date: Wed Mar 1 23:57:11 2006 New Revision: 84 Log: some cosmetics Modified: trunk/libnut/demuxer.c Modified: trunk/libnut/demuxer.c ============================================================================== --- trunk/libnut/demuxer.c (original) +++ trunk/libnut/demuxer.c Wed Mar 1 23:57:11 2006 @@ -105,7 +105,7 @@ int i; if (ready_read_buf(bc, count) < count) return buf_eof(bc); *val = 0; - for(i = 0; i < count; i++){ + for (i = 0; i < count; i++) { *val = (*val << 8) | *(bc->buf_ptr++); } return 0; @@ -116,7 +116,7 @@ *val = 0; do { len = ready_read_buf(bc, 16); - for(i = 0; i < len; i++){ + for (i = 0; i < len; i++) { uint8_t tmp= *(bc->buf_ptr++); *val = (*val << 7) | (tmp & 0x7F); if (!(tmp & 0x80)) return 0; @@ -157,7 +157,7 @@ #define get_s_(bc, var, name) get_s(bc, var) #endif -#define CHECK(expr) do { int _a; if ((_a = (expr))) { err = _a; goto err_out; } } while(0) +#define CHECK(expr) do { if ((err = (expr))) goto err_out; } while(0) #define ERROR(expr, code) do { if (expr) { err = code; goto err_out; } } while(0) #define GET_V(bc, v) do { uint64_t _tmp; CHECK(get_v_((bc), &_tmp, #v)); (v) = _tmp; } while(0) #define GET_S(bc, v) do { int64_t _tmp; CHECK(get_s_((bc), &_tmp, #v)); (v) = _tmp; } while(0) @@ -569,7 +569,7 @@ ERROR(compare_ts(nut, pd->pts, pd->stream, nut->sc[i].last_dts, i) < 0, -ERR_OUT_OF_ORDER); } - if (saw_syncpoint) *saw_syncpoint = after_sync; + if (saw_syncpoint) *saw_syncpoint = !!after_sync; err_out: return err; } @@ -737,12 +737,8 @@ off_t pos = bctello(nut->i); uint64_t tmp; CHECK(get_bytes(nut->i, 8, &tmp)); - if (tmp == INDEX_STARTCODE) { - GET_V(nut->i, tmp); - nut->before_seek = bctello(nut->i) + tmp; - nut->seek_status = 2; - } - nut->i->buf_ptr -= bctello(nut->i) - pos; + if (tmp == INDEX_STARTCODE) nut->seek_status = 2; + nut->i->buf_ptr -= 8; flush_buf(nut->i); } nut->last_headers = 1; @@ -768,7 +764,7 @@ err = 0; } nut->seek_status = 0; - seek_buf(nut->i, nut->before_seek, SEEK_SET); + if (nut->before_seek) seek_buf(nut->i, nut->before_seek, SEEK_SET); nut->before_seek = 0; } *s = malloc(sizeof(nut_stream_header_t) * (nut->stream_count + 1));

1 0

Re: FINALIZE NUT SPEC!!
by Rich Felker 28 Feb '06

28 Feb '06

Moving to nut-devel list... On Mon, Feb 27, 2006 at 08:08:55PM +0100, Michael Niedermayer wrote: > yes, but that realtime data could very well become available afterwards > lets say some realtime news of a earthquake, its magnitude, number of > casualties and so on wont be known immedeatly > furthermore estimates will change, and if you dont like that example, take > any other realtime stream (some DJ with music videos whatever) where info > is added in realtime some mistakes will creep in and it should be possible > to correct them, in such a way that automatic remuxing updates all info > thats not hard but with our strict rules its impossible. so with these > restrictions i see no sense in info streams OK, how about this: drop the requirement that they be binary-identical, and instead the last packet with the given chapter_id is the one that's valid. Keep the rest of the semantics for what range it applies to. This allows updating/correcting info in a live stream and will give the correct results if you convert it to an info packet for permenant storage (using the latest/most correct data). Also this allows you to specify length if it becomes known. > > Also I don't like supporting this for arbitrary codecs. The reason is > > that you have to do one of two things in the demuxer, and both are > > wrong for certain cases. You must either: > > > > 1. Output the whole NUT packet as one unit. This is wrong for codecs > > that actually use frames. > > > > 2. Split the NUT packet into frames according to packet_size and > > output each individually. This is horribly wrong for PCM as it will > > make CPU usage insanely high. > > > > IMO special-casing PCM/ADPCM as codecs where any block of data up to X > > samples long is a valid 'frame' is a cleaner solution, but I'm open to > > alternatives for sure. > > what about allowing it only for <4kbyte && <4ksamples ? It still doesn't solve the problem about how do demux. How about this instead: - Storing multiple frames together is allowed only if the concatenation of all frames is also a valid single frame for the format. This way the demuxer can always output the whole frame as a unit and doesn't have to decide whether that's safe or not. But, having the packet_size (should be called frame_size...?) in the header is still useful, since it would allow more fine-grained seeking in a special-purpose application like a media editor. We can also put a limit on number of samples. IMO there's no real need for a size limit, and if there is a size limit it should probably be relative to max_distance rather than absolute. If we really want to avoid absolutes, we could make the sample limit relative to max_pts_diff too... Rich

3 5