February 2006 - NUT-devel - lists.mplayerhq.hu

Suggestions [PATCH]
by Michael Niedermayer 05 Mar '06

05 Mar '06

Hi the Attached patch does the following: 1. cleanup packets so all have forward_ptr significantly improves extendability and simplifies stuff slightly larger overhead (~0.006% which is negligible) 2. split checksum into frame and syncpoint checksum simplification overhead changes by + 4byte for every packet >64kb (<0.006% worst case) overhead changes by - syncpoint size + 4 for every packet with dts_diff > 1sec overall this could even reduce the overhead, the worst case increase is also negligible 3. split index simplification for lavf muxer currently we just write the data out seek back and update the forward pointer, this is safe as long as the packet is < the io buffer size in lavf, if its larger we need to alloc&realloc a buffer and work within that or do 2 passes to find the forward ptr value, both are more messy index_ptr becomes unneccesary by this improves error resistance 4. always store chapter_start/len simplification 5. change file structure definion so extendability is restored 6. drop info streams i see no sense in them ... 7. dissalow width or height == 0 8. add leading coeff to the crc polynom -- Michael

3 19

Re: FINALIZE NUT SPEC!!
by Rich Felker 28 Feb '06

28 Feb '06

Moving to nut-devel list... On Mon, Feb 27, 2006 at 08:08:55PM +0100, Michael Niedermayer wrote: > yes, but that realtime data could very well become available afterwards > lets say some realtime news of a earthquake, its magnitude, number of > casualties and so on wont be known immedeatly > furthermore estimates will change, and if you dont like that example, take > any other realtime stream (some DJ with music videos whatever) where info > is added in realtime some mistakes will creep in and it should be possible > to correct them, in such a way that automatic remuxing updates all info > thats not hard but with our strict rules its impossible. so with these > restrictions i see no sense in info streams OK, how about this: drop the requirement that they be binary-identical, and instead the last packet with the given chapter_id is the one that's valid. Keep the rest of the semantics for what range it applies to. This allows updating/correcting info in a live stream and will give the correct results if you convert it to an info packet for permenant storage (using the latest/most correct data). Also this allows you to specify length if it becomes known. > > Also I don't like supporting this for arbitrary codecs. The reason is > > that you have to do one of two things in the demuxer, and both are > > wrong for certain cases. You must either: > > > > 1. Output the whole NUT packet as one unit. This is wrong for codecs > > that actually use frames. > > > > 2. Split the NUT packet into frames according to packet_size and > > output each individually. This is horribly wrong for PCM as it will > > make CPU usage insanely high. > > > > IMO special-casing PCM/ADPCM as codecs where any block of data up to X > > samples long is a valid 'frame' is a cleaner solution, but I'm open to > > alternatives for sure. > > what about allowing it only for <4kbyte && <4ksamples ? It still doesn't solve the problem about how do demux. How about this instead: - Storing multiple frames together is allowed only if the concatenation of all frames is also a valid single frame for the format. This way the demuxer can always output the whole frame as a unit and doesn't have to decide whether that's safe or not. But, having the packet_size (should be called frame_size...?) in the header is still useful, since it would allow more fine-grained seeking in a special-purpose application like a media editor. We can also put a limit on number of samples. IMO there's no real need for a size limit, and if there is a size limit it should probably be relative to max_distance rather than absolute. If we really want to avoid absolutes, we could make the sample limit relative to max_pts_diff too... Rich

3 5

Re: [NUT-devel] FINALIZE NUT SPEC!!
by Oded Shimon 27 Feb '06

27 Feb '06

Move thread to nut-devel... Sorry, I should have mentioned in the first mail that all replies should go to nut-devel if possible. On Mon, Feb 27, 2006 at 01:34:52PM -0500, Rich Felker wrote: > On Mon, Feb 27, 2006 at 03:43:52PM +0200, Oded Shimon wrote: > > Also I want to improove speed, everytime I seeked back the damaged region > > when reaching it again it linear searched the entire ~200mb of damage > > again... > > This might be desirable in the case of an incremental download, but in > general it's probably good to cache known bad regions. Which actually also brings the issue of seeking in a "growing" file. In the first seek you do, libnut does a discovery seek of finding the last syncpoint in the file, and marks it known. The only way it can invalidated is by playing past it (libnut automatically realizes this simply by adding a syncpoint to the syncpoint cache past the last syncpoint...). I should make some kind of signal in the API to indicate file has grown... > > I object to index splitting because it is unnecessary complexity for very > > very rare occassion - partial damage in a 80kb chunk in a seekable stream. > > Now you can copy the index anywhere, it's even more unnecessary... > > I tend to agree. If one sector of the index is broken it's likely that > many are. And if the index is unusable, it's not the end of the world. > Just seek without index or rebuild the index. In theory it does sound bad to loose such a nice feature over a "single byte" when you can salvage some of it by splitting, but in practice I don't ever see such a situation happenning. Seekable streams (streams which have an index at all) are probably permanent storage. Damage in permanent storage is practically always in HUGE chunks, not in a single byte... > > The big problem I have with just info packets is that there's no way to > > really load them because they can be in any arbitrary location. Like we > > restricted the index being only after main headers and not any arbitrary > > location so demuxers can find it, same with info packets... > > Global (chapter_id==0) info would still be required to be with > headers, and users would have to put any other chapter info they > wanted to be globally available with the headers too. Storing it other > places would just be for streaming purposes. > > However, for many streaming applications, you need the full power of a > stream to represent realtime data reasonably. So even if we allow info > packets at places other than with headers, I still want info streams. If we're having info streams, then no "arbitrary info packets" please... They are an annoying complexity. (I have an api function, get_packet(), which when called either always gives a single frame, or throws an error. If it sees main headers, it skips over the entire header stuff, and recurses... With arbitrary info packets, this will have to turn nasty as a call could either give a frame, or an info packet...) One nice advntage I see to dumping info streams is that, in reality, info streams don't really have "stream-like" behavior... (They make back_ptr big etc.) - ods15

2 1

[NUT] (ods15): r83 - /trunk/libnut/reorder.c
by syncmail＠mplayerhq.hu 27 Feb '06

27 Feb '06

Author: ods15 Date: Mon Feb 27 19:01:20 2006 New Revision: 83 Log: add stdio.h for off_t Modified: trunk/libnut/reorder.c Modified: trunk/libnut/reorder.c ============================================================================== --- trunk/libnut/reorder.c (original) +++ trunk/libnut/reorder.c Mon Feb 27 19:01:20 2006 @@ -1,5 +1,6 @@ #include <inttypes.h> #include <stdlib.h> +#include <stdio.h> #include "nut.h" #include "priv.h"

1 0

dump info streams, just info packets
by Oded Shimon 27 Feb '06

27 Feb '06

It's non ideal IMO, I like the current system where the only packets that can appear during normal demuxing is only frame, syncpoint, or main header. But still, my suggestion for just info packets is: all info packets with chapter_id >= 0 MUST appear at BOF (or maybe MUST appear after all copies of main headers), and chapter_id <0 carry no restrictions... I still preffer an info stream solution if possible... - ods15

1 0

Patch for info frames
by Oded Shimon 26 Feb '06

26 Feb '06

Comments... - ods15

3 7

[NUT] (ods15): r82 - /trunk/libnut/priv.h
by syncmail＠mplayerhq.hu 26 Feb '06

26 Feb '06

Author: ods15 Date: Sun Feb 26 09:01:27 2006 New Revision: 82 Log: info table is obsolete Modified: trunk/libnut/priv.h Modified: trunk/libnut/priv.h ============================================================================== --- trunk/libnut/priv.h (original) +++ trunk/libnut/priv.h Sun Feb 26 09:01:27 2006 @@ -157,25 +157,6 @@ int sync_overhead; }; -static const struct { char * name, * type; } info_table [] = { - {NULL , NULL }, // end - {NULL , NULL }, - {NULL , "UTF8"}, - {NULL , "v"}, - {NULL , "s"}, - {"StreamId" , "v"}, - {"Author" , "UTF8"}, - {"Title" , "UTF8"}, - {"Language" , "UTF8"}, - {"Description" , "UTF8"}, - {"Copyright" , "UTF8"}, - {"Encoder" , "UTF8"}, - {"Keyword" , "UTF8"}, - {"Cover" , "JPEG"}, - {"Cover" , "PNG"}, - {"Disposition" , "UTF8"}, -}; - static inline uint32_t crc32(uint8_t * buf, int len){ int32_t crc = 0; int i;

1 0

Patch for info packets
by Oded Shimon 26 Feb '06

26 Feb '06

This is my patch... Comments welcome, I hope this patch will please everyone... - ods15

3 17

FINALIZE NUT SPEC!!
by Oded Shimon 25 Feb '06

25 Feb '06

The info frames patch I just sent to nut-devel is the last withstanding problem I know of for NUT. libnut seeking has been rather heavily tested, even with EOR. We could still add (remove?) some fields for Info packets/frames, but that can be done after finalizing. libnut/nututils is far from ready for production, but I think mpcf.txt _IS_. Is there anything else? Is anyone displeased? (even Ivan is happy and stopped flaming! :) Speak now or forever (or atleast until NUT 2.0 :) hold you peace... (The spec could really use a cleaning up, but the bitstream is locked...) - ods15

2 1

[NUT] (ods15): r81 - /trunk/libnut/muxer.c
by syncmail＠mplayerhq.hu 25 Feb '06

25 Feb '06

Author: ods15 Date: Sat Feb 25 19:50:31 2006 New Revision: 81 Log: sync to spec for info rationals Modified: trunk/libnut/muxer.c Modified: trunk/libnut/muxer.c ============================================================================== --- trunk/libnut/muxer.c (original) +++ trunk/libnut/muxer.c Sat Feb 25 19:50:31 2006 @@ -390,7 +390,7 @@ put_s(tmp, -3); put_s(tmp, field->val); } else if (!strcmp(field->type, "r")) { - put_s(tmp, -(field->den + 2)); + put_s(tmp, -(field->den + 3)); put_s(tmp, field->val); } else { if (strcmp(field->type, "UTF-8")) {

1 0