[NUT-devel] Info packets in NUT stream (spec bugs?)

[Michael Niedermayer]

Hi

On Sun, Nov 19, 2006 at 06:58:59PM +0200, Oded Shimon wrote:
> On Fri, Nov 17, 2006 at 01:02:22PM +0100, Michael Niedermayer wrote:
> > On Fri, Nov 17, 2006 at 09:31:40AM +0200, Oded Shimon wrote:
> > > This doesn't seem like a streaming protocol, more like a butchering of the 
> > > nut spec (most obvious, allowing syncpoints not be written - this is an 
> > > obvious violation of max_distance). This doesn't necessarily make it a bad 
> > > idea (if you are completely error free, and not seeking, syncpoints are 
> > > redundant and non-trivial to calculate for their back_ptr), but it does 
> > > not seem like something to be written in a seperate document and being 
> > > called a "streaming protocol" - it's a modified NUT format spec for 
> > > streaming... And I think it's probably a bad idea to have several modified 
> > > specs (it's somewhat like mpeg-ts, mpeg-ps, mpeg-bla).
> > 
> > no mpeg-ts and mpeg-ps are much more different then what i proposed
> > 
> > the problem simply is that different environments/systems have different
> > requirements, if there is packet loss then some additional things are needed
> > if not they are a waste, surely some minimum error recovery stuff can always
> > be added but then again you where the one saying you will ignore the spec
> > and not repeat headers as its not needed in your case! so what do you
> > argue against? seems like its what you yourself wanted?
> 
> I don't consider it a butchering of the spec to omit repeated headers in 
> the case of realtime streaming as it is an impossible rule to inforce and 
> illogical in the case of realtime streaming. The only way to truely 
> enforce this rule in realtime streams is by making all 3 header copies 
> right at the begginning, with no frame inbetween - otherwise the user can 
> stop the dumping immediately after it begun and it would be impossible to 
> get all 3 copies even if the streamer repeated headers.
> Even if they were repeated - they would not be useful for finding in 
> dumped files as they would not be in 2^n locations (unless you expect the 
> streaming server to start doing calculations for EACH client..)

well, if you want to be able to start playing in the middle (your comment
strongly suggests that) you either need to repeat headers often or transmit
them out of band, or allow clients to seeks to past data somehow ...

also i think we once said that timestamps must start at 0, you seem to break
that too, iam not sure if theres anything in the spec which forbids it or
if theres anything which would break, iam just mentioning it ...

and anyway the not duplicated headers are a minor issue about which i dont
really care as long as its clear that this is not ok in normal nut files
it of course can happen in truncated or otherwise damaged files and a
demuxer still should handle that sanely ...


[...]
> > if you buther the spec by omiting repeated header then butcher it enough
> > so it cannot be played if dumped raw otherwise such broken streams will
> > spread not neccessarily originating from stream dumps but maybe rather
> > people who like to safe 0.1% space
> 
> Well, I somewhat doubt that will actually happen, but I accept the
> argument...

you want to omit them in your streams IIRC because of the space saving so
why shouldnt others too?


> 
> > and then info packets, i will not accpet info to be unfindable, that is
> > files stored on disk must have at least
> > 
> > O(number of distinct info packets) to enumerate all distinct info packets
> > O(file size) is not ok
> > simply adding a single back pointer to midstream info packets would help
> > alot here
> 
> I have no idea for a solution. Your proposed solution doesn't work as 
> there is no way to find the last info packet. even with your stuffing 
> info packet in headers idea: 1. how would you find the headers? 2. who 
> said the headers are necessarily after the last info packet in the file? 
> Our main assumption is that in realtime stream dumping, they can be 
> cut/truncated at absoloutely any arbitrary point in the file.

1. every midstream info packet (and only them) MUST have a pointer to the
   previous non redundant/repeated midstream info packet
2. the distance between midstream info packets MUST be <= C*max_distance
   unless that is impossible (too large stream header / frame) in which
   case the distance MUST be as small as possible for the specific case
3. a info packet MUST either be part of the normal info packets or it
   MUST be repeated like described in 2. within the area to which it applies
   and it MUST be in that area at least once


> 
> 
> 
> BTW, if we ultimately decide on disallowing mid-stream info packets, we 
> need to remove these 2 parts from the spec:
> ---
> If a demuxer has seen several info packets with the same chapter_id and
> stream_id then it MUST ignore all but the one with the highest position in
> the file
> 
> demxuxers SHOULD NOT search the whole file for info packets
> ---
> 
> As they make no sense without them (all headers must be repeated 
> identically after all main headers)

leaving them does no harm droping them changes the frozen spec again ...
it also makes adding mid stream info packets at a later time much
harder if the rules are droped now


> 
> BTW2, If we decide we do allow mis-stream info packets, then this rule 
> somewhat fails:
> ---
>     chapter_id n MUST not be used unless there are at least n chapters in the
>     file
> ---
> 
> As a streamer would start at chapter_id=1, and linearly grow - you can 
> join at any arbitrary point, even when chapter_id=5926.

is a arbitrary cut part (both begin and end cut) always a valid nut file?
if yes then every file is a valid nut file as it could be the data of a
single frame

i would rather say that the whole stream from its true begin forms a valid
nut file and when you start cutting it becomes a damaged stream which
must be dealt with correctly by demuxers if all headers and one syncpoint is
available


[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

In the past you could go to a library and read, borrow or copy any book
Today you'd get arrested for mere telling someone where the library is