[NUT-devel] Info packets in NUT stream (spec bugs?)

[Oded Shimon]

On Fri, Nov 17, 2006 at 01:02:22PM +0100, Michael Niedermayer wrote:
> On Fri, Nov 17, 2006 at 09:31:40AM +0200, Oded Shimon wrote:
> > This doesn't seem like a streaming protocol, more like a butchering of the 
> > nut spec (most obvious, allowing syncpoints not be written - this is an 
> > obvious violation of max_distance). This doesn't necessarily make it a bad 
> > idea (if you are completely error free, and not seeking, syncpoints are 
> > redundant and non-trivial to calculate for their back_ptr), but it does 
> > not seem like something to be written in a seperate document and being 
> > called a "streaming protocol" - it's a modified NUT format spec for 
> > streaming... And I think it's probably a bad idea to have several modified 
> > specs (it's somewhat like mpeg-ts, mpeg-ps, mpeg-bla).
> 
> no mpeg-ts and mpeg-ps are much more different then what i proposed
> 
> the problem simply is that different environments/systems have different
> requirements, if there is packet loss then some additional things are needed
> if not they are a waste, surely some minimum error recovery stuff can always
> be added but then again you where the one saying you will ignore the spec
> and not repeat headers as its not needed in your case! so what do you
> argue against? seems like its what you yourself wanted?

I don't consider it a butchering of the spec to omit repeated headers in 
the case of realtime streaming as it is an impossible rule to inforce and 
illogical in the case of realtime streaming. The only way to truely 
enforce this rule in realtime streams is by making all 3 header copies 
right at the begginning, with no frame inbetween - otherwise the user can 
stop the dumping immediately after it begun and it would be impossible to 
get all 3 copies even if the streamer repeated headers.
Even if they were repeated - they would not be useful for finding in 
dumped files as they would not be in 2^n locations (unless you expect the 
streaming server to start doing calculations for EACH client..)

It's a butchering of the spec only because you can semantically consider 
the file truncated and incomplete - in your suggest of max_distance 
butchering, you're explicitly butchering the spec to the point that 
compliant demuxers will not be able to play it without different modes for 
streaming and files.

> if you buther the spec by omiting repeated header then butcher it enough
> so it cannot be played if dumped raw otherwise such broken streams will
> spread not neccessarily originating from stream dumps but maybe rather
> people who like to safe 0.1% space

Well, I somewhat doubt that will actually happen, but I accept the
argument...

> and then info packets, i will not accpet info to be unfindable, that is
> files stored on disk must have at least
> 
> O(number of distinct info packets) to enumerate all distinct info packets
> O(file size) is not ok
> simply adding a single back pointer to midstream info packets would help
> alot here

I have no idea for a solution. Your proposed solution doesn't work as 
there is no way to find the last info packet. even with your stuffing 
info packet in headers idea: 1. how would you find the headers? 2. who 
said the headers are necessarily after the last info packet in the file? 
Our main assumption is that in realtime stream dumping, they can be 
cut/truncated at absoloutely any arbitrary point in the file.



BTW, if we ultimately decide on disallowing mid-stream info packets, we 
need to remove these 2 parts from the spec:
---
If a demuxer has seen several info packets with the same chapter_id and
stream_id then it MUST ignore all but the one with the highest position in
the file

demxuxers SHOULD NOT search the whole file for info packets
---

As they make no sense without them (all headers must be repeated 
identically after all main headers)

BTW2, If we decide we do allow mis-stream info packets, then this rule 
somewhat fails:
---
    chapter_id n MUST not be used unless there are at least n chapters in the
    file
---

As a streamer would start at chapter_id=1, and linearly grow - you can 
join at any arbitrary point, even when chapter_id=5926.

- ods15