[NUT-devel] Info packets in NUT stream (spec bugs?)

[Oded Shimon]

On Mon, Nov 20, 2006 at 12:37:22PM -0500, Rich Felker wrote:
> On Mon, Nov 20, 2006 at 03:29:29PM +0100, Michael Niedermayer wrote:
> > well, if you want to be able to start playing in the middle (your comment
> > strongly suggests that) you either need to repeat headers often or transmit
> > them out of band, or allow clients to seeks to past data somehow ...
> 
> His idea is that the server would semi-dynamically generate a stream
> by first writing the headers when a client connects, then copying
> everything else from disk. This is what I had in mind too.

Not necessarily from disk. All it has to buffer is the main headers once, 
then it just has to wait for it's lead muxer to give a syncpoint, and 
then give everything to the client like any other client...

> > that too, iam not sure if theres anything in the spec which forbids it or
> > if theres anything which would break, iam just mentioning it ...
> 
> Starting at arbitrary timestamp is totally legal and it was always my
> intent for it to be legal. Obviously streaming is not possible either
> if you don't allow for this..

I've never actually tested it, but AFAIK libnut is completely safe and 
non-breaking on this issue.

> > you want to omit them in your streams IIRC because of the space saving so
> > why shouldnt others too?
> 
> Well in Oded's case it happens that they're utterly useless, but as
> long as we follow what you and I said above I don't think there's a
> problem.

I assume you mean what you said regarding "complete file" semantics and 
index. In which case, yes, I agree too.

> > > I have no idea for a solution. Your proposed solution doesn't work as 
> > > there is no way to find the last info packet. even with your stuffing 
> > > info packet in headers idea: 1. how would you find the headers? 2. who 
> > > said the headers are necessarily after the last info packet in the file? 
> > > Our main assumption is that in realtime stream dumping, they can be 
> > > cut/truncated at absoloutely any arbitrary point in the file.
> > 
> > 1. every midstream info packet (and only them) MUST have a pointer to the
> >    previous non redundant/repeated midstream info packet
> > 2. the distance between midstream info packets MUST be <= C*max_distance
> >    unless that is impossible (too large stream header / frame) in which
> >    case the distance MUST be as small as possible for the specific case
> > 3. a info packet MUST either be part of the normal info packets or it
> >    MUST be repeated like described in 2. within the area to which it applies
> >    and it MUST be in that area at least once
> 
> Seems like the sanest proposal for this so far, but.... introduces
> huge ugly complexity to implementation.

It's actually not that bad in implementation - just keep last_info and 
last_info_redundant (to know if to place another one now), a few more if's 
together with syncpoint writing, and you're done.

Or do you mean complexity in demuxer? In which case, yes, it is somewhat 
ugly... But I don't really agree in demuxer searching for info packets 
anywhere after main headers anyway.

I have two questions though:
How do you know if to search for these redundant packets or 
not? I assume you do NOT write them at all if there are no mid-stream info 
packets at all, in which case, when looking for them, how do you know if 
you simply haven't scanned enough of C*max_distance from EOF, or if there 
aren't mid-stream info packets at all?
Also, for the begginning of file, do you write these info_redundant until 
you have one info packet or not?
Sounds like this would need another flag in the main header.

> > leaving them does no harm droping them changes the frozen spec again ...
> > it also makes adding mid stream info packets at a later time much
> > harder if the rules are droped now
> 
> Yes, let's not harm syntactic extensibility. I doubt the sanity of
> this mid-stream info at all, but...

It does feel like a bikeshed issue, but it is very important to some 
people (not me BTW).

[...]
> I'd like to request a minor
> revision: instead of requiring positive chater id's to start from 1,
> just require them to be contiguous. This keeps implementation simple
> (array storage can be used), but the first element need not reflect
> "chapter 1" in a user sense.

I preffer this.

- ods15