[NUT-devel] Info packets in NUT stream (spec bugs?)

[Rich Felker]

On Mon, Nov 20, 2006 at 03:29:29PM +0100, Michael Niedermayer wrote:
> well, if you want to be able to start playing in the middle (your comment
> strongly suggests that) you either need to repeat headers often or transmit
> them out of band, or allow clients to seeks to past data somehow ...

His idea is that the server would semi-dynamically generate a stream
by first writing the headers when a client connects, then copying
everything else from disk. This is what I had in mind too.

> also i think we once said that timestamps must start at 0, you seem to break

Absolutely not! If this were required then you could not split long
movies into multiple files to split across media without losing
information (and mildly breaking a/v sync since it's impossible for
all streams to start exactly at 0 after splitting a file unless you
choose a cut point divisible by all frame durations...).

> that too, iam not sure if theres anything in the spec which forbids it or
> if theres anything which would break, iam just mentioning it ...

Starting at arbitrary timestamp is totally legal and it was always my
intent for it to be legal. Obviously streaming is not possible either
if you don't allow for this..

> and anyway the not duplicated headers are a minor issue about which i dont
> really care as long as its clear that this is not ok in normal nut files
> it of course can happen in truncated or otherwise damaged files and a
> demuxer still should handle that sanely ...

Agree.

> > > if you buther the spec by omiting repeated header then butcher it enough
> > > so it cannot be played if dumped raw otherwise such broken streams will
> > > spread not neccessarily originating from stream dumps but maybe rather
> > > people who like to safe 0.1% space
> > 
> > Well, I somewhat doubt that will actually happen, but I accept the
> > argument...
> 
> you want to omit them in your streams IIRC because of the space saving so
> why shouldnt others too?

Well in Oded's case it happens that they're utterly useless, but as
long as we follow what you and I said above I don't think there's a
problem.

> > I have no idea for a solution. Your proposed solution doesn't work as 
> > there is no way to find the last info packet. even with your stuffing 
> > info packet in headers idea: 1. how would you find the headers? 2. who 
> > said the headers are necessarily after the last info packet in the file? 
> > Our main assumption is that in realtime stream dumping, they can be 
> > cut/truncated at absoloutely any arbitrary point in the file.
> 
> 1. every midstream info packet (and only them) MUST have a pointer to the
>    previous non redundant/repeated midstream info packet
> 2. the distance between midstream info packets MUST be <= C*max_distance
>    unless that is impossible (too large stream header / frame) in which
>    case the distance MUST be as small as possible for the specific case
> 3. a info packet MUST either be part of the normal info packets or it
>    MUST be repeated like described in 2. within the area to which it applies
>    and it MUST be in that area at least once

Seems like the sanest proposal for this so far, but.... introduces
huge ugly complexity to implementation.

> > BTW, if we ultimately decide on disallowing mid-stream info packets, we 
> > need to remove these 2 parts from the spec:
> > ---
> > If a demuxer has seen several info packets with the same chapter_id and
> > stream_id then it MUST ignore all but the one with the highest position in
> > the file
> > 
> > demxuxers SHOULD NOT search the whole file for info packets
> > ---
> > 
> > As they make no sense without them (all headers must be repeated 
> > identically after all main headers)
> 
> leaving them does no harm droping them changes the frozen spec again ...
> it also makes adding mid stream info packets at a later time much
> harder if the rules are droped now

Yes, let's not harm syntactic extensibility. I doubt the sanity of
this mid-stream info at all, but...

> > BTW2, If we decide we do allow mis-stream info packets, then this rule 
> > somewhat fails:
> > ---
> >     chapter_id n MUST not be used unless there are at least n chapters in the
> >     file
> > ---
> > 
> > As a streamer would start at chapter_id=1, and linearly grow - you can 
> > join at any arbitrary point, even when chapter_id=5926.
> 
> is a arbitrary cut part (both begin and end cut) always a valid nut file?
> if yes then every file is a valid nut file as it could be the data of a
> single frame
> 
> i would rather say that the whole stream from its true begin forms a valid
> nut file and when you start cutting it becomes a damaged stream which
> must be dealt with correctly by demuxers if all headers and one syncpoint is
> available

An arbitrary dd-cut nut file is not a valid complete not file, and not
valid at all if the beginning has been cut. On the other hand, a file
cut with a proper cutting tool SHOULD be valid, e.g. if you're cutting
a movie to split it across undersized media (cdr). Requiring the
chapters to be renumbered after cutting is particularly ugly and
destructive of semantic information. I'd like to request a minor
revision: instead of requiring positive chater id's to start from 1,
just require them to be contiguous. This keeps implementation simple
(array storage can be used), but the first element need not reflect
"chapter 1" in a user sense.

Alternatively you can just keep all chapters stored in both/all files
after split, pointing to timestamp ranges outside of the current file.
I have no idea if this is valid or not... :)

Rich