[NUT-devel] huge vs. damaged forward_ptrs in packets

[Rich Felker]

On Thu, Mar 09, 2006 at 03:36:02PM +0100, Michael Niedermayer wrote:
> Hi
> 
> On Wed, Mar 08, 2006 at 08:01:45PM -0500, Rich Felker wrote:
> > On Thu, Mar 09, 2006 at 01:40:45AM +0100, Michael Niedermayer wrote:
> > > > Also, can we have a field in the header instead of arbitrary 64k? I'm
> > > > fine with a requirement that it MUST be <= 64k. Some users may want
> > > > more intense checksum.. Also some muxers with very small output buffer
> > > > may want to use a smaller checksum block..
> > > 
> > > iam somewhat against this, at least for the main header its not possible
> > > so it would mean 2 different rules depending on header type
> > 
> > Well, mainheader is important enough that it might always be good to
> > have a checksum on the first N bytes of it..
> > 
> > > OTOH ill immedeatly agree to a smaller threshold, 16kb or so maybe?
> > 
> > Yeah, or even smaller. The majority of file contents will be frames,
> > not packets, anyway so IMO overhead doesn't really matter here. What
> > if all packets (regardless of size) have such a checksum, but it only
> > covers the 'critical' fields (i.e. forward_ptr, etc.)? The only
> > possible problem I see is that syncpoints would be a special case or
> > have nasty overhead.. Maybe it's a bad idea tho.
> 
> hmm maybe we can find a solution without a special case, though i
> wouldnt mind putting the checksum always on every header ...
> 
> what about a checksum after every 4kb of a packet?
> it would even allow to recover part of a large damaged packet
> or combine the undamaged blocks of 2 damaged packets
> not that i think that would be important in practice ...

You mean just placing the checksum at arbitrary byte positions
independent of contents? I'm at least mildly against this since it
seems to complicate the implementation, requiring the data to be
copied around to make up for the checksums inserted at random
positions in it before it can be parsed.

IMO the only piece of data that needs to be protected more than the
whole-packet checksum is the forward_ptr (otherwise the demuxer may
read unboundedly into the future trying to verify the whole-packet and
its checksum). What if, instead of a checksum on the first N bytes of
a packet when the packet size is >N, we instead do just a checksum on
startcode+forward_ptr when the packet size is >N? This would eliminate
the need for buffering to write the data, and would get all the
benefits of the secondary-checksum (i.e. protecting forward_ptr).

Rich