Hi $subj, and what about a change like: --- mpcf.txt 2006-03-08 13:36:02.000000000 +0100 +++ mncf.txt 2006-03-05 14:49:57.000000000 +0100 @@ -312,6 +306,8 @@ packet_header startcode f(64) forward_ptr v + if(forward_ptr>64k) + header_checksum u(32) packet_footer reserved_bytes @@ -591,6 +576,9 @@ for frame headers the checksum contains the framecode byte and all following bytes upto the checksum itself +header_checksum + checksum over the startcode and forward pointer + Syncpoint tags: --------------- -- Michael
On Wed, Mar 08, 2006 at 01:52:47PM +0100, Michael Niedermayer wrote:
Hi
$subj, and what about a change like:
--- mpcf.txt 2006-03-08 13:36:02.000000000 +0100 +++ mncf.txt 2006-03-05 14:49:57.000000000 +0100 @@ -312,6 +306,8 @@ packet_header startcode f(64) forward_ptr v + if(forward_ptr>64k) + header_checksum u(32)
packet_footer reserved_bytes @@ -591,6 +576,9 @@ for frame headers the checksum contains the framecode byte and all following bytes upto the checksum itself
+header_checksum + checksum over the startcode and forward pointer + Syncpoint tags: ---------------
It's somewhat of a hack, but it works.. No real objections here. BTW, about removing 'metadata' stream type, could you push back 'userdata' to being 3?... - ods15
On Wed, Mar 08, 2006 at 01:52:47PM +0100, Michael Niedermayer wrote:
Hi
$subj, and what about a change like:
--- mpcf.txt 2006-03-08 13:36:02.000000000 +0100 +++ mncf.txt 2006-03-05 14:49:57.000000000 +0100 @@ -312,6 +306,8 @@ packet_header startcode f(64) forward_ptr v + if(forward_ptr>64k) + header_checksum u(32)
packet_footer reserved_bytes @@ -591,6 +576,9 @@ for frame headers the checksum contains the framecode byte and all following bytes upto the checksum itself
+header_checksum + checksum over the startcode and forward pointer +
I'm a little bit confused. Is this a second checksum, or replacing the old checksum at the end of the header? Also, can we have a field in the header instead of arbitrary 64k? I'm fine with a requirement that it MUST be <= 64k. Some users may want more intense checksum.. Also some muxers with very small output buffer may want to use a smaller checksum block.. Rich
Hi On Wed, Mar 08, 2006 at 11:20:22AM -0500, Rich Felker wrote:
On Wed, Mar 08, 2006 at 01:52:47PM +0100, Michael Niedermayer wrote:
Hi
$subj, and what about a change like:
--- mpcf.txt 2006-03-08 13:36:02.000000000 +0100 +++ mncf.txt 2006-03-05 14:49:57.000000000 +0100 @@ -312,6 +306,8 @@ packet_header startcode f(64) forward_ptr v + if(forward_ptr>64k) + header_checksum u(32)
packet_footer reserved_bytes @@ -591,6 +576,9 @@ for frame headers the checksum contains the framecode byte and all following bytes upto the checksum itself
+header_checksum + checksum over the startcode and forward pointer +
I'm a little bit confused. Is this a second checksum, or replacing the old checksum at the end of the header?
second, yeah it doesnt help if you find a packet with 1gb size and a missmatching checksum due to the whole packet and forward_ptr being random junk
Also, can we have a field in the header instead of arbitrary 64k? I'm fine with a requirement that it MUST be <= 64k. Some users may want more intense checksum.. Also some muxers with very small output buffer may want to use a smaller checksum block..
iam somewhat against this, at least for the main header its not possible so it would mean 2 different rules depending on header type OTOH ill immedeatly agree to a smaller threshold, 16kb or so maybe? [...] -- Michael
On Thu, Mar 09, 2006 at 01:40:45AM +0100, Michael Niedermayer wrote:
Also, can we have a field in the header instead of arbitrary 64k? I'm fine with a requirement that it MUST be <= 64k. Some users may want more intense checksum.. Also some muxers with very small output buffer may want to use a smaller checksum block..
iam somewhat against this, at least for the main header its not possible so it would mean 2 different rules depending on header type
Well, mainheader is important enough that it might always be good to have a checksum on the first N bytes of it..
OTOH ill immedeatly agree to a smaller threshold, 16kb or so maybe?
Yeah, or even smaller. The majority of file contents will be frames, not packets, anyway so IMO overhead doesn't really matter here. What if all packets (regardless of size) have such a checksum, but it only covers the 'critical' fields (i.e. forward_ptr, etc.)? The only possible problem I see is that syncpoints would be a special case or have nasty overhead.. Maybe it's a bad idea tho. Rich
Hi On Wed, Mar 08, 2006 at 08:01:45PM -0500, Rich Felker wrote:
On Thu, Mar 09, 2006 at 01:40:45AM +0100, Michael Niedermayer wrote:
Also, can we have a field in the header instead of arbitrary 64k? I'm fine with a requirement that it MUST be <= 64k. Some users may want more intense checksum.. Also some muxers with very small output buffer may want to use a smaller checksum block..
iam somewhat against this, at least for the main header its not possible so it would mean 2 different rules depending on header type
Well, mainheader is important enough that it might always be good to have a checksum on the first N bytes of it..
OTOH ill immedeatly agree to a smaller threshold, 16kb or so maybe?
Yeah, or even smaller. The majority of file contents will be frames, not packets, anyway so IMO overhead doesn't really matter here. What if all packets (regardless of size) have such a checksum, but it only covers the 'critical' fields (i.e. forward_ptr, etc.)? The only possible problem I see is that syncpoints would be a special case or have nasty overhead.. Maybe it's a bad idea tho.
hmm maybe we can find a solution without a special case, though i wouldnt mind putting the checksum always on every header ... what about a checksum after every 4kb of a packet? it would even allow to recover part of a large damaged packet or combine the undamaged blocks of 2 damaged packets not that i think that would be important in practice ... [...] -- Michael
On Thu, Mar 09, 2006 at 03:36:02PM +0100, Michael Niedermayer wrote:
Hi
On Wed, Mar 08, 2006 at 08:01:45PM -0500, Rich Felker wrote:
On Thu, Mar 09, 2006 at 01:40:45AM +0100, Michael Niedermayer wrote:
Also, can we have a field in the header instead of arbitrary 64k? I'm fine with a requirement that it MUST be <= 64k. Some users may want more intense checksum.. Also some muxers with very small output buffer may want to use a smaller checksum block..
iam somewhat against this, at least for the main header its not possible so it would mean 2 different rules depending on header type
Well, mainheader is important enough that it might always be good to have a checksum on the first N bytes of it..
OTOH ill immedeatly agree to a smaller threshold, 16kb or so maybe?
Yeah, or even smaller. The majority of file contents will be frames, not packets, anyway so IMO overhead doesn't really matter here. What if all packets (regardless of size) have such a checksum, but it only covers the 'critical' fields (i.e. forward_ptr, etc.)? The only possible problem I see is that syncpoints would be a special case or have nasty overhead.. Maybe it's a bad idea tho.
hmm maybe we can find a solution without a special case, though i wouldnt mind putting the checksum always on every header ...
what about a checksum after every 4kb of a packet? it would even allow to recover part of a large damaged packet or combine the undamaged blocks of 2 damaged packets not that i think that would be important in practice ...
You mean just placing the checksum at arbitrary byte positions independent of contents? I'm at least mildly against this since it seems to complicate the implementation, requiring the data to be copied around to make up for the checksums inserted at random positions in it before it can be parsed. IMO the only piece of data that needs to be protected more than the whole-packet checksum is the forward_ptr (otherwise the demuxer may read unboundedly into the future trying to verify the whole-packet and its checksum). What if, instead of a checksum on the first N bytes of a packet when the packet size is >N, we instead do just a checksum on startcode+forward_ptr when the packet size is >N? This would eliminate the need for buffering to write the data, and would get all the benefits of the secondary-checksum (i.e. protecting forward_ptr). Rich
On Thu, Mar 09, 2006 at 10:39:10AM -0500, Rich Felker wrote:
You mean just placing the checksum at arbitrary byte positions independent of contents? I'm at least mildly against this since it seems to complicate the implementation, requiring the data to be copied around to make up for the checksums inserted at random positions in it before it can be parsed.
IMO the only piece of data that needs to be protected more than the whole-packet checksum is the forward_ptr (otherwise the demuxer may read unboundedly into the future trying to verify the whole-packet and its checksum). What if, instead of a checksum on the first N bytes of a packet when the packet size is >N, we instead do just a checksum on startcode+forward_ptr when the packet size is >N? This would eliminate the need for buffering to write the data, and would get all the benefits of the secondary-checksum (i.e. protecting forward_ptr).
Forget it; Oded just told me that this is what you originally proposed. :) Sorry I misunderstood. I thought the secondary checksum was going to cover all bytes in the first N bytes of the packet, not just the startcode+forward_ptr. Rich
Hi On Thu, Mar 09, 2006 at 11:09:45AM -0500, Rich Felker wrote:
On Thu, Mar 09, 2006 at 10:39:10AM -0500, Rich Felker wrote:
You mean just placing the checksum at arbitrary byte positions independent of contents? I'm at least mildly against this since it seems to complicate the implementation, requiring the data to be copied around to make up for the checksums inserted at random positions in it before it can be parsed.
yes, it was a bad idea, forget it
IMO the only piece of data that needs to be protected more than the whole-packet checksum is the forward_ptr (otherwise the demuxer may read unboundedly into the future trying to verify the whole-packet and its checksum). What if, instead of a checksum on the first N bytes of a packet when the packet size is >N, we instead do just a checksum on startcode+forward_ptr when the packet size is >N? This would eliminate the need for buffering to write the data, and would get all the benefits of the secondary-checksum (i.e. protecting forward_ptr).
Forget it; Oded just told me that this is what you originally proposed. :) Sorry I misunderstood. I thought the secondary checksum was going to cover all bytes in the first N bytes of the packet, not just the startcode+forward_ptr.
yes, so we are back to the question of what value does N have 4k, 16k? variable and stored in main header and main header has always a 2nd checksum or every header always has a 2nd checksum and syncpoints have some >N rule? somehow all these seem pretty much the same to me, only thing iam strongly against is allowing very large N personally i think a fixed N would be the best solution [...] -- Michael
On Thu, Mar 09, 2006 at 06:44:48PM +0100, Michael Niedermayer wrote:
personally i think a fixed N would be the best solution
OK, that's fine with me. The reason I was originally against it was mainly my misunderstanding. I'm certainly happy with anything between 4k and 16k and probably anything between 1k and 64k. Sizes like 512, 1k, 2k, and 4k are nice in that they correspond to sector sizes on common media (damage usually comes in whole sectors or not at all). But it really doesn't matter too much to me. Rich
participants (3)
-
Michael Niedermayer -
Oded Shimon -
Rich Felker