[FFmpeg-devel] [PATCH] MMX implementation of VC-1 inverse transforms

[Michael Niedermayer]

On Sat, Jan 19, 2008 at 07:20:32PM +0100, Christophe GISQUET wrote:
[...]
> >                                  Figure 158: Matrix for 1-D 4-point Inverse Transform
> >                                           EMxN = ( DMxN . TM + 4) >> 3
> >                                   RMxN = (T?N . EMxN + CN . 1M + 64) >> 7
> [...]
> >  E8x8 , E8x4 , E4x8 , E4x4   are intermediate matrices. The values in the intermediate matrices
> > shall be limited to the range >=-4096 and <4095.
> 
> That means that EMxN<<3, the result of the initial horizontal pass
> before shift, has values in the -32768..32752 range.

-32768..32759


> 
> Wouldn't that imply that the final result of any of the combinations in
> both 1D idcts is within that range? Then 16bits signed arithmetics
> without bias can be used. The code wouldn't even have to perform the
> pre-shifting we discussed for the 4x4 dct.

for the first idct yes it should fit in 16bit without tricks
the second can overflow
for example
-1814       0       0       0
    0       0       0       0
    0       0       0       0
    0       0       0       0
after 1st pass:
-3855   -3855   -3855   -3855
    0       0       0       0
    0       0       0       0
    0       0       0       0
before >>7:
-65535 -65535  -65535  -65535
-65535 -65535  -65535  -65535
-65535 -65535  -65535  -65535
-65535 -65535  -65535  -65535
after 2nd pass:
 -512    -512    -512    -512
 -512    -512    -512    -512
 -512    -512    -512    -512
 -512    -512    -512    -512


> 
> Maybe my hopes are a bit too high there...
> 
> > one could now argue that the limiting is a little unclearly defined. But IMO
> > the equations clearly define the relation between input and output and they
> > dont contain any limiting themselfs so the encoder has to ensure that the
> > limits are not violated.
> 
> The initial terms in input are within -2048..2046. I bet there are not
> many beyond +/-1024, 

yes ...

[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Let us carefully observe those good qualities wherein our enemies excel us
and endeavor to excel them, by avoiding what is faulty, and imitating what
is excellent in them. -- Plutarch
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20080119/ed66fdf4/attachment.pgp>