[Ffmpeg-devel] few remarks for h264 decoder

[Loren Merritt]

On Sat, 31 Dec 2005, G?bor Kov?cs wrote:

> I think I found another one, but it's mmx releated which I don't actually use 
> and I don't even have sample clip for this, so there is a chance I'am wrong.
>
> 9.
> It's about deblocking. alpha_table[] has zero values for low qscale. In this 
> case ABS( p0 - q0 ) < alpha should always fail. The problem with mmx code 
> that it uses alpha-1. In H264_DEBLOCK_MASK() packuswb will saturate -1 to 0 
> and the |p0-q0| > alpha-1
> condition may allow deblocking when |p0-q0| = 0. Probably rare problem.

If I read this correctly:
alpha=0 and beta=0 occur in the same place in the tables (qp<=15), so the 
failure case is that it will erroneously enable filtering when p1=p0=q0=q1 
... where it does nothing, because the block is flat anyway.
So it will only trigger when alpha_c0_offset != beta_offset.

--Loren Merritt