[MPlayer-dev-eng] [PATCH] fix artifacts from spp on qp 1 MPEG2

[Alexander Strange]

MPEG2 blocks at qp 1 get overfiltered by spp, apparently because  
"qp>>1" turns it into 0
which causes an integer overflow later. Attached patch clips qp at 1.
(see straight lines in http://img513.imageshack.us/img513/6021/ 
qp1aq7.png, sample
available after I figure out how to cut out the middle of an MPEG  
stream)

I don't know if it's really worth running the filter on qp 1  
macroblocks, anyway...

I also tried the attached spp-mpeg2quant.diff, which uses  
non_linear_qscale from libavcodec
instead of just >>1, but the effect on very high qp is so much  
stronger it looks worse than without it.
I'm not even sure if that's the theoretically right way to do it.
(Plus it causes the table to be duplicated in the binary.)

-------------- next part --------------
A non-text attachment was scrubbed...
Name: spp-overflow.diff
Type: application/octet-stream
Size: 453 bytes
Desc: not available
URL: <http://lists.mplayerhq.hu/pipermail/mplayer-dev-eng/attachments/20070908/a3188330/attachment.obj>
-------------- next part --------------

-------------- next part --------------
A non-text attachment was scrubbed...
Name: spp-mpeg2quant.diff
Type: application/octet-stream
Size: 1661 bytes
Desc: not available
URL: <http://lists.mplayerhq.hu/pipermail/mplayer-dev-eng/attachments/20070908/a3188330/attachment-0001.obj>