[MPlayer-dev-eng] New inverse-telecine filter

[Zoltan Hidvegi]

> On Tue, Dec 02, 2003 at 10:10:08AM -0600, Zoltan Hidvegi wrote:
> > This is a new inverse telecine filter, similar to detc, ivtc and pullup,
> > but better.  Here is a summary how it works.
> 
> The name is very misleading. It implies the filter is better than
> pullup, which is certainly not true, since it's impossible to do
> correct inverse telecine without buffering fields ahead. At the very
> least, you need one field into the future. And there's a reason pullup
> buffers 6 fields...

For mencoder-g1 it is better than pullup, since pullup cannot keep the
framerate.  I'm interested in encoding, and as far as I know, G2 does
not have an encoder yet, and with G1 pullup is not very useful for
encoding because of the lack of rate control.  The licomb metrics of
pullup are much less accurate than the metrics I've used.  pullup
takes one field, approximates the other and takes the difference, so
it has to do a half-pixel shift for one field while leaving the other
field in place.  It is more accurate to shift both fields by a quarter
instead.  It's basically a [1 -3 3 -1] filter, instead of [-1 2 -1]
used by licomb.  Normally for linearly approximating smooth functions
the quarter shift has 4x less error than the half shift, but I have to
do two quarter shifts, so it's just 2x better, but actually it happens
to be better than that, since [1 -3 3 -1] is zero for quadratic
functions, so the error is cubic.  The frame-break detection algorithm
of pullup did not seem to be that good, but that's certainly fixable.

Also the licomb metric in pullup may have a bug, as it really looks at
16 lines (8 line pairs), I think it should really be 4 pairs.

It may be true that you need lookahead if you want to be absolutely
correct, but once you find a frame boundary, you can display the next
two fields as long as those two fields seem to match well.  There may
be an other matching field coming up, and it may look better if you
can selectively merge the better matching pixels from the repeated
field, or just chose to merge the field which matches better, but it
rarely makes much difference.  It works pretty well, I've encoded
slowed down 22.150fps movies from ATSC broadcast, I've seen 2-2-3-3
pulldown pattern handled perfectly, and noisy TV sources, interlaced
overlays etc. are all handled pretty well.  And for watching live TV
it is an advantage that you have no delay, and it was one of my goals,
to have a filter good for live TV.

An other problem I had, which may be just me not really understanding
mplayer, is that pullup, detc, ivtc and many other filters call
vf_get_image with mpi->width, mpi->height, even though other filters
use mpi->w, mpi->h, and I think the later is the correct one.

> The pullup module was already designed to handle all the buffer
> management and field merging with minimal copy overhead. It's also

My pullup2 has zero copy for most frames, only the merged frames,
usually 1 out of 4 are copied.  The pullup filter can do direct
rendering, but not after a crop filter, which used EXPORT.  I'm
encoding HDTV content where much of the content is scaled up sdtv,
with black ares on the left and the right, so I have to crop from
1920x1080 down to 1408x1052.  And crop does not pass the field flags,
and my patch to pass those was not accepted.  But even if it is, after
crop you cannot do direct rendering.  Maybe in G2 you can?  So
actually my filter does less copy than pullup if you need to crop.

I tried to make my filter as fast as possible, since I'd like to be
able to use it on live HDTV in real-time, and my 1.73GHz Athlon can
hardly keep up with that.  Unfortunately, the bottleneck is libmpeg2
and the vo.  Pullup actually can speed up the vo because it lowers the
frame rate.

> meant to allow plugging in new decision algorithms, although this code
> isn't in place yet. If you think you have better algorithms for
> choosing how to put frames together, I would suggest making them
> switchable in pullup, rather than writing an entirely new filter. We
> already have too many separate filters for playing around with ivtc
> algorithms...

I'll try to understand pullup better, but some parts of it, especially
the buffer management, seemed to be more complicated than necessary,
maybe because it is really for G2?

> > The filter works on
> > even-first (a.k.a. top-field first) frames, for a bottom-first frame the
> > first line is skipped to make it top-first.
> 
> This is rather awkward. Why not just invert stride and start at the
> bottom?

Sure, that works too, but I do not see how adding the stride once is
more awkward that reversing it?  You already skip the pixels close to
the edges, skipping one more line at the top does not make any
difference.  This is only for calculating the metrics.  I also skip on
the left to make sure I start at an 8-byte aligned address to speed up
MMX.

> > Even and odd are the SAD (sum of absolute difference) between the
> > current field, and the same field in the previous frame.  Noise is the
> > difference between the two fields within the same frame.  It measures the
> > distance of the 4 odd lines in the 8x8 block from the even lines.  First
> > the even lines are used to approximate the image half a pixel below the
> > even lines, and the odd lines are used to approximate the image half a
> > pixel above the off lines, and for each pixel the difference between
> > these two are calculated. 
> 
> OK, this is not the same as the noise/temp metric, but the "licomb"
> metric in pullup. This seems to be a good one. Confusing to mix names
> though.

I was using noise/temp since I was looking at detc first.  And it is
trivial to replace the metric calculations of detc with licomb or my
metrics, which actually makes detc much faster and quite usable.
licomb is the name of the compare function, and I wanted a separate
name for inter-frame and intra-frame licomb, and I was lazy to invent
new names.  I know that for the pullup filter everything looks like a
60fps sequence, so there is no such difference.

> > Using these statistics, the find_breaks routine tries to find frame
> > breaks, see the code for details.  This routine is probably not perfect
> > yet.
> 
> Again, you reuse terminology from the other filters to mean something
> radically different.

I'm not that familiar with mplayer, so I was really not aware that I
was confusing the terminology, please tell me what terminology I
should use.

> > If frame timestamps were available, the filter could be
> > easily modified to use those instead of frame counting.  The
> 
> Yes. My pullup is designed for G2, where it will not have to do any
> framerate management at all. Instead a separate filter could regulate
> the output fps if desired, but preferably one would encode to
> variable-fps container formats like nut.

Actually, mpeg is also variable fps considering the games you can play
with repeat-first-field etc., but of course, mencoder does not support
that.  But I want to encode movies with the tools available today, so
I wanted to have something which I can use before mencoder-g2 is ready
for prime-time.  I actually hacked myself an mencoder which makes the
timestamps global, and had the pullup2 filter use that instead of the
counts, then disable A-V sync in mencoder, and let the filter handle
it.  It is really quite trivial to do, but I assume that such a change
would never be accepted, and for mpeg-2 and TV sources, frame counting
and watching the mpeg flags works quite well.  Except, HDTV (mpeg-2)
sometimes have jumps forward in the audio timestamp when they switch
between commercials or from an ad block to the movie.  such a skip
should be treated as silence, but mplayer just keeps playing the next
audio packet, which throws of A-V sync badly.  Someone has sent a
bugreport about some very similar problem about a month ago which was
not mpeg-2, so it is probably a general problem in the audio demuxer.

Zoli