[MPlayer-dev-eng] New inverse-telecine filter

[D Richard Felker III]

On Wed, Dec 03, 2003 at 05:26:31PM -0600, Zoltan Hidvegi wrote:
> > > in the middle of the frame, it would show up as a bad match.  My field
> > > compare is more accurate than licomb, so I can do better decisions
> > > just by comparing adjacent fields.
> > 
> > Maybe so. BTW, I've heard that using a linear [3 1] filter for qpel
> > translate isn't very good, and that a 3tap or even 4tap filter should
> > be used instead. That's what I did in vf_tfields. Any opinions on
> > whether this would improve the metric?
> 
> I've actually experimented with more complicated metrics, but it was
> not better.  [3 1] is not that good for qpel, but better than [1 1]
> hpel used in licomb.  And I qpel on both fields, and somehow the
> errors tend to cancel out, that's how you get zero on all quadratic
> curves.  I've tried 4-tap cubic interpolation, and it was worse.  I
> think the problem is that many-tap works well for smoth changes, but
> do badly at sharp edges.  Perhaps n-tap looks good on screen, but it
> does not necessarily mean that it is a good metric.  I've tried many
> metrics, testing them on known progressive sources, where they should
> show near perfect match between the fields of a progressive frame.
> More complicated filters were slower, and were not more accurate.

OK, I was just curious if you'd tested. BTW, your metric is less
accurate than the [1 1] hpel used in pullup in at least one case. I
tried adding qpel [3 1] compare to pullup, and the floating "E" at the
end of lain-op-evil-m.avi gets mishandled for one frame, giving an
interlaced output frame. The affinities at that point are detected as
.0+.1..2..3++4. with qpel instead of .0+.1++2..3++4. with hpel. In the
former case, pullup sees a clear duration-2 frame ahead, so it decides
to treat the first three fields as a duration-3 frame so it doesn't
have to merge fields, but this turns out to be incorrect.

In any case, this behavior is probably incorrect. .0+.1..2..3++4.
should be treated as a duration-2, followed by a broken field,
followed by another duration-2, and I'm planning to commit such a
change, but I still find it unfortunate that your qpel method doesn't
recognize that fields 1 and 2 clearly go together like my hpel metric
does...

If you want to investigate, it's around frame #195 of the clip (hard
to give exact frame numbers because of the buffering :).

> One of the worst sequence is the Miramax opening logo with the
> nightline of New York, some parts of which looks interlaced by any
> metrics I can come up with.  But that's a rather extreme example,
> which hardly ever happens in real movies.

Could you tell me where I could obtain a copy of this clip?

> > > I have a flag in vf_priv_s called swapped (bad name, I know), which is
> > > set it the current frame is BFF.  And if it is set, the metric routine
> > > will reduce the frame hight by 1 and it will crop one line from the
> > > top, then the frame becomes TFF.  And I assume that repeat-first-field
> > > frames are never interlaced.
> > 
> > Ah, yeah, special-casing RFF frames like that would make it easy... :)
> 
> Do you have any examples where an RFF frame is interlaced?  That would
> look bad even on interlace-scan TV sets.

No, that would be entirely invalid, so it shouldn't exist. :)
Special-casing RFF like that is perfectly legitimate, and in fact I do
a sort of special-casing too, but I do it with fake all-0 metrics in
order to keep with the "input is just a sequence of fields" principle.

> > > I see.  My metric for the neighbouring fields (like yours, but yours
> > > is 2x bigger because of the 16-line bug), is trying to measure the
> > > distance between the neighbouring fields just like the diff metric
> > > measures the distance between the even or odd fields.  They should be
> > > comparable, so for example, the distance between the odd fields should
> > > be close to noise+temp, where noise is the distance between the two
> > > fields in the current frame, and temp is the distance between the
> > > current first field and the previous last field. 
> > 
> > I think Pythagoras might prefer to see some squares and square roots
> > in there... :)
> 
> No, this is a one-dimensional space, so I do not think so.

OK, fair enough. I'm still a little skeptical of attempts to correlate
the metrics like this, since I had bad luck with such an approach
early on in vf_detc, but perhaps it could be useful for deciding
whether your video is pure-interlaced rather than telecined.

> > I particularly like the 'breaks' method, because it's very definitive,
> > and based on the most objective information you have available:
> > duplicate fields. The affinity method, though usually correct, is much
> > more heuristic in nature, and in fact it will behave very incorrectly
> > during certain pathological cases. One example I know is a "fade-in"
> > from random noise to a still picture. In this case, each field has
> > affinity towards the next field. :(
> 
> But telecined fade-in will still happen in 24fps steps, which should
> be detectable, unless the fade-in is very slow (like slower than 2
> luma levels per film-frame) and there is no other motion at all.  And
> you can change the thresholds on my filter to be more sensitive, if
> you have a low noise source.  Do you have samples with such fade-in
> problems?

Yes, and they're 30 fps, so your idea falls apart... ;)

BTW, even if they were 24 fps, you need a duplicate-field ("breaks")
based metric to detect this, and that was my point. The "affinity"
decision can't work in all cases.

> If they do interlaced video effects, like fading on the top of
> telecined film, that's bad, I do not think my filter can handle that
> well, it will probably detect interlace content and will try to do
> some de-interlacing, but I do not think it'll look good.

Agreed, this is very difficult to handle, but it's not what I'm
talking about. Get lain-ep11-noisefx.avi to see what I was talking
about and try your filter on it.

BTW, the episode that clip is from is virtually impossible to
automatically inverse telecine due to lots of special effects and poor
editing (dropping or duplication of interlaced frames to adjust
playback speed) so it probably needs to be done by hand anyway... But
the noise-fade example still applies.

Rich