[MPlayer-dev-eng] More on inverse telecine!

[D Richard Felker III]

On Sun, Apr 13, 2003 at 12:59:04PM -0500, Billy Biggs wrote:
> D Richard Felker III (dalias at aerifal.cx):
> > Well there are several metrics I use.
> 
>   Thanks a ton for describing them.  This is really helpful.  The
> metrics I'm using are mostly blurred difference, so rather than take
> into account noise later, blur the input before trying to detect
> differences across fields.

Hmm, I'm curious as to how this works... Do you do a blend filter like
-vf pp=lb and then look for the 'ghosts'?

> > Then, once I have all these stats, I take the max of each individually
> > over all 8x8 blocks, and I take the max of the following quantities as
> > well:
> 
>   This is pretty cool.

The idea of taking max is that you can focus in on very small
artifacts, such as just a mouth moving in an otherwise still anime
scene.

> > One dead giveaway that a frame is part of the telecine pattern is when
> > the max even-odd is massively smaller than max odd-even. The
> > difference is often as significant as 80 vs. 3000!
> 
>   Yes.
> 
>   The difference in mine is that I keep a history of the last 5 frames,
> enough for one full sequence, and then look for the minimum difference
> in the top and bottom sequences, and match that against a perfect
> sequence, and see how many match.

That's kinda what I wanted to do in 2pass, but I got lazy...

>   What's neat about the history/matching is that you can also match
> against different cut positions, and try and keep probabilities around.
> More on this after some experiments though.
> 
>   I think that even though history stuff can get too complicated, as
> long as you keep your history REALLY SMALL, things are pretty much ok.

Yeah, I don't call history over the last 5 frames "really small"
though... :)

> > the scene change, but there's also another nasty possibility! The
> > first frame after the scene change can be one of the interlaced
> > frames, and you have no previous reference frame to compare it against
> > to tell if it's interlaced!
> 
>   This is hilarious.  Exactly the problem I had with a recording I did
> of part of an episode of 'Power Rangers'.  Checking for these pulldown
> edits was why I wrote my 'offline' algorithm in reetpvr.  It's two-pass,
> and tries to narrow down on where edits occur, and I hand-wrote
> strategires for each possibility.

Yeah, I kept wanting to do 2pass code, but it was such a pain to make
a stats file over and over with different metrics as I tested which
ones were useful and which weren't. So now I've decided one a 1pass
algorithm with 1frame delay instead.

Keep in mind the 1frame delay isn't very noticable (1/48 second at
most) for realtime playing. And for encoding, the way I wrote the
filter, the delay should not get passed on to the output file at all,
although the very last frame of the file will be lost... If this is
problematic, you could count the number of frames first, and tell the
filter this before starting so it could know to display the last frame
immediately without waiting for another to compare it against.

>   Of course for realtime playback, I don't have that luxury.

But 1frame delay could work for realtime. :)

>   Still, I should maybe better describe my offline algorithm's handling
> of cuts to you as it might be helpful.  Old code is in CVS at
> www.sf.net/projects/reetpvr/

Thanks. It'd be easier if you could just explain it, but if that's too
much trouble I'll read the code sometime.

> > Right now, the only case I find it to be missing is that
> > horrible one-line-per-frame vertical scrolling.
> 
>   I need to find an example of this, you're freaking me out. :)

I don't know if you have access to mphq, so get it from my site.

http://brightrain.aerifal.cx/~dalias/telecine/lain-ep1-laced-1pel-scroll.avi

> > By the way, one more thing about sticking to the telecine pattern.
> > Lots of times, especially in anime, you have some motion, followed by
> > a period of total stillness. If you track the pattern and keep merging
> > frames during the stillness, you might actually reduce the picture
> > quality, due to quantization errors in the source file/dvd and slight
> > noise/discrepencies between the frames due to analog processing during
> > production. This won't hurt visually in most cases, but it does waste
> > bits if you'll be encoding.
> 
>   I think you'll get more pain from the NTSC chroma crawl artifacts.
> Having a good filter for those will probably help a ton, and also help
> tons for detecting the pulldown phase.

Yeah, I made the mistake of adding chroma checking to vf_detc. It
messes things up horribly because of "chroma crawl" or whatever the
crappy artifacts from the analog signal were. For my new filter, I
decided to just ignore chroma entirely in the detection.

But you're right about the visual aspects too, and that's another
reason it's good to drop a frame rather than merging if there's no
motion. merging can create an interlaced look to the chroma planes
because of this mess... :(

>   But that said, the real thing missing here is something I was
> specifically trying to avoid:  a metric for detecting 'interlaced
> frames'.  I think that pulldown detection should be mostly biasd towards
> the difference metrics, since they work best, but we should still have

My current algorithm is to mark a frame if it matches "strongly" based
on any of the difference metrics or the noise metrics...

> some thresholds for 'should I bother', and only merge or whatever if
> doing so would avoid comb artifacts.  Ugh ugh ugh though.

...then only do the actual merge if it matches "ok" on the noise
metrics. Otherwise, drop the frame (assuming it's an unrecoverable
interlaced frame, e.g. one right before scene change).

Rich