[MPlayer-users] Objectionable yadif haloing with certain top crop parameters

[mplayer-list at media.mit.edu]

tl;dr:  "mplayer -vf yadif" produces terrible results when an even but
non-multiple-of-4 number of scanlines are cropped off the top.  It is
only the -top crop-, not whether the total number of scanlines is a
multiple of 4.  It happens regardless of whether the input is mpeg2
(with mplayer doing the cropping) or H264 (with Handbrake doing the
cropping as part of the transcoding process and no cropping arg
supplied to mplayer).  VLC displays the same behavior.  I can't
seem to find a prior report of this behavior anywhere.

Details:

I'm starting with NTSC SD interlaced recordings from Hauppage PVR-250
capture cards.  If I crop 0 or 4 lines off the top using "-vf crop="
(e.g., to avoid line-21 closed-captioning data) and use yadif, it
looks fine.  If I crop 2 or 6 lines off the top, it looks terrible.

If I transcode using Handbrake to H264 in an MKV container, either
with the output as a progressive file (but no deinterlacing filters
applied), or as an MBAFF (--encopts:tff, because mediainfo reports
that output from these cards is always top field first), then the
output also looks bad using "mplayer -vf yadif" if I have cropped 2 or
6 lines from the top, but it's okay if I've cropped 0 or 4 lines from
the top.  (If I'm cropping 2 or 6 lines, and also telling Handbrake to
produce interlaced output via MBAFF, I must also tell Handbrake to
crop 2 lines off the bottom so the frame size is a multiple of 4.)
Note that if I'm cropping with Handbrake, I am -not- supplying any
crop argument to mplayer.

In the "bad" case, fast motions, such as a white performer waving
their hands around gesturing or clapping, has very objectionable,
large colored halos and preimages---red or blue blurry copies of
the hands displaced from the hands by substantial amounts (like the
size of the entire hand), either leading or lagging the motion.
For example, hands coming together in a clap have red halos leading
the motion, while the hands themselves have turned into blue blobs,
until the moment of contact, whereupon they become pink hands again.
This is really obvious when single-stepping through the frames.

The "good" case has no color haloing and the hands, while somewhat
blurry from motion, look much more like hands than blobs.  And, of
course, without "-vf yadif", I see sharp but combed hands with no
haloing.

I've observed this effect in mplayer from version 1.0rc4-4.4.5
(Ubuntu 10.10) all the way through 1.1-4.8 (Ubuntu 14.04), using
both Handbrake 0.9.9 and 0.10.0.  I get identical results using
VLC media player 2.1.6 Rincewind (revision 2.1.6-0-gea01d28),
also under Ubuntu 14.04, which isn't entirely surprising because
I believe that VLC uses mplayer's yadif.  [VLC doesn't seem able
to play PVR-250 files, so I can't play one of those and ask VLC
to crop it, but using VLC's yadif to play a cropped Handbrake file
shows the same dependence as mplayer on how many scanlines Handbrake
cropped off the top.]

It's interesting that this happens even if Handbrake was the entity
that cropped the video, and even if mplayer's input is H264.  It seems
entirely related to the top crop value, but I'm not sure how that
value makes it all the way through Handbrake's transcoding process
in a way that then screws up yadif later.

I can supply exact command lines and input/output samples if that
would help diagnosis, and/or open a real ticket, if my description
above isn't enough and/or this is an unknown bug.  (I've never
submitted a bug report about mplayer before, and get the distinct
impression from the documentation that doing so can be a major
effort, but don't want to do so unless we know where the blame
lies.)

Thanks!

P.S.  I am neither deinterlacing not decombing during the transcode
from interlaced mpeg2 to either progressive or interlaced H264,
because I'm trying to preserve my options to apply better deinterlacers
in the future (either better software, or VDPAU), hence I don't want
to lose information by deinterlacing during transcoding.  (I also want
to preserve my ability to use an interlaced output device (a CRT) if I
can get an entirely interlaced path and hence do no deinterlacing at
all on output if that path is active.)  A single-step transcode with
deinterlacing or decombing does -not- show these effects, even though
I believe Handbrake uses yadif internally while decombing.  (I am not
positive about that, though.)  At the moment, 38% of my transcoded
corpus has a top crop of 2 mod 4 instead of 0 mod 4; fortunately, I
had not yet thrown away the originals when I noticed the bug, but I'm
not too happy about having to retranscode a large pile of stuff to
work around the bug.

(I don't have VDPAU-capable hardware configured yet, so I can't tell
if it might also have issues with such top crops, nor am I 100%
confident that VDPAU works well with MBAFF files; if anyone can run
either test, I'd appreciate it.)