[FFmpeg-user] ffmpeg Newbie: VDPAU/ION, H.264, Non-Standard y Resolutions (and XBMC)

[Ian Harper]

Apologies, long post, but please bear with it if the title makes any
sense to you. 
Okay, how to phrase this... I'm happy to wade into ffmpeg and try to
track this down, but it's all a new world so I thought I'd at least
ask first!
Linux (Ubuntu 10.04 and 10.10), Nvidia ION platform (Acer Aspire Revo
3600), several different Nvidia driver versions tried. Easiest test is
simply to go with the latest git XBMC code or a pre-compile such as
the PVR version that uses the mainline codebase.
I picked up an issue on XBMC when the internal version of ffmpeg was
updated early this year (from ffmpeg r24229 to c3beafa0f1 - here's
the commit [1] that introduced the problem. After much bludgeoning,
this seems to be the essence of the problem as seen in XBMC, at
least:

1. Corruption on certain H.264 video streams - the very bottom of the
image dissolves into green soup. Attractive, but somewhat distracting.
2. The files played okay on the previous ffmpeg code (the version in
10.1 Dharma, versus the version that's now in the pre-11.0 Eden
release).
3. It only shows on XBMC when VDPAU is enabled, but that could be due
to how XBMC handles VDPAU/non-VDPAU *or* due to how ffmpeg uses VDPAU.
So, disable VDPAU, the files play; enable it, they fall apart.
4. It appears on files from the on-demand UK terrestrial
broadcasters, such as the BBC (especially HD) and Channel 5 - so, for
example, iPlayer for BBC HD programmes streamed within XBMC.
5. It seems to be related to the "odd" file dimensions they use:
1080x720 plays, 1080x724 doesn't - 640x360 does, 640x372 doesn't.
From reading around, it may be that these additional lines should be
cropped out, but not all players understand that.
6. libvdpau works, because mplayer with VDPAU can play the files on
the same system (-vo vdpau -vc ffh264vdpau).
7. ffplay testing is inconclusive, since it doesn't seem to support
VDPAU no matter how much you shout at it. It thus struggles to play
an HD file on this system - but the few frames it does play show
corruption at the bottom of the image (albeit a different corruption
from XBMC's). Adding a -vf 1280:720:0:0 to ffplay does away with that
corruption (although the system is still too slow to play the files).
More information to anaesthetise you here [2]if you're feeling
masochistic.
SOOOO, WHERE NEXT.. OH, YES: MY POINT!
I'm about to embark on a round of endless re-compilation - XBMC using
external ffmpeg, compiled against multiple SVN and git versions to see
if I can find where the issue arrived. 
However, before I do that - because it'll drive me and the family
insane, most likely - does this make sense to anyone? Has anyone else
seen problems with non-standard y resolutions, or issues with such
files and VDPAU, or anything else that rings a bell with this ramble?
Any thoughts from the collected wisdom very much appreciated.

Links:
------
[1]
https://github.com/xbmc/xbmc/commit/1a6a927ec5a0c305f58fa44bc0d023e007820b64
[2] http://forum.xbmc.org/showthread.php?t=99448