[MEncoder-users] Re: Re: new doom9 codec comparission (submission)

[Michael Niedermayer]

Hi

the first thing id like to say, is that this is a public list and everyone
can comment and insult, please do not take the words of someone as a official
word from ffmpeg or mplayer

i also would like to say that my comments are just comments and
where absolutely not intended to claim that the results of the codec
comparission to be bad/inaccurate or the testing methods to be less then optimal
iam just an open source developer and as such like to disscuss and comment
about what i think has been done well and what not and how it could be
improved, this is needed for projects involving many people, if you cant
point at what you think are errors in someone else work then things wont
get better ...


On Mon, Dec 19, 2005 at 02:14:17PM +0100, Doom9 Feedback Hotline wrote:

[ comment by rich and awnser by doom9 removed ]

following comment added again as its what we are talking about

> > >and experience during this comparison, I actually find the result of
> > >decoding xvid decoded by xvid more visuallypleasing than when xvid is
> > >decoded by lavc (and that's in the avi container where the auto idct
> > >selection works just fine).XviD stands a lot less to lose by being decoded

> >and my reply was just to show this to be false, as such comments are
> >making lavc look bad to who reads this
> 
> This is where you go wrong. You can disagree with my findings, that's your 
> prerogative, but claiming they are false is just outrageous.. you do not 
> have my visual system so how on earth can you claim that my findings are 
> false? Given the exact same content and the same playback mechanism, you 
> might rate things differently, and that's perfectly okay and the reason why 
> I'm so meticulous in putting all information I think is necessary to 
> reproduce a test in the comparison.. so you can see with your own eyes and 
> draw your own conclusions. A metrics comparison can be right or wrong 
> because somebody screwed up in the technical part (not taking into account 
> frame delays introduced by B-frames for instance, duplicating frames to 
> make the length of two encoded files match, screw up the statistics by 
> calculation errors, etc), but those points are all moot for a subjective 
> comparison - in such case, right or wrong is literally a matter of point of 
> view. Unlike some other people here I do not claim to have the absolute 
> truth, I can only give what I see, it's up to everybody reading the 
> comparison to judge for themselves and trying to reproduce the results to 
> verify if they would come to the same conclusion. If they do, they could've 
> saved themselves a lot of effort, if they didn't, they might both have 
> learned something knew, and can be sure that now they make the right choice 
> for themselves.

ok, iam not a natively english speaking person and the word "false" may have
been a bad choice, ill try to rephrase what i wanted to say
you apparently compared ffdshow against xvid on windows which involves many
different parts beside the decoders themselfs, based on this apparently
you attack libavcodec and claim it to be less visually pleasing for _DECODING_
xvid then xvid itself
you also attacked lavc in several other messages here, for example:
...
> and guess what.. once again XviD comes out on top. You can skew the results
> by forcing XviD to be decoded by libavcodec to make the difference less
> apparent, but it cannot hide it's still there.

> Any call for using libavcodec instead of xvid to decode XviD is thus an
> attempt to twist the outcome to lavc's favor, and that's what I call
> cheating. So if you can win by cheating, fine but not in my comparison.
...

i compared both decoders and found them to give binary identical output for
all non broken streams i tried, based on this i claim that what you saw was
likely caused by something else then the decoders themself, and i further
claim that your comments are damaging to libavcodec and unscientfic as
you apparently didnt exclude differences besides the decoders in ffdshow
vs. xvid

also note, iam NOT claiming that ANYTHING in your conclusion about the ENCODING
comparission is inaccurate, false or anything! iam just saying that lavc
does DECODE xvid as well as xvid does decode itself, its the differences
in ffdshow and windows weird renderes which is likely responsible, and
it makes me somewhat angry to see you attack lavc because ffdshow looks bad
and again that is all about decoding not encoding


> 
> >also note iam also always interrested in bugreports which would show
> >examples where my pp code is worse, if noone sends us bugreports we
> >cant know about the issues and cant fix them ...
> 
> How can you say something is broken? As a fellow programmer, I wouldn't 
> dare call somebody broken until I have good reasons to make such a claim. I 

if my postprocessing code looks subjectively worse then some other which
implementes the same algorithm then its buggy


> can only point out that if you decode the xvid clip with ffdshow and then 
> with the xvid dshow filter (deblocking turned on in both cases), the 
> results of the latter look more visually pleasing to me. That's a 
> subjective call, I cannot say either one is broken or does it better, just 
> that one looks better to me than the other. That's all the info you need to 
> try and reproduce.. I'll be interested to hear your findings. And please 
> note, that this is about DirectShow filters so you have no choice but to do 
> this on a Windows box.. using mplayer just isn't the same.

mplayer supports dshow codecs and IIRC the deblock code is in the xvid
codec, so a comparission with mplayer seems possible
if theres no subjective difference betweem mplayers pp filter and dshow
xvid pp then its no longer my problem, but the problem of the ffdshow
developers, as in that case they messed something up, iam not ffdshow
developer and dont have the time to debug something on s OS i dont use


> 
> >no double blind tests?
> 
> You have two samples, feel free to do that on your own. I'm reasonably 
> confident that you cannot argue that amount of a difference away. And it 
> clearly says in the qualification that I cannot tell the difference between 
> lavc and xvid in the still scenes. That is, they are different but I cannot 
> make the call which one looks better.. they just look different.

i never said i disagree, i just tried to comment upon what could be improved


> 
> Finally, I need to say something: no other participant has ever give me 
> that amount of grief. I find it very unprofessional of anybody to try and 
> twist things to their favor by trying to force a certain decoder (they 
> probably know to be superior and make other codecs look worse?) via bully 

please read what you write too, you attack lavc every time you can, even 
though ive already shown that lavc and xvid produce binary identical output
when decoding xvid you still claim lavc would make others look worse, why is
my claim that ateme _might_ be designed more toward optimaly decoding the more
popular xvid then lavc so different?


> tactics, and by unwarranted attacks on my testing methods. In five years of 
> codec testing, this has not happened a single time before (I shudder at the 
> thought of looking up discussions that ensued after I tested libavcodec 
> last time). All participants, being open source teams or large commercial 
> entities, have always been courteous, and never destructive, even when 
> faced with unfavorable results, at times even mistakes of my own that would 
> have given them plenty of reasons to raise hell. They might respectfully 
> disagree with my findings, which is everybody's right, but what I'm 
> experiencing here goes way beyond that.

please note, that not everyone who attacks you is a member of the ffmpeg team
some are just random users or developers of other projects who often 
unintentionally speak in a very aggressive and insulting way

[...]

-- 
Michael