[MPlayer-dev-eng] comprehensive lavcopts test - in progress.

[Roman Gaufman]

Interesting stuff you're doing with lavcopts testing, but I'm making a test a 
little more comprehensive and need opinions/suggestions:

================================
== To summarize what will be tested: ==
================================
4 types of files (anim/movie-high/low-motion) with 297 tests with different 
lavcopts and 2 vbitrate settings (594 tests total for each file).
each test will return:
	1) psnr value
	2) filesize
	3) ratio of psnr value to filesize
	4) ratio of differences of psnr and filesize to general lavcops
	5) 2 zoomed in crops of 2 freeze frames (carefully chosen)
	6) time test took to run
	7) bar graph to show gain/loss in quality/size vs general args
	8) bar graph to show total psnr and file size

(each bar on the bar graphs will appear as 1 bar on top of the other , for 
psnr and file size)


==============================
== To explain things in more detail: ==
==============================

4 common scenarios in files: 
--------------------------------
1) paycheck-highmotion - direct copy from dvd, 30 second very high motion
2) paycheck-lowmotion - direct copy from dvd, 30 second very low motion bit
3) southpark-highmotion - a high quality dvd rip, 30 second very high motion
4) southpark-lowmotion - a high quality dvd rip, 30 second very low motion.

Since average bitrate cant be tested for such short files, especially since 
there is no low/high motion deviance in each file, know that high motion 
scenes will look much better in your final encoding, and filesize will be 
closer to what you asked for. 

This test is to check what improves quality at what gain/loss in filesize for 
common parts of your encoding. Which gives us the bigger picture of what will 
make your file smaller overall. So this little misconception shouldn't make 
the test any less fair.

If you think I am wrong, I'm open to suggestions how to make it more fair.

2 vbitrate settings:
---------------------
1) 900kbit
2) 1600kbit

These are common vbitrates used in encodings. If you disagree, and would like 
higher/lower bitrates tested as well, please let me know and I will add them 
to test. From my experience, these are the 2 common extremes, and although 
some would like lower bitrates tested, most would agree that any higher than 
1600kbit isnt really needed.

2pass encoding:
--------------------
What I was thinking is to use a 2pass encode. That will give each opt 
combination more space to work with, and the pass2 will shrink file to be 
closer to the filesize we want. This I think is better than using a constant 
bitrate as many args do not work as expected when forced to use a constant 
bitrate, even though the bitrate deviation will be very limited in this test.

why ratio of filesize to psnr? and why vb_strategy=1:vratetol=32000
---------------------------------------------------------------------------
I think that will help figure out the total advantage of each test. 

File size wont be very consistent because of vb_strategy=1 and vratetol=32000, 
but I think those opts should be used because they will give each test more 
room to work, and give you a clearer idea about what each lavcopt does 
without limiting it with bitrate too much, but still working out if the gain 
in quality justifies the gain in filesize, and where this would apply best in 
your encoding.

===========================
== More technical about script: ==
===========================

Varibles:
---------
cmd        -- is the general command ran for all tests. 
lavcopts  -- each lavcopt I want to test in combination with the general cmd.
preopts   -- a preopt that is added to EACH lavcopt to repeat entire test.


Number of tests:
------------------
So what we end up is: 99 lavcopts
			       3 preopts for each lavcopt
			       4 files (2 pass each)
			       2 vbitrate settings (900kbit ad 1600kbit)
				= 2376 tests

Not too many, but seems quite comprehensive. Just to explain in words, 99 
lavcopts will be tested with 3 preopts for EACH lavcopt. Those 297 tests will 
be tested for 4 files with 2 different video bitrate settings (once with 
900kbit and once with 1600kbit), this makes a total of 2376 tests.


Error handeling:
-----------------
Also, my python script ignores mencoder errors, it will just put details in a 
log file and move on, so then we can also see what args failed without 
disturbing the test. And whatever preopt options that are identical to the 
lavcopts are also ignored, so mdb=2 wont run twice and wont be set for mbd=1, 
etc.

Graph generation:
--------------------
Script is still in the early stages, and I'm not sure what to use to do the 
graphs just yet. I could use webalizer, and I heard of something similar in 
python as a module. So cant comment much about that so far, if you have any 
information/suggestions, please let me know.


Actual tests I will run:
-----------------------
Bellow is the cmd, preopts, lavcopts, please help me review them, add more 
combinations, and give me any other suggestions to make a more fair 
comprehensive test.

cmd="mencoder %s %s -nosound -ovc lavc -lavcopts 
vcodec=mpeg4:vbitrate=%s:psnr:autoaspect:vb_strategy=1:vratetol=32000:vpass=%s:
%s -o videoout/%s-%s-%s'             % (x,z,y,i+1,opt,n,x,y)"

preopts="mbd=2:v4mv,trell".split(',')

lavcopts="v4mv,naq,qpel,alt,ilme,cbp,mv0,qprd,vqmod_amp,vqmod_frez,\
vqsquid=0,last_pred=a,\
vqmin=1:lmin=1,vqmin=3:lmin=3,vqmin=5:lmin=5,vqmin=30:lmin=30,\
lmax=1,lmax=10,lmax=50,lmax=100,\
mbqmin=1,mbqmin=3,mbqmin=10,\
vqdiff=1,vqdiff=2,vqdiff=4,vqdiff=7,vqdiff=10,\
vmax_b_frames=1,vmax_b_frames=2,vmax_b_frames=3,vmax_b_frames=4,\
vme=1,vme=2,vme=3,vme=4,vme=5,\
mbd=1,mbd=2,\
keyint=50,keyint=100,keyint=150,\
sc-threshold=-10000,sc_threshold=-1000,sc_threshold=-100,\
sc_threshold=100,sc_threshold=1000,sc_threshold=10000,\
vb_qfactor=-3,vb_qfactor=0,vb_qfactor=3,\
vi_qfactor=-1,vi_qfactor=0,vi_qfactor=1.5,\
vb_qoffset=-1,vb_qoffset=0,vb_qoffset=2.5.\
vqblur=0,vqblur=3,\
vqcomp=0,vqcomp=1,vqcomp=3,\
vq_init_cplx=1000,vq_init_cplx=100,vq_init_cplx=0,\
tcplx_mask=0.1,tcplx_mask=0.5,tcplx_mask=1,\
scplx_mask=0,scplx_mask=0.1,scplx_mask=0.5,scplx_mask=1.0,\
ildct,ildct:ildctcmp=100,\
precmp=2:cmp=2:subcmp=2,precmp=3:cmp=3:subcmp=3,precmp=5:cmp=5:sumcmp=5,\
predia=-5,predia=1,predia=5,\
dia=-3,dia=-2,dia=2,dia=3,dia=5,\
trell,trell:qns=3,trell:qns=2,trell:qns=1,\
preme=0,preme=2,\
subq=1,subq=4,\
nr=1,nr=5,nr=10,nr=100,nr=1000,nr=10000,nr=500000".split(',')


Code of script?
----------------
I will post code of script when it is done, currently it does everything but 
draw graphs, and the error logging isnt very good at the moment. So give me 
some time to work on it.