Actually, have a look at the --help output of the x264 binary: -t, --trellis <integer> Trellis RD quantization. Requires CABAC. I didn't check the source, really, but if the x264 guys write it down ...