Much Ado About Everything

If you want to encode video on Linux, you’re in for a world of pain. Let’s compare encoding a video file to crossing a river. Expect to have to generally learn how to build a bridge to cross to the other side – and then build it. As opposed to – at the very worst – following the instructions on how to build a bridge if using Windows, or more likely just walk right across it, as someone else probably built it and left it there already. In Linux, someone probably built the bridge before you too, but they either built it right for their specifications or they teared it down so for anyone coming afterwards it’s as it was never built in the first place.

In other words, expect to have to compile stuff yourself, apply patches and spend many hours if not days experimenting rather than being productive.

The attraction is that you hopefully can use 64-bit operating system and applications and utilize the full power of the processor, including multithreading. In practice though, things are not what they seem.

There is only one program that will actually encode a video to H.264: it’s called X264. That’s not a problem – it’s the best of its kind, comparing favourably to commercial software, and is what Windows programs use too. The problem is that if you have a multicore cpu, it will likely not utilize it nowhere near its full potential. For dual core machines differences are not drastic and is probably the reason why it slipped under the radar, but for a quad core it is very important to apply this patch. Read here why – if you have patience. I am uncertain whether this problem affects Windows – but the author mentioned that his problems occur in Linux only, so it could be an issue with how the X.264 author coded threading for Linux. Windows users score another one.

That’s not all though. There are more patches that affect the video quality. In the video I encoded there were noticeable blocking artifacts in the dark background, making the scene look ugly and last generation (macroblocks were endemic on mpeg1 and mpeg2, especially if the bitrate was low). Unfortunately there is no easy solution other than increasing the bitrate too high for my taste (say to 3000kb/s for this particular sequence – I’d prefer 2000kb/s or even less). There is however a fairly well accepted solution – adaptive quantization which can be added with this AQ quality patch. This patch adds adaptive quntization options to the command: –aq-strength and –aq-sensitivity . Alas, not even this one is without problems and needs a patching of its own. However, while this patch works if you’re using X.264 directly, it is useless if the front end doesn’t support it, that being ffmpeg or mencode. Unfortunately that makes this patch only a step above useless, unless you’re willing to complicate your life yet even more. Here’s how you would go about doing that.  I will provide an example as well, later on.
However, all of these are (probably) already compiled into Windows programs, thanks to their agile development teams and large communities. This is the trouble with Linux. With Windows someone else is going to do it for you. With Linux, it’s all up to you. It used to be the case that the tradeoff was worth it – you would gain a lot of flexibility and power by moving to Linux and learning how to do things yourself. This is not the case any more – if you use meGUI or some other tool, you’ll have no less flexibility yet much more convenience – and perhaps even speed and quality.

It is rare to use X.264 directly. That’s because it’s an encoder, not a mixer and not a general purpose tool. It expects its data to be in raw YUV format and it will output it unwrapped (as in, not in a container). It does not deal with any kind of audio (obviously), nor it knows anything about mp4 container which is where you want the output to end up packaged into. Therefore you will be using either ffmpeg or mencoder. Those aren’t front ends but kind of a swiss knife tools that can (de)multiplex video and audio from and to various containers, use filters to resize and pre- or postprocess, deinterlace, pulldown or pullup and so on.

Unfortunately neither of these tools is the “one size fits all”. That isn’t a problem either as diversity is good. The real problem is: each of those damn tools uses its own syntax to define parameters that will be used with X.264! Yes, you read it right: even though all they do is pass those parameters to underlying X.264, each of them has its own way of defining it. Not only that, but for many parameters it is not obvious which parameter in one tool corresponds to the one in other tool! Tell me that there is a good reason that an option called “bime” in mencoder needs to be called “bidir_refine 1″ in ffmpeg! Especially since neither project is using its own implementation of the standard – they both will translate and pass the same thing to X.264. Unbelievable! It’s a stunning display of worst that open source can deviate into – an anarchy, a number of groups that each goes its separate way, oblivious – or worse – to others doing the same thing. Whatever the reason, the end result is even more suffering on top of pain already inflicted with this complicated subject matter.

Anyway, the fact that you had to compile x.264 unfortunately means that you’ll have to recompile mencoder and ffmpeg too. No thanks to seemingly hard linked dependencies. It’s not enough to just disable the system-supplied X.264 and install your own; the tools that depend on it will stubbornly want the other version. Whatever happened to dynamic libraries? Oh, who knows, in the end you’ll have to compile stuff yourself anyway and it’s probably for the best since it’s likely whoever compiled the os-supplied versions disabled a bunch of features that you’ll need.

More to come, but I post this anyway to make sure I don’t forget or lose these commands.

Here is a command for two pass encoding which creates a PSP playable file. You can use it if you have say an anime fansub in an avi file. Adjust the frame rate if necessary. It will produce a big file, possible a little bigger than the one you started with! The quality will be excellent but it will be overkill – H.264 is more efficient than even XviD so it makes no sense to use higher bitrate than you began with, especially since you’re probably resizing the picture down!

ffmpeg -i $1 -y -r 23.98 -s 480×272 -vcodec libx264 -acodec libfaac -f mp4 -b 1000k -maxrate 4000k -coder ac -flags loop -flags2 +wpred+mixed_refs+brdo -level 21 -subq 6 -refs 2 -bf 3 -bidir_refine 1 -ac 2 -ab 160k -trellis 1 -threads 0 -pass 1 -partitions +partp8x8+partb8x8+parti4x4+partp4x4-parti8x8 $1.mp4

ffmpeg -i $1 -y -r 23.98 -s 480×272 -vcodec libx264 -acodec libfaac -f mp4 -b 1000k -maxrate 4000k -coder ac -flags loop -flags2 +wpred+mixed_refs+brdo -level 21 -subq 6 -refs 2 -bf 3 -bidir_refine 1 -ac 2 -ab 160k -trellis 1 -threads 0 -pass 2 -partitions +partp8x8+partb8x8+parti4x4+partp4x4-parti8x8 $1.mp4

And here is the command that can do that in one pass and will produce much smaller file. Change crf to higher value to compress more, or lower value to compress less. Value of 20 is typical.

ffmpeg -i “$1″ -y -r 23.98 -s 480×272 -vcodec libx264 -acodec libfaac -f mp4 -maxrate 4000k -coder ac -flags loop -flags2 +wpred+mixed_refs+brdo -level 21 -subq 6 -refs 2 -bf 3 -bidir_refine 1 -ac 2 -ab 128k -trellis 1 -threads 0 -crf 22 -partitions +partp8x8+partb8x8+parti4x4+partp4x4-parti8x8 -t “$1″ “$1″.mp4

There a few things to remember regarding PSP encoding: you can use up to 3 B frames, up to 2 reference frames and you cannot use B Pyramid. You must not use adaptive spatial transform 8×8 matrix (a High profile feature) nor I 8×8 partition. You also need to specify frame rate (in the example above it’s 23.98) because your video will not play if it’s longer than few minutes. You also need to take care to set the level appropriately (I believe you can use level 3.0 if you want to use higher 720×480 resolution though I can’t see a good reason to do so, other than perhaps ability to use the same file on PS3). And be sure to use one of newer firmwares (3.30 or higher) as some features weren’t supported in earlier versions.

On the other hand, to encode a ripped DVD file with two-pass for PS3 use the command below. This page provides details about the video acquisition (ripping, deinterlacing etc). In the case below, I had an anamorphic 1:2.35 movie shot in 24fps and provided at circa 60Hz with 3:2 pulldown. This is probably going to be most common case if you watch a lot of mainstream movies. If you do have special needs and say need to deal with deinterlacing, read here. Anyhow, adjust the crop parameters as per this guide and adjust map parameters to use the audio stream that you want (e.g. english 5.1 DTS). Use mplayer to find out what is what.
ffmpeg -map 0:0 -map 0:1 -i input.mpg -y -croptop 62 -cropbottom 66 -cropleft 0 -cropright 0 -padtop 62 -padbottom 66 -vcodec libx264 -acodec libfaac -title “Random DVD” -f mp4 -coder ac -b 2000k -bufsize 10000k -maxrate 25000k -g 300 -level 41 -ac 2 -ab 172k -flags2 dct8x8+bpyramid+brdo+wpred+mixed_refs -refs 5 -bf 3 -me umh -subq 6 -trellis 0 -directpred auto -flags +loop -deblockalpha -2 -deblockbeta -1 -partitions +partp8x8+partb8x8+parti4x4+parti8x8 -threads auto -pass 1 output.mp4

ffmpeg -i input.mpg -y -map 0:0 -map 0:1 -y -croptop 62 -cropbottom 66 -cropleft 0 -cropright 0 -padtop 62 -padbottom 66 -vcodec libx264 -acodec libfaac -title “Random DVD” -f mp4 -coder ac -b 2000k -bufsize 10000k -maxrate 25000k -g 300 -level 41 -ac 2 -ab 172k -flags2 dct8x8+bpyramid+brdo+wpred+mixed_refs -refs 5 -bf 3 -me umh -subq 6 -trellis 0 -directpred auto -flags +loop -deblockalpha -2 -deblockbeta -1 -partitions +partp8x8+partb8x8+parti4x4+parti8x8 -threads auto -pass 2 output.mp4

The example below might not be as good as it can be – you could end up with some macroblocking in dark areas. As mentioned before, you might want to use adaptive quantization to deal with that, unfortunately that means using X264 directly.  The script below will do that.

rm tmp.fifo.y4m
mkfifo tmp.fifo.y4m
ffmpeg -y -i input.mpg -croptop 62 -cropbottom 66 -cropleft 0 -cropright 0 -map 0:0 -f yuv4mpegpipe -t 50 -an tmp.fifo.y4m & x264 -o output.mp4 –fps 59.94 –threads auto –progress –ref 5 –bframes 3 –b-pyramid –level 4.1 –sar 1:1 –b-rdo –mixed-refs –8x8dct –partitions all –direct auto –weightb –trellis 0 –aq-strength 0.8 –aq-sensitivity 17 –crf 20 tmp.fifo.y4m 720×352

The script will only deal with video – you will need to extract audio track separately (using mencode or ffmpeg) and then multiplex them together using mp4box.  Painful, to say the least.

Things to remember about PS3: make sure to set the level to 4.1 or 4.2. A lot of tweaking can be done above, I’ll get on that later.