I stopped relying long ago on RAID-5 after I had two separate installs that the controller card corrupted more than one disk in the array at the same time, causing total data loss.

When you're dealing with more than a terabyte of data, restoring from any sort of backup medium becomes a painful process. All of my data is backed up on DVD (yeah yeah, I've heard the complaints before, but you don't know what I know about DVD backups) but restoring a terabyte of data from DVD's can take a week or so.

Enter DRBD. DRBD stands for Distributed Block Device. Essentially it's RAID-1 that works over Ethernet. DRBD rides on top of whatever physical storage medium and network you have, but below the file system level. You run it on multiple machines, and set up an identical hard drive configuration on each machine. The DRBD partition is automatically replicated from the primary server to the secondary. Using tools like "heartbeat" you can even monitor this system automatically and promote the secondary server to the primary in the event of a failure.

On my last server to upgrade to Hardy, MySQL failed to start. Not sure why it only happened to this server, but trying to start MySQL resulted in:

/etc/init.d/mysql: ERROR: Using expire_logs_days without log_bin crashes the server. See README.Debian.gz

Sure enough, looking in /etc/mysql/my.cnf showed that log_bin was commented out, and expire_logs_days was enabled.

Uncommenting log_bin = /var/log/mysql/mysql-bin.log resolved the issue.

This weekend was the day to upgrade my internal mail server from Feisty to Hardy. I've already done this on several servers and didn't expect any major problems.

Step 1: Upgrade Feisty with all latest packages:

apt-get update;apt-get dist-upgrade

Followed by a reboot. Went perfectly!

Step 2: Upgrade Fesity to Gutsy. If you're more than one version behind, you really have to take it in stages. Replace all occurrences in /etc/apt/sources.list of "feisty" with "gutsy", then:

apt-get update;apt-get dist-upgrade

followed by a reboot. Went mostly perfectly - it overwrote the newer version of eGroupware I'd installed with something that wouldn't run, so I restored my version of eGroupware and made sure everything worked.

Step 3: Upgrade Gutsy to Hardy LTS. Replace all occurrences in /etc/apt/sources.list of "gutsy" with "hardy" then:

apt-get update;apt-get dist-upgrade

Then things went downhill. During installation, I ran into the locales bug. The system froze on the upgrade completely. The fix? You need physical access to the machine so thankfully this wasn't one of my colocated servers.

First, do a hard reboot of the box, because CTRL-C isn't going to unhang it any time soon. During boot, at Grub select the previous kernel version from the boot menu. After it boots, log in as root (I hope you have a root password set!) then continue the upgrade process using:

dpkg --configure -a