So I built all of these instances to give hardware based failover options in case of total failure or need for upgrade. I honestly am not counting on any of these instances to be a seamless failover scenario, but I figure it could work in a crisis and worst case, I have a copy of important data that is ready to go live.

Needless to say, through setting up 4 pairs of servers I got handy with the routine that I knew worked. Here was my process:

DRBD Setup

1.) Kickstart both boxes with the following KS disk layout:

part / –size 10000
part /data –size 5000 –grow
part /boot –size 100
part swap –size 2000

The data slice is what I will share between the DRBD pair.

*NOTE – The next steps – Step 2 through 9 need to be completed on both servers in the HA, DRBD setup.

2.) Download the latest DRBD source.

3.) I wanted to make the install much harder than it actually is, so I spent time on many sites staring at kernel patches and such.. Oh well. When all was said and done, I really just needed to make sure the kernel source was in my kickstart, then run the following from within the source directory:

make clean all
make install
make install-tools

4.) Brew up a config file for DRBD ( there should be a default in /etc/drbd.conf ) :

# The important logicstics as best I can tell.
# Protocol – there are 3 options I chose C as it appeared to be the best to use with mysql
# syncer – this was a head scratcher. When I first tried to sync my disks I had left it at default
# and our 56G chunks were taking about 6 hours to complete. I’m not only on GB interfaces here
# but the servers in question are blades and they share a chassis backplane, so I cranked it up
# 500M is close to the threshold, don’t remember what it was but I hit it when I was being all
# crazy with this setting to start.
# Server Names – in this example, server1 and server2, make sure they are in /etc/host
# Resource – this is going to be used through out the example as a way for the DRBD system to identify t
# the config params that create this disk replication scenario

global {
usage-count yes;
}
common {
protocol C;
syncer { rate 500M; }
}
resource r0 {
on server1 {
device /dev/drbd1;
disk /dev/sda3;
address 192.168.195.124:7789;
meta-disk internal;
}
on server2 {
device /dev/drbd1;
disk /dev/sda3;
address 192.168.195.125:7789;
meta-disk internal;
}
}

5.) Next, I umounted the disk in question, since it was built with kickstart, it came up as a mounted ext3 slice at /data

6.) Then I zeroed out /dev/sda3 ( what was our /data mount ) with the following command:

dd if=/dev/zero bs=1M count=1 of=/dev/sda3 ; sync

7.) Next, we need to create the device that we refer to in the config file /dev/drbd1 . I found a bash line in a search that creates several so I just did it up in case I ever wanted to expand this config:

for i in `seq 0 1` ; do mknod -m 0660 /dev/drbd$i b 147 $i; done

Just change the seq numbers if you want more or less.

8.) Next we use the drbdadm command to create our resource ( r0 ):

drbdadm create-md r0

9.) Start the drbd service

/etc/init.d/drbd start

10.) Pick the server that you want to be the primary in the pair and run the following command:

drbdadm — –overwrite-data-of-peer primary r0

This will create the sync between the 2 disks you can watch the progress of the sync with the following command:

watch cat /proc/drbd

11.) Once the disks are in sync, create a file system on both sides. This where it can get kind of tricky if you’re not sure what it’s doing. We just told server1 to be the primary server in the pair so when you look at /proc/drbd you’ll see the following that verifies this is the case:

1: cs:Connected st:Primary/Secondary ds:UpToDate/UpToDate C r—

I added the following to /etc/fstab so I could mount and make a file system:

/dev/drbd1 /data ext3 defaults 1 2

Then:

mount /data

Then:

/sbin/mkfs -t ext3 /dev/drbd1

12.) Now we are going to create the filesystem on the secondary server. ( I’m not sure this is entirely necessary, but it’s a good exercise in moving the slice around ).

* Unmount the /data slice on the primary server:

umount /data

* Make the primary box ( server1 in our case ) the secondary in the drbd pair:

drbdadm secondary r0

* Make the secondary box ( server2 in our case ) the primary in the drbd pair:

drbdadm primary r0

* Add the /etc/fstab line

/dev/drbd1 /data ext3 defaults 1 2

* Mount the slice:

mount /data

* Create the filesystem:

/sbin/mkfs -t ext3 /dev/drbd1

Once this is complete, use the same series of commands to make server1 the primary, remove all of the /etc/fstab entries for /data ( Linux HA is going to take care of this for us ) and move on to the HA config below.

Linux HA Setup

The goal of this setup is as follows:

a.) Choose a VIP for your HA pair

b.) Build a config file that starts and stops services on startup and failover

HA is pretty simple when all is said and done ( at least in this config ). It comes rolled up with scripts that take care of the following:

- Manage VIP
- Manage DRBD resource
- Mounting file systems
- Starting services

Install

To install in my setup I just ran:

yum -y install heartbeat

Config

Once heartbeat is installed there are 3 files to worry about:

- ha.cf
- haresources
- authkeys

All of these files will live in /etc/ha.d

All of these files must be identical on both servers in the pair.

1.) ha.cf

keepalive 100ms
deadtime 2
warntime 10
initdead 120
bcast eth1 eth0
autojoin none
auto_failback on
node server1 server2

You can play with the values of each of these and there is further information here.

Again in this config ( and this should already be done ) .. make sure you have server1 and server2 in /etc/hosts

2.) haresources

This is basically the command file, it tells HA what to do and in what order. Here is my config file:

Server1 IPaddr::192.168.195.212/24 drbddisk::r0 \
Filesystem::/dev/drbd1::/data::ext3::defaults mysql.server

Breakdown:

Server1 – This will be the same on both primary and secondary, it tells HA who should be in charge.
IPaddr – This identifies the VIP that will be used between the two systems. If you have multiple NICs, it is smart enough to figure out which one it should be a sub of.

* SIDE NOTE – In the haresources file, the syntax basically indicates that you are asking HA to run a script. HA looks for scripts in /etc/ha.d/resource.d and /etc/init.d by default, take the following example:

IPaddr::192.168.195.212/24

This means you are asking HA to run the following:

/etc/ha.d/resource.d/IPaddr 192.168.195.212/24 start

The result will be an added ethernet interface with the VIP for the pair.

* END SIDE NOTE

drbddisk – This tells HA to use the drbddisk command in /etc/ha.d/resource.d to initialize the drbd resource ( r0 in our case ) and make it primary.

Filesystem – This tells HA to mount the drbd resource at /data

mysql.server – This tells HA to use the script in /etc/init.d to fire up mysql

* SIDE NOTE

You can imagine how important the sequence is here. If I were to try to mount /data before I initialize drbd, it’s going to fail. Unfortunately, HA is not real clear on what is going wrong when it’s failing, it just fails. There are log file lines you can use in ha.cf, but everything goes to /var/messages by default so don’t waste your time.

Also, if you are setting up multiple HA pairs that should not know about other pairs, you need to change the following parameters in your ha.cf:

- Remove this: bcast eth1 eth0
- Add this: mcast eth0 224.1.2.6 694 1 0

For each pair you will want to modify the broadcast address so that it is unique, so for the second pair you could use this:

- mcast eth0 224.1.2.7 694 1 0

Changing the “6″ to a “7″ makes sure that pairs aren’t chatting with each other and failing. The result of not doing this is a /var/log/messages file that grows out of control ( I’m not kidding either, with 4 pairs on the same net this thing was growing at about 500M an hour ). It also drives the server load up on the pair from all the logd action, to the tune of load 0 when only one pair to load 2 when 4 pairs on the same net.

* END SIDE NOTE

3.) authkeys

This can be pretty generic. I thought initially changing this secret around would avoid the issue listed in the last side note, it doesn’t. The only catch with this file is that it has to have perms of 600 or HA won’t start.

auth 2
2 sha1 MySecret

Verification

As I have mentioned, the haresources file basically contains script to argument pairs. It is good practice to walk through all of these with a “start”, then go back through with a “stop”. This will ensure that what HA will run will actually work when HA attempts to run them.

Once you are satisfied with the results of your testing, just run /etc/init.d/heartbeat start on each box.  If all goes well, you should see the new VIP on the primary box, the mounted /data slice, and you should be able to log into mysql

* SIDE NOTE

To install mysql I did the following:

- Downloaded preferred tarball
- Changed this in my.cf -  datadir=/data/mysql_data
- Added this to my.cf – bind-address=192.168.195.212 ( This ensures that mysql binds to the VIP on startup)

* END SIDE NOTE

This may be a bit scattered, read it a couple times and it should all be clear.  This is basically a compilation of 2 days worth of poking at this process to find something that consistently worked.

Good Luck