IP Failover For Web Cluster

22 Feb 2010

Article by Aashish

keepalived provides a strong and robust health checking for LVS clusters. It nginx implements a framework of health checking on multiple layers for server failover, and VRRPv2 stack to handle director failover. How do I install and configure Keepalived for reverse proxy server such as nginx or lighttpd?

If your are using a LVS director to loadbalance a server pool in a production environment, you may want to have a robust solution for healthcheck & failover. This will also work with reverse proxy server such as nginx.

lb0 – Linux box directly connected to the Internet via eth1. This is master load balancer.
lb1 – Linux box directly connected to the Internet via eth1. This is backup load balancer. This will become active if master networking failed.

202.54.1.1 – This ip moves between lb0 and lb1 server. It is called virtual IP address and it is managed by keepalived.
eth0 is connected to LAN and all other backend software such as Apache, MySQL and so on.

You need to install the following softwares on both lb0 and lb1:

keepalived for IP failover.
iptables to filter traffic
nginx or lighttpd revers proxy server.

DNS settings should be as follows:

hackadmin.in – Our sample domain name.
lb0.hackadmin.in – 202.54.1.11 (real ip assigned to eth1)
lb1.hackadmin.in – 202.54.1.12 (real ip assigned to eth1)
www.hackadmin.com – 202.54.1.1 (VIP for web server) do not assign this IP to any interface.

Install Keepalived

Visit keepalived.org to grab latest source code. You can use the wget command to download the same (you need to install keepalived on both lb0 and lb1):

# cd /opt

# wget http://www.keepalived.org/software/keepalived-1.1.19.tar.gz

# tar -zxvf keepalived-1.1.19.tar.gz

# cd keepalived-1.1.19

Install Kernel Headers

You need to install the following packages:

Kernel-headers – includes the C header files that specify the interface between the Linux kernel and userspace libraries and programs. The header files define structures and constants that are needed for building most standard programs and are also needed for rebuilding the glibc package.
kernel-devel – this package provides kernel headers and makefiles sufficient to build modules against the kernel package.

Make sure kernel-headers and kernel-devel packages are installed. If not type the following install the same:

Compile keepalived

Type the following command:
# ./configure –with-kernel-dir=/lib/

modules/$(uname -r)/build

Sample outputs:

checking for gcc… gcc
checking for C compiler default output file name… a.out
checking whether the C compiler works… yes
checking whether we are cross compiling… no
checking for suffix of executables…
checking for suffix of object files… o
…
…..
..
config.status: creating keepalived/check/Makefile
config.status: creating keepalived/libipvs-2.6/Makefile

Keepalived configuration
————————
Keepalived version : 1.1.19
Compiler : gcc
Compiler flags : -g -O2
Extra Lib : -lpopt -lssl -lcrypto
Use IPVS Framework : Yes
IPVS sync daemon support : Yes
Use VRRP Framework : Yes
Use Debug flags : No

Compile and install the same:
# make && make install

Create Required Softlinks

Type the following commands to create service and run it at RHEL / CentOS run level #3 :
# cd /etc/sysconfig
# ln -s /usr/local/etc/sysconfig/keepalived .
# cd /etc/rc3.d/
# ln -s /usr/local/etc/rc.d/init.d/keepalived S100keepalived
# cd /etc/init.d/
# ln -s /usr/local/etc/rc.d/init.d/keepalived .

Configuration

Your main configuration directory is located at /usr/local/etc/keepalived and configuration file name is keepalived.conf. First, make backup of existing configuration:
# cd /usr/local/etc/keepalived
# cp keepalived.conf keepalived.conf.bak
Edit keepalived.conf as follows on lb0:

vrrp_instance VI_1 {
interface eth0
state MASTER
virtual_router_id 51
priority 101
authentication {
auth_type PASS
auth_pass Add-Your-Password-Here
}
virtual_ipaddress {
202.54.1.1/29 dev eth1
}
}

Edit keepalived.conf as follows on lb1 (note priority set to 100 i.e. backup load balancer):

vrrp_instance VI_1 {
interface eth0
state MASTER
virtual_router_id 51
priority 100
authentication {
auth_type PASS
auth_pass Add-Your-Password-Here
}
virtual_ipaddress {
202.54.1.1/29 dev eth1
}
}

Save and close the file. Finally start keepalived on both lb0 and lb1 as follows:
# /etc/init.d/keepalived start

Verify: Keepalived Working Or Not

/var/log/messages will keep track of VIP:
# tail -f /var/log/messages
Sample outputs:

Feb 21 04:06:15 lb0 Keepalived_vrrp: Netlink reflector reports IP 202.54.1.1 added
Feb 21 04:06:20 lb0 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth1 for 202.54.1.1

Verify that VIP assigned to eth1:
# ip addr show eth1
Sample outputs:

3: eth1: mtu 1500 qdisc pfifo_fast qlen 10000
link/ether 00:30:48:30:30:a3 brd ff:ff:ff:ff:ff:ff
inet 202.54.1.11/29 brd 202.54.1.254 scope global eth1
inet 202.54.1.1/29 scope global secondary eth1

ping failover test

Open UNIX / Linux / OS X desktop terminal and type the following command to ping to VIP:
# ping 202.54.1.1
Login to lb0 and halt the server or take down networking:
# halt
Within seconds VIP should move from lb0 to lb1 and you should not see any drops in ping. On lb1 you should get the following in /var/log/messages:

Feb 21 04:10:07 lb1 Keepalived_vrrp: VRRP_Instance(VI_1) forcing a new MASTER election
Feb 21 04:10:08 lb1 Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE
Feb 21 04:10:09 lb1 Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE
Feb 21 04:10:09 lb1 Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.
Feb 21 04:10:09 lb1 Keepalived_healthcheckers: Netlink reflector reports IP 202.54.1.1 added
Feb 21 04:10:09 lb1 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth1 for 202.54.1.1