keepalived provides a strong and robust health checking for LVS clusters. It nginx implements a framework of health checking on multiple layers for server failover, and VRRPv2 stack to handle director failover. How do I install and configure Keepalived for reverse proxy server such as nginx or lighttpd?

If your are using a LVS director to loadbalance a server pool in a production environment, you may want to have a robust solution for healthcheck & failover. This will also work with reverse proxy server such as nginx.

lb0 – Linux box directly connected to the Internet via eth1. This is master load balancer.
lb1 – Linux box directly connected to the Internet via eth1. This is backup load balancer. This will become active if master networking failed.

202.54.1.1 – This ip moves between lb0 and lb1 server. It is called virtual IP address and it is managed by keepalived.
eth0 is connected to LAN and all other backend software such as Apache, MySQL and so on.

You need to install the following softwares on both lb0 and lb1:

keepalived for IP failover.
iptables to filter traffic
nginx or lighttpd revers proxy server.

DNS settings should be as follows:

hackadmin.in – Our sample domain name.
lb0.hackadmin.in – 202.54.1.11 (real ip assigned to eth1)
lb1.hackadmin.in – 202.54.1.12 (real ip assigned to eth1)
www.hackadmin.com – 202.54.1.1 (VIP for web server) do not assign this IP to any interface.

Install Keepalived

Visit keepalived.org to grab latest source code. You can use the wget command to download the same (you need to install keepalived on both lb0 and lb1):

# cd /opt

# wget http://www.keepalived.org/software/keepalived-1.1.19.tar.gz

#  tar -zxvf keepalived-1.1.19.tar.gz

# cd keepalived-1.1.19

Install Kernel Headers

You need to install the following packages:

Kernel-headers – includes the C header files that specify the interface between the Linux kernel and userspace libraries and programs. The header files define structures and constants that are needed for building most standard programs and are also needed for rebuilding the glibc package.
kernel-devel – this package provides kernel headers and makefiles sufficient to build modules against the kernel package.

Make sure kernel-headers and kernel-devel packages are installed. If not type the following install the same:

Compile keepalived

Type the following command:
# ./configure –with-kernel-dir=/lib/

This article explains the process that will allow you to recover a lost MySQL password:

Stop the MySQL server process

Start the MySQL (mysqld) server/daemon process with the
–skip-grant-tables option so that it will not prompt for password.

Connect to mysql server as the root user.

Setup new mysql root account password.

Exit and restart the MySQL server.

Example:

# service mysqld stop

Output:

Stopping MySQL database server: mysqld.

Then start MySql in safe mode

# mysqld_safe –skip-grant-tables

Output

[1] 5988
Starting mysqld daemon with databases from /var/lib/mysql

Then connect the mysql without any password

# mysql -u root

( Then setup password )

mysql> use mysql;

mysql> update user set password=PASSWORD(”NEW-ROOT-PASSWORD”) where User=’root’;

How do I redirect 80 port to 8123 using iptables?

You can easily redirect incoming traffic by inserting rules into PREROUTING chain of the nat table. You can set destination port using the REDIRECT target.

Syntax

The syntax is as follows to redirect tcp $srcPortNumber port to $dstPortNumber:

iptables -t nat -A PREROUTING -i eth0 -p tcp –dport $srcPortNumber -j REDIRECT –to-port $dstPortNumbe

The syntax is as follows to redirect udp $srcPortNumber port to $dstPortNumber:

iptables -t nat -A PREROUTING -i eth0 -p udp –dport $srcPortNumber -j REDIRECT –to-port $dstPortNumber

Replace eth0 with your actual interface name. The following syntax match for source and destination ips:

iptables -t nat -I PREROUTING –src $SRC_IP_MASK –dst $DST_IP -p tcp –dport $portNumber -j REDIRECT –to-ports $rediectPort

Examples:

In The following example redirects TCP port 25 to port 2525:

# iptables -t nat -A PREROUTING -i eth0 -p tcp –dport 25 -j REDIRECT –to-port 2525

this example all incoming traffic on port 80 redirect to port 8123

# iptables -t nat -I PREROUTING –src 0/0 –dst 192.168.1.5 -p tcp –dport 80 -j REDIRECT –to-ports 8123

How Do I View NAT Rules?

Type the following command:

# iptables -t nat -L -n -v

How Do I Save NAT Redirect Rules?

Type the following command:

# iptables-save

Apache Performance Modules

Apache is a powerful and widely-used World-Wide Web (Web) server. One of its strengths is that the modules that it is made of are customizable according to the user’s requirements. Ashish Kumar discusses the benefits and the process of customization, along with a brief introduction to some useful modules.

List of Standard Modules

This appendix (alphabetically) lists of all of the standard modules that are part of the current (version 1.3.x) Apache distribution. Table 1 the modules that are compiled-in by default and Table 2 lists the ones which are not.

Table 1. Apache Standard Modules Compiled-In by Default.

Table 2. Apache Standard Modules Not Compiled-In by Default.

Appendix II : List of Nonstandard Modules

This appendix is a list of some nonstandard Apache modules. The selection is biased towards modules for programming language support and Web site administration. See Table 3.

How do I restrict the number of connections used by a single IP address to my server for port 80 and 25 using iptables?

You need to use the connection limit modules which allows you to restrict the number of parallel TCP connections to a server per client IP address (or address block). This is useful to protect your server or vps box against flooding, spamming or content scraping.

Syntax
The syntax is as follows:

# /sbin/iptables -A INPUT -p tcp –syn –dport $port -m connlimit –connlimit-above N -j REJECT –reject-with tcp-reset

save the changes see iptables-save man page, the following is redhat and friends specific command service iptables save

Example: Limit SSH Connections Per IP / Host

Only allow 3 ssh connections per client host:

# /sbin/iptables  -A INPUT -p tcp –syn –dport 22 -m connlimit –connlimit-above 3 -j REJECT

save the changes see iptables-save man page, the following is redhat and friends specific command service iptables save

Example: Limit HTTP Connections Per IP / Host

Only allow 20 http connections per IP (MaxClients is set to 60 in httpd.conf):

# /sbin/iptables -A INPUT -p tcp –syn –dport 80 -m connlimit –connlimit-above 20 -j REJECT –reject-with tcp-reset
save the changes see iptables-save man page, the following is redhat and friends specific command service iptables save

Skip proxy server IP 1.2.3.4 from this kind of limitations:

# /sbin/iptables -A INPUT -p tcp –syn –dport 80 -d ! 1.2.3.4 -m connlimit-above 20 -j REJECT –reject-with tcp-reset

Enjoy it….

Storage resource located on an iSCSI server known as a “target”. An iSCSI target usually represents nothing but hard disk storage. As with initiators, software to provide an iSCSI target is available for most mainstream operating systems.

iSCSI initiator (client)

An initiator functions as an iSCSI client. An initiator typically serves the same purpose to a computer as a SCSI bus adapter would, except that instead of physically cabling SCSI devices (like hard drives and tape changers), an iSCSI initiator sends SCSI commands over an IP network.

Debian / Ubuntu Linux Install tgt

Type the following command to install Linux target framework user-space tools:

$ sudo apt-get install tgt

CentOS / RHEL / Red Hat Linux Install tgt

RHEL 5.2 and older version do not have tgt tools. However, RHEL 5.3 (preview version) comes with tgt tools.

tgtadm – Linux SCSI Target Administration Utility

tgtadm is used to monitor and modify everything about Linux SCSI target software: targets, volumes, etc. This tool allows a system to serve block-level SCSI storage to other systems that have a SCSI initiator. This capability is being initially deployed as a Linux iSCSI target, serving storage over a network to any iSCSI initiator.

Start tgtd

To start the tgtd, enter:

# /usr/sbin/tgtd

Under RHEL 5.3 to start the tgtd service, enter:

# /etc/init.d/tgtd start

Define an iscsi target name

The following example creates a target with id 1 (the iqn is 19 iqn.2001-04.com.example:storage.disk2.amiens.sys1.xyz) and adds a 20 logical unit (backed by /dev/hdc1)with lun 1.

# tgtadm –lld iscsi –op new –mode target –tid 1 -T iqn.2001-04.com.example:storage.disk2.amiens.sys1.xyz
To view the current configuration, enter:

# tgtadm –lld iscsi –op show –mode target

Sample output:

Target 1: iqn.2001-04.com.example:storage.disk1.amiens.sys1.xyz

System information:

Driver: iscsi

Status: running

I_T nexus information:

LUN information:

LUN: 0

Type: controller

SCSI ID: deadbeaf1:0

SCSI SN: beaf10

Size: 0

Online: No

Poweron/Reset: Yes

Removable media: No

Backing store: No backing store

Account information:

ACL information:

Add a logical unit to the target (/dev/sdb1):

# tgtadm –lld iscsi –op new –mode logicalunit –tid 1 –lun 1 -b /dev/sdb1

Note:- about home computer / test system

Most production boxes will only use iSCSI root with real iSCSI devices, but for testing purposes it can be quite useful to set up an iSCSI target on your image server. This is useful for testing and learning iSCSI target and iSCSI initiator at home, simply use filesystem for testing purpose. Use dd command to create diskbased filesystem:

# dd if=/dev/zero of=/fs.iscsi.disk bs=1M count=512

Add /fs.iscsi.disk as a logical unit to the target:

# tgtadm –lld iscsi –op new –mode logicalunit –tid 1 –lun 1 -b /fs.iscsi.disk

Now, you should able to view details:

# tgtadm –lld iscsi –op show –mode target

Sample output:

Target 1: iqn.2001-04.com.example:storage.disk1.amiens.sys1.xyz

System information:

Driver: iscsi

Status: running

I_T nexus information:

LUN information:

LUN: 0

Type: controller

SCSI ID: deadbeaf1:0

SCSI SN: beaf10

Size: 0

Online: No

Poweron/Reset: Yes

Removable media: No

Backing store: No backing store

LUN: 1

Type: disk

SCSI ID: deadbeaf1:1

SCSI SN: beaf11

Size: 512M

Online: Yes

Poweron/Reset: Yes

Removable media: No

Backing store: /fs.iscsi.disk

Account information:

ACL information:

Accept iSCSI Target

To enable the target to accept any initiators, enter:

# tgtadm –lld iscsi –op bind –mode target –tid 1 -I ALL

This should open network port # 3260:

# netstat -tulpn | grep 3260

Sample output:

tcp 0 0 0.0.0.0:3260 0.0.0.0:* LISTEN 27328/tgtd
tcp6 0 0 :::3260 :::* LISTEN 27328/tgtd

And you are done. Your system is configured as iSCSI Target. Remote client computer can access this computers hard disk over network. Your can use cluster aware filesystem to setup real shared storage for small business. Open TCP port 3260 in your firewall, if required.

For Client
Install Required Package

iscsi-initiator-utils RPM package – The iscsi package provides the server daemon for the iSCSI protocol, as well as the utility programs used to manage it. iSCSI is a protocol for distributed disk access using SCSI commands sent over Internet Protocol networks. This package is available under Redhat Enterprise Linux / CentOS / Fedora Linux and can be installed using yum command:

# yum install iscsi-initiator-utils

A note about Debian / Ubuntu Linux

If you are using Debian / Ubuntu Linux install open-iscsi package, enter:

$ sudo apt-get install open-iscsi

iSCSI Configuration

There are three steps needed to set up a system to use iSCSI storage:

1. iSCSI startup using the init script or manual startup. You need to edit and configure iSCSI

via /etc/iscsi/iscsid.conf file

2. Discover targets.

3. Automate target logins for future system reboots.

4. You also need to obtain iSCSI username, password and storage server IP address (target host)

Step # 1: Configure iSCSI

Open /etc/iscsi/iscsid.conf with vi text editor:

# vi /etc/iscsi/iscsid.conf

Setup username and password:

node.session.auth.username = My_ISCSI_USR_NAME

node.session.auth.password = MyPassword

discovery.sendtargets.auth.username = My_ISCSI_USR_NAME

discovery.sendtargets.auth.password = MyPassword

Where,

* node.session.* is used to set a CHAP username and password for initiator authentication by the target(s).
* discovery.sendtargets.* is used to set a discovery session CHAP username and password for the initiator authentication by the target(s)

You may also need to tweak and set other options. Refer to man page for more information. Now start the iscsi service:

# /etc/init.d/iscsi start

Step # 2: Discover targets
Now use iscsiadm command, which is a command-line tool allowing discovery and login to iSCSI targets, as well as access and management of the open-iscsi database. If your storage server IP address is 192.168.1.5, enter:

# iscsiadm -m discovery -t sendtargets -p 192.168.1.5

# /etc/init.d/iscsi restart

Now there should be a block device under /dev directory. To obtain new device name, type:

# fdisk -l

or

# tail -f /var/log/messages

Output:
Oct 10 12:42:20 ora9is2 kernel: Vendor: EQLOGIC Model: 100E-00 Rev: 3.2
Oct 10 12:42:20 ora9is2 kernel: Type: Direct-Access ANSI SCSI revision: 05
Oct 10 12:42:20 ora9is2 kernel: SCSI device sdd: 41963520 512-byte hdwr sectors (21485 MB)
Oct 10 12:42:20 ora9is2 kernel: sdd: Write Protect is off
Oct 10 12:42:20 ora9is2 kernel: SCSI device sdd: drive cache: write through
Oct 10 12:42:20 ora9is2 kernel: SCSI device sdd: 41963520 512-byte hdwr sectors (21485 MB)
Oct 10 12:42:20 ora9is2 kernel: sdd: Write Protect is off
Oct 10 12:42:20 ora9is2 kernel: SCSI device sdd: drive cache: write through
Oct 10 12:42:20 ora9is2 kernel: sdd: unknown partition table
Oct 10 12:42:20 ora9is2 kernel: sd 3:0:0:0: Attached scsi disk sdd
Oct 10 12:42:20 ora9is2 kernel: sd 3:0:0:0: Attached scsi generic sg3 type 0
Oct 10 12:42:20 ora9is2 kernel: rtc: lost some interrupts at 2048Hz.
Oct 10 12:42:20 ora9is2 iscsid: connection0:0 is operational now
/dev/sdd is my new block device.

Step # 3: Format and Mount iSCSI Volume

You can now partition and create a filesystem on the target using usual fdisk and mkfs.ext3 commands:

# fdisk /dev/sdd

# mke2fs -j -m 0 -O dir_index /dev/sdd1

OR

# mkfs.ext3 /dev/sdd1

Tip: If your volume is large size like 1TB, run mkfs.ext3 in background using nohup:

# nohup mkfs.ext3 /dev/sdd1 &

Mount new partition:

# mkdir /mnt/iscsi

# mount /dev/sdd1 /mnt/iscsi

Step #4: Mount iSCSI drive automatically at boot time

First make sure iscsi service turned on at boot time:

# chkconfig iscsi on

Open /etc/fstab file and append config directive:

/dev/sdd1 /mnt/iscsi ext3 _netdev 0 0

How To Set Red hat / CentOS Linux Remote Backup / Snapshot Server

Q. I am using an HP RAID 6 server running RHEL 5.x. I’d like this box to act as a backup server for my other Red Hat DNS and Web server. The server must keep backup in hourly, daily and monthly format. How do I configure my Red Hat / CentOS Linux server as remote backup or snapshot server?

A. rsnapshot is easy, reliable and a good disaster recovery backup solution. It is a remote backup program that uses rsync to take backup snapshots of your filesystems. It uses hard links to save space on disk and offers following features:

• Filesystem snapshot – for local or remote systems.

• Database backup – MySQL backup

• Secure – Traffic between remote backup server is always encrypted using openssh

• Full backup – plus incremental

• Easy to restore – Files can restored by the users who own them, without the root user getting involved.

• Automated backup – Runs in background via cron.

• Bandwidth friendly – rsync used to save bandwidth

Sample setup

• snapshot.example.com – HP box with RAID 6 configured with Red Hat / CentOS Linux ac as backup server for other clients.

• DNS ns1.example.com – Red Hat server act as primary name server.

• DNS ns2.example.com – Red Hat server act as secondary name server.

• www.example.com – Red Hat running Apache web server.

• mysql.example.com – Red Hat mysql server.

Install rsnapshot

Login to snapshot.example.com. Download rsnapshot rpm file, enter: WARNING! These examples only works on Red hat / CentOS / Suse / RHEL / Fedora Linux. See Debian / Ubuntu Linux backup server instructions here.

# cd /tmp

# wget http://www.rsnapshot.org/downloads/rsnapshot-1.3.0-1.noarch.rpm

# wget http://www.rsnapshot.org/downloads/rsnapshot-1.3.0-1.noarch.rpm.md5

Verify rpm file for integrity, enter

# md5sum -c rsnapshot-1.3.0-1.noarch.rpm.md5

Sample output:
rsnapshot-1.3.0-1.noarch.rpm: OK
Install rsnapshot, enter:

# rpm -ivh rsnapshot-1.3.0-1.noarch.rpm
Sample output:
Preparing… ########################################### [100%]
1:rsnapshot ########################################### [100%]

Configure rsnapshot

You need to perform following steps

Step # 1: Configure passwordless login

To perform remote backup you need to setup passwordless login using openssh. Create ssh rsa key and upload them to all servers using scp (note you are overwriting ~/ssh/authorized_keys2 files).You need to type following commands on snapshot.example.com server:

# ssh-keygen -t rsa

# scp .ssh/id_rsa.pub root@ns1.example.com:.ssh/authorized_keys2

# scp .ssh/id_rsa.pub root@ns2.example.com:.ssh/authorized_keys2

# scp .ssh/id_rsa.pub root@www.example.com:.ssh/authorized_keys2

# scp .ssh/id_rsa.pub root@mysql.example.com:.ssh/authorized_keys2

Step # 2: Configure rsnapshot

The default configuration file is located at /etc/rsnapshot.conf. Open configuration file using a text editor, enter:

# vi /etc/rsnapshot.conf

Configuration rules

You must follow two configuration rules:

• rsnapshot config file requires tabs between elements.

• All directories require a trailing slash. For example, /home/ is correct way to specify directory, but /home is wrong.

First, specify root directory to store all snapshots such as /snapshots/ or /dynvol/snapshot/ as per your RAID setup, enter:

snapshot_root /raiddisk/snapshots/

You must separate snapshot_root and /raiddisk/snapshots/ by a [tab] key i.e. type snapshot_root hit [tab] key once and type /raiddisk/snapshots/.

Define snapshot intervals

You need to specify backup intervals i.e. specify hourly, daily, weekly and monthly intervals:

interval hourly 6

interval daily 7

interval weekly 4

interval monthly 3

The line “interval hourly 6″ means 6 hourly backups a day. Feel free to adapt configuration as per your backup requirements and snapshot frequency.

Remote backup directories

To backup /var/named/ and /etc/ directory from ns1.example.com and ns2.example.com, enter:

backup root@ns1.example.com:/etc/ ns1.example.com/

backup root@ns1.example.com:/var/named/ ns1.example.com/

backup root@ns2.example.com:/etc/ ns2.example.com/

backup root@ns2.example.com:/var/named/ ns2.example.com/

To backup /var/www/, /var/log/httpd/ and /etc/ directory from www.example.com, enter

backup root@www.example.com:/var/www/ www.example.com/

backup root@www.example.com:/etc/ www.example.com/

backup root@www.example.com:/var/log/httpd/ www.example.com/

To backup mysql database files stored at /var/lib/mysql/, enter:

backup root@mysql.example.com:/var/lib/mysql/ mysql.example.com/dbdump/Save and close the file. To test your configuration, enter:

# rsnapshot configtest

Sample output:

Syntax OK

Schedule cron job

Create /etc/cron.d/rsnapshot cron file. Following values used correspond to the examples in
#vim /etc/rsnapshot.conf.

0 */4 * * * /usr/bin/rsnapshot hourly

50 23 * * * /usr/bin/rsnapshot daily

40 23 * * 6 /usr/bin/rsnapshot weekly

30 23 1 * * /usr/bin/rsnapshot monthly

Save and close the file. Now rsnapshot will work as follows to backup files from remote boxes:

1. 6 hourly backups a day (once every 4 hours, at 0,4,8,12,16,20)

2. 1 daily backup every day, at 11:50PM

3. 1 weekly backup every week, at 11:40PM, on Saturdays (6th day of week)

4. 1 monthly backup every month, at 11:30PM on the 1st day of the month

How do I see backups?

To see backup change directory to

# cd /raiddisk/snapshots/

# ls -l

Sample output:
drwxr-xr-x 4 root root 4096 2008-07-04 06:04 daily.0
drwxr-xr-x 4 root root 4096 2008-07-03 06:04 daily.1
drwxr-xr-x 4 root root 4096 2008-07-02 06:03 daily.2
drwxr-xr-x 4 root root 4096 2008-07-01 06:02 daily.3
drwxr-xr-x 4 root root 4096 2008-06-30 06:02 daily.4
drwxr-xr-x 4 root root 4096 2008-06-29 06:05 daily.5
drwxr-xr-x 4 root root 4096 2008-06-28 06:04 daily.6
drwxr-xr-x 4 root root 4096 2008-07-05 18:05 hourly.0
drwxr-xr-x 4 root root 4096 2008-07-05 15:06 hourly.1
drwxr-xr-x 4 root root 4096 2008-07-05 12:06 hourly.2
drwxr-xr-x 4 root root 4096 2008-07-05 09:05 hourly.3
drwxr-xr-x 4 root root 4096 2008-07-05 06:04 hourly.4
drwxr-xr-x 4 root root 4096 2008-07-05 03:04 hourly.5
drwxr-xr-x 4 root root 4096 2008-07-05 00:05 hourly.6
drwxr-xr-x 4 root root 4096 2008-07-04 21:05 hourly.7
drwxr-xr-x 4 root root 4096 2008-06-22 06:04 weekly.0
drwxr-xr-x 4 root root 4096 2008-06-15 09:05 weekly.1
drwxr-xr-x 4 root root 4096 2008-06-08 06:04 weekly.2

How do I restore backup?

Let us say you would like to restore a backup for www.example.com. Type the command as follows (select day and date from ls -l output):

# cd /raiddisk/snapshots/
# ls -l

# cd hourly.0/www.example.com/

# scp -r var/www/ root@www.example.com:/var/www/

# scp -r etc/httpd/ root@www.example.com:/etc/httpd/

How do I exclude files from backup?

To exclude files from backup, open rsnapshot.conf file and add following line:

exclude_file /etc/rsnapshot.exclude.www.example.com

Create /etc/rsnapshot.exclude.www.example.com as follows:

/var/www/tmp/

/var/www/*.cache

That’s It!

For example, if you are adding routes where the route’s gateway will be on the network in use on eth0, you will create the following file: route-eth0

In the file add the following parameters:

GATEWAY0=192.168.195.2
NETMASK0=255.255.255.0
ADDRESS0=10.0.0.0

For each subsequent route statement increment the number that is appended to each parameter.

yum -y install net-snmp

Then copy over my default snmpd.conf file and fire it up.

In these instances however, snmpd either refused to start or gave the following error in /var/log/messages:

Apr 16 18:14:35 msx-db01 kernel: audit(1208384075.250:6): avc: denied { read } for pid=27712 comm="snmpd" name="snmpd.conf" dev=sda3 ino=7241736 scontext=root:system_r:snmpd_t tcontext=root:object_r:tmp_t tclass=file
Apr 16 18:14:35 msx-db01 kernel: audit(1208384075.265:7): avc: denied { read } for pid=27712 comm="snmpd" name="snmpd.conf" dev=sda3 ino=7241736 scontext=root:system_r:snmpd_t tcontext=root:object_r:tmp_t tclass=file
Apr 16 18:14:35 msx-db01 snmpd[27712]: Warning: no access control information configured. It's unlikely this agent can serve any useful purpose in this state. Run "snmpconf -g basic_setup" to help you configure the snmpd.conf file for this agent.
Apr 16 18:14:35 msx-db01 snmpd[27712]: NET-SNMP version 5.1.2


I tried to use snmpconf as suggested even though I knew my snmpd.conf file was fine. snmpconf was not on the system so I didn’t waste any time trying to find out how to get it.

So the fix is to install the net-snmp-libs package, not sure what it updates but it works.

yum -y install net-snmp-libs

The catch after that is (at least in all 3 of my situations) the following error on the yum install:

Transaction Check Error: file /usr/share/man/man8/ext2online.8.gz from install of e2fsprogs-1.35-12.11.el4_6.1 conflicts with file from package e2fsprogs-1.35-12.4.EL4

So, back to the hack admin package manager of choice *yum* :

yum update e2fsprogs

Once it’s updated, the net-snmp-libs goes in as expected.

So what’s the deal with all this? I dunno, what the hell do utilities for ext2 have to do with my ability to run snmp monitoring? Who gives a shit and if you know, you’re probably on the wrong site. At any rate, my monitoring works and I’ll leave the rest to the server fairies.

Moral of the story: If your shit works, do you really need to understand it?