Hack Admin » Replication

MySQL Server Replication with ssl

admin — Thu, 04 Mar 2010 19:14:12 +0000

via Bhushan

TYPES OF MYSQL REPLICATION

Statement-based Replication
Row-based Replication
Mixed

To change the type of Replication modify my.cnf configuration file and change

binlog_format=mixed | row | statement

mysql> SHOW VARIABLES LIKE ‘binlog_format’;

+—————+——-+

| Variable_name | Value |

+—————+——-+

| binlog_format | MIXED

+—————+——-+

Processes/Threads inside MySQL that are responsible for replication

MASTER - Binlog Dump Thread
SLAVE - I/O Thread
- - - - SQL Thread

Statements useful to check the status of these threads as replication goes:

mysql> SHOW PROCESSLIST\G

mysql> SHOW MASTER STATUS\G

mysql> SHOW SLAVE STATUS\G

Directories and File Locations

Datadir – /var/lib/mysql
General Log dir. - /var/log
Bin Log dir. - /var/log/mysql
Configuration file - /etc/mysql/my.cnf
SSL Certificates - /etc/mysql/ssl
Relay Log file - /var/lib/mysql/slavehost-relay-bin.NNNNNN
Status Files - master.info, relay-log.info

Note:

- All modification/updates to data should be done on Master only, and not on any Slave. Slave should be used for queries.

setup replication

MySQL MASTER = 192.168.1.100:3306

MySQL SLAVE = 192.168.1.111:3306

MASTER host

root@sage:~# mkdir /etc/mysql/ssl

root@sage:~# cd /etc/mysql/ssl/

root@sage:~# rm -rf *

Create CA certificate

root@sage:~# openssl genrsa 2048 > ca-key.pem

root@sage:~# openssl req -new -x509 -nodes -days 1000 -key ca-key.pem > ca-cert.pem

Create server certificate

root@sage:~# openssl req -newkey rsa:2048 -days 1000 -nodes -keyout server-key.pem > server-req.pem

root@sage:~# openssl x509 -req -in server-req.pem -days 1000 -CA ca-cert.pem -CAkey ca-key.pem -set_serial 01 > server-cert.pem

Copy ca-cert file to MySQL clients & slaves

root@sage:~# scp ca-cert.pem root@slave-host-IP:/etc/mysql/ssl/

Modify configuration file

root@sage:~# vi /etc/mysql/my.cnf

Enable Binary logging in Mixed format. And specify a Unique Server ID of Master

[mysqld]

log-bin = /var/log/mysql/mysql-bin

binlog_format = mixed

server-id = 1

ssl-key = /etc/mysql/ssl/server-key.pem

ssl-cert = /etc/mysql/ssl/server-cert.pem

ssl-ca = /etc/mysql/ssl/ca-cert.pem

Test SSL connectivity using MySQL-Client

root@sage:~# /etc/init.d/mysql restart

root@sage:~# mysql --ssl-ca=/etc/mysql/ssl/ca-cert.pem -u root -p

mysql> SHOW VARIABLES LIKE ‘%ssl%’;

+—————+——————————–+

| Variable_name | Value |

+—————+——————————–+

| have_openssl | YES |

| have_ssl | YES |

| ssl_ca | /etc/mysql/ssl/ca-cert.pem |

| ssl_capath | |

| ssl_cert | /etc/mysql/ssl/server-cert.pem |

| ssl_cipher | |

| ssl_key | /etc/mysql/ssl/server-key.pem |

+—————+——————————–+

mysql> SHOW STATUS LIKE ‘Ssl_cipher’;

+—————+——————–+

| Variable_name | Value |

+—————+——————–+

| Ssl_cipher | DHE-RSA-AES256-SHA |

+—————+——————–+

confirms that SSL is supported & enabled on MASTER

Create mysql user on master that has the privileges to do replication.

mysql -u root -p

mysql> GRANT REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO replssl@’%’ IDENTIFIED BY ‘replipass’ REQUIRE SSL;

If user already exists

mysql> GRANT USAGE ON *.* TO 'repl'@'%' REQUIRE SSL;

mysql> FLUSH PRIVILEGES;

mysql> SHOW GRANTS FOR repl;

Find the location where Master is writing now

mysql> show master status;

+—————-+——–+————-+—————+

| File | Position |Binlog_Do_DB |Binlog_Ignore_DB

+—————-+——–+————-+——————

|mysql-bin.000004| 7705 | |

+—————-+——–+————-+—————–

They are: mysql-bin.000004, 7705

Take snapshot of Mysql data on Master and then scp it to slave.

mysql> FLUSH TABLES WITH READ LOCK;

root@sage:~# tar czvf ~/mysql-snapshot.tar.gz /var/lib/mysql

mysql> UNLOCK TABLES;

Copy snapshot to the slave

root@sage:~# scp mysql-snapshot.tar/gz user@slave-IP:~

SLAVE side

To configure this host as a replication slave, you can choose between

two methods :

- Use the CHANGE MASTER TO command

CHANGE MASTER TO MASTER_HOST=, MASTER_PORT=,

MASTER_USER=, MASTER_PASSWORD= ……

- Set the variables in /etc/mysql/my.cnf.

Create client certificate

root@sage:~# openssl req -newkey rsa:2048 -days 1000 -nodes -keyout client-key.pem > client-req.pem

root@sage:~# openssl x509 -req -in client-req.pem -days 1000 -CA ca-cert.pem -CAkey ca-key.pem -set_serial 01 > client-cert.pem

root@sage:~# vi /etc/mysql/my.cnf

[mysqld]

server-id = 2

master-host = 192.168.1.100

master-user = repl

master-password = replipass

master-port = 3306

log-bin = /var/log/mysql/mysql-bin

binlog_format = mixed

tmpdir = /tmp/

[client]

ssl-ca=/etc/mysql/ssl/ca-cert.pem

#ssl-key=/etc/mysql/ssl/client-key.pem

#ssl-cert=/etc/mysql/ssl/client-cert.pem

“If the account has no special SSL requirements or was created using a GRANT statement that includes the REQUIRE SSL option, a client can connect securely by using just the --ssl-ca option:

shell> mysql --ssl-ca=cacert.pem

To require that a client certificate also be specified, create the account using the REQUIRE X509 option. Then the client must also specify the proper client key and certificate files or the server will reject the connection:

shell> mysql --ssl-ca=cacert.pem \
       --ssl-cert=client-cert.pem \
       --ssl-key=client-key.pem

In other words, the options are similar to those used for the server. Note that the Certificate Authority certificate has to be the same. “

Ref: http://dev.mysql.com/doc/refman/5.0/en/secure-using-ssl.html

root@sage:~# /etc/init.d/mysql restart

Test connectivity to Master from Slave

root@sage:~# mysql –ssl-ca=/etc/mysql/ssl/ca-cert.pem -u root -p -h 192.168.1.100

root@sage:~# mysql -u root -p

mysql> SLAVE STOP;

mysql> mysql> CHANGE MASTER TO MASTER_HOST=’192.168.1.100′, MASTER_PORT=3306, MASTER_USER=’replssl’, MASTER_PASSWORD=’1′,MASTER_LOG_FILE=’mysql-bin.000004′, MASTER_LOG_POS=7705, MASTER_SSL=1, MASTER_SSL_CA=’/etc/mysql/ssl/ca-cert.pem’;

mysql> START SLAVE;

mysql> SHOW SLAVE STATUS\G

mysql> SHOW PROCESSLIST\G

Note : If we have given only GRANT … REQUIRE SSL to replication user then MASTER_SSL=1, MASTER_SSL_CA are to be specidfied. ITo require that a client certificate also be specified, create the account using the REQUIRE X509 option.

Quick and Dirty DRBD with HA

admin — Tue, 06 May 2008 13:46:04 +0000

I had to dig a while on this one. I think it was mostly because it took me a while to get what was going on. Basically, I have 4 server pairs that I needed to setup. Three where to be mysql instances, a master instance, a slave instance that was only a partial replication of the master and a global slave instance. The 4th pair was for crunching stats from another series of XML servers.

So I built all of these instances to give hardware based failover options in case of total failure or need for upgrade. I honestly am not counting on any of these instances to be a seamless failover scenario, but I figure it could work in a crisis and worst case, I have a copy of important data that is ready to go live.

Needless to say, through setting up 4 pairs of servers I got handy with the routine that I knew worked. Here was my process:

DRBD Setup

1.) Kickstart both boxes with the following KS disk layout:

part / –size 10000
part /data –size 5000 –grow
part /boot –size 100
part swap –size 2000

The data slice is what I will share between the DRBD pair.

*NOTE – The next steps – Step 2 through 9 need to be completed on both servers in the HA, DRBD setup.

2.) Download the latest DRBD source.

3.) I wanted to make the install much harder than it actually is, so I spent time on many sites staring at kernel patches and such.. Oh well. When all was said and done, I really just needed to make sure the kernel source was in my kickstart, then run the following from within the source directory:

make clean all
make install
make install-tools

4.) Brew up a config file for DRBD ( there should be a default in /etc/drbd.conf ) :

# The important logicstics as best I can tell.
# Protocol – there are 3 options I chose C as it appeared to be the best to use with mysql
# syncer – this was a head scratcher. When I first tried to sync my disks I had left it at default
# and our 56G chunks were taking about 6 hours to complete. I’m not only on GB interfaces here
# but the servers in question are blades and they share a chassis backplane, so I cranked it up
# 500M is close to the threshold, don’t remember what it was but I hit it when I was being all
# crazy with this setting to start.
# Server Names – in this example, server1 and server2, make sure they are in /etc/host
# Resource – this is going to be used through out the example as a way for the DRBD system to identify t
# the config params that create this disk replication scenario

global {
usage-count yes;
}
common {
protocol C;
syncer { rate 500M; }
}
resource r0 {
on server1 {
device /dev/drbd1;
disk /dev/sda3;
address 192.168.195.124:7789;
meta-disk internal;
}
on server2 {
device /dev/drbd1;
disk /dev/sda3;
address 192.168.195.125:7789;
meta-disk internal;
}
}

5.) Next, I umounted the disk in question, since it was built with kickstart, it came up as a mounted ext3 slice at /data

6.) Then I zeroed out /dev/sda3 ( what was our /data mount ) with the following command:

dd if=/dev/zero bs=1M count=1 of=/dev/sda3 ; sync

7.) Next, we need to create the device that we refer to in the config file /dev/drbd1 . I found a bash line in a search that creates several so I just did it up in case I ever wanted to expand this config:

for i in `seq 0 1` ; do mknod -m 0660 /dev/drbd$i b 147 $i; done

Just change the seq numbers if you want more or less.

8.) Next we use the drbdadm command to create our resource ( r0 ):

drbdadm create-md r0

9.) Start the drbd service

/etc/init.d/drbd start

10.) Pick the server that you want to be the primary in the pair and run the following command:

drbdadm — –overwrite-data-of-peer primary r0

This will create the sync between the 2 disks you can watch the progress of the sync with the following command:

watch cat /proc/drbd

11.) Once the disks are in sync, create a file system on both sides. This where it can get kind of tricky if you’re not sure what it’s doing. We just told server1 to be the primary server in the pair so when you look at /proc/drbd you’ll see the following that verifies this is the case:

1: cs:Connected st:Primary/Secondary ds:UpToDate/UpToDate C r—

I added the following to /etc/fstab so I could mount and make a file system:

/dev/drbd1 /data ext3 defaults 1 2

Then:

mount /data

Then:

/sbin/mkfs -t ext3 /dev/drbd1

12.) Now we are going to create the filesystem on the secondary server. ( I’m not sure this is entirely necessary, but it’s a good exercise in moving the slice around ).

* Unmount the /data slice on the primary server:

umount /data

* Make the primary box ( server1 in our case ) the secondary in the drbd pair:

drbdadm secondary r0

* Make the secondary box ( server2 in our case ) the primary in the drbd pair:

drbdadm primary r0

* Add the /etc/fstab line

/dev/drbd1 /data ext3 defaults 1 2

* Mount the slice:

mount /data

* Create the filesystem:

/sbin/mkfs -t ext3 /dev/drbd1

Once this is complete, use the same series of commands to make server1 the primary, remove all of the /etc/fstab entries for /data ( Linux HA is going to take care of this for us ) and move on to the HA config below.

Linux HA Setup

The goal of this setup is as follows:

a.) Choose a VIP for your HA pair

b.) Build a config file that starts and stops services on startup and failover

HA is pretty simple when all is said and done ( at least in this config ). It comes rolled up with scripts that take care of the following:

- Manage VIP
- Manage DRBD resource
- Mounting file systems
- Starting services

Install

To install in my setup I just ran:

yum -y install heartbeat

Config

Once heartbeat is installed there are 3 files to worry about:

- ha.cf
- haresources
- authkeys

All of these files will live in /etc/ha.d

All of these files must be identical on both servers in the pair.

1.) ha.cf

keepalive 100ms
deadtime 2
warntime 10
initdead 120
bcast eth1 eth0
autojoin none
auto_failback on
node server1 server2

You can play with the values of each of these and there is further information here.

Again in this config ( and this should already be done ) .. make sure you have server1 and server2 in /etc/hosts

2.) haresources

This is basically the command file, it tells HA what to do and in what order. Here is my config file:

Server1 IPaddr::192.168.195.212/24 drbddisk::r0 \
Filesystem::/dev/drbd1::/data::ext3::defaults mysql.server

Breakdown:

Server1 – This will be the same on both primary and secondary, it tells HA who should be in charge.
IPaddr – This identifies the VIP that will be used between the two systems. If you have multiple NICs, it is smart enough to figure out which one it should be a sub of.

* SIDE NOTE – In the haresources file, the syntax basically indicates that you are asking HA to run a script. HA looks for scripts in /etc/ha.d/resource.d and /etc/init.d by default, take the following example:

IPaddr::192.168.195.212/24

This means you are asking HA to run the following:

/etc/ha.d/resource.d/IPaddr 192.168.195.212/24 start

The result will be an added ethernet interface with the VIP for the pair.

* END SIDE NOTE

drbddisk – This tells HA to use the drbddisk command in /etc/ha.d/resource.d to initialize the drbd resource ( r0 in our case ) and make it primary.

Filesystem – This tells HA to mount the drbd resource at /data

mysql.server – This tells HA to use the script in /etc/init.d to fire up mysql

* SIDE NOTE

You can imagine how important the sequence is here. If I were to try to mount /data before I initialize drbd, it’s going to fail. Unfortunately, HA is not real clear on what is going wrong when it’s failing, it just fails. There are log file lines you can use in ha.cf, but everything goes to /var/messages by default so don’t waste your time.

Also, if you are setting up multiple HA pairs that should not know about other pairs, you need to change the following parameters in your ha.cf:

- Remove this: bcast eth1 eth0
- Add this: mcast eth0 224.1.2.6 694 1 0

For each pair you will want to modify the broadcast address so that it is unique, so for the second pair you could use this:

- mcast eth0 224.1.2.7 694 1 0

Changing the “6″ to a “7″ makes sure that pairs aren’t chatting with each other and failing. The result of not doing this is a /var/log/messages file that grows out of control ( I’m not kidding either, with 4 pairs on the same net this thing was growing at about 500M an hour ). It also drives the server load up on the pair from all the logd action, to the tune of load 0 when only one pair to load 2 when 4 pairs on the same net.

* END SIDE NOTE

3.) authkeys

This can be pretty generic. I thought initially changing this secret around would avoid the issue listed in the last side note, it doesn’t. The only catch with this file is that it has to have perms of 600 or HA won’t start.

auth 2
2 sha1 MySecret

Verification

As I have mentioned, the haresources file basically contains script to argument pairs. It is good practice to walk through all of these with a “start”, then go back through with a “stop”. This will ensure that what HA will run will actually work when HA attempts to run them.

Once you are satisfied with the results of your testing, just run /etc/init.d/heartbeat start on each box. If all goes well, you should see the new VIP on the primary box, the mounted /data slice, and you should be able to log into mysql

* SIDE NOTE

To install mysql I did the following:

- Downloaded preferred tarball
- Changed this in my.cf - datadir=/data/mysql_data
- Added this to my.cf – bind-address=192.168.195.212 ( This ensures that mysql binds to the VIP on startup)

* END SIDE NOTE

This may be a bit scattered, read it a couple times and it should all be clear. This is basically a compilation of 2 days worth of poking at this process to find something that consistently worked.

Good Luck