MySQL can sometimes create big problems on a server when you have users abusing it.
This article will teach you how to correctly identify the queries that are creating a problem for your server.

MySQL can log those queries that are taking longer then X seconds but this future is not turned on by default.
Here’s how you turn it on:
Login to your server as root
Open my.cnf with your favorite editor. Example:
vim /etc/my.cnf

Into the [mysqld] section add the fallowing lines
log-slow-queries = /var/log/mysql-slow.log
long_query_time = 3

This is just an example. You can use any file name that you want and you can modify the long_query_time to any value. In this example I will be logging to /var/log/mysql-slow.log any queries that are taking longer then 3 seconds.

Go ahead and save the configuration.
For vim: CTRL+X and YES

Now we have to actually create the log file.
touch /var/log/mysql-slow.log

Now we are changing the owner of the file so that mysql and actually write to it.
chown mysql.root /var/log/mysql-slow.log

Now we restart mysql
service mysql restart

It should restart successfully. If it doesn’t check that you didn’t brake my.cnf by examining the error file in your data directory.

Wait a few minutes and then examine the slow queries log
A few examples on how to do it:

cat /var/log/mysql-slow.log
tail /var/log/mysql-slow.log
tail -50 /var/log/mysql-slow.log

After you have identified the offending query go ahead and optimize or remove it.
Again test the results by looking at your server load and the mysql slow queries log.

After you fixed all the problems go ahead and comment the slow queries logging as it will slow your server a bit if you leave it on. my.cnf should now look similar to this:

#log-slow-queries = /var/log/mysql-slow.log
#long_query_time = 3

And don’t forget to restart MySQL after this.

service mysql restart

Hope this helps !

Install MySQL Performance Tuning Primer Script

Tuning the performance of MySQL can be a really hard job to do.
There are many things to consider and no two servers are identical so there is no universal solution.
Tuning Primer is a script that will help you tune your mysql installation by providing very healthy recommendations based on past mysql records.
For the script to be efficient you must run the mysql server for at least 48 hours.
Installation is extremely simple:

Download the script
wget http://day32.com/MySQL/tuning-primer.sh

Change the permissions for the file
chmod 755 tuning-primer.sh

Run it
./tuning-primer.sh

Apply the sugesttions

Enjoy!

Recently, as I was migrating a set of DBs from an aquired company onto new hardware in a new location, I ran across a DB that seemed to consistently become corrupt.  It was usually the larger tables and typically on the slave system.  At first, it was bad enough that I blamed the hardware.  So the box was replaced and things went well for a while.  Recently the head developer for this system came to me with corrupted tables.  He was seeing the output in his code and was feeding me the tables so that I could repair them.  We did them one by one and it became clear that I really needed to check the whole DB.

It would have been relatively simple to throw together a script to do this, but I was apparenlty feeling lazy and did a google search instead.  Low an behold, I ran across a utility that I have read about countless times in the past, and I’m sure at some point I had actually run.   mysqlcheck.  I believe this nice little script has been packaged with mysql since the late 3.x strains.

At any rate, I think I’m documenting it here so that I won’t forget 6 months from now when I need the same functionality again.

So, on the system in question, I had about 20 instances and only one of them needed to be checked.  This is a big deal as the check script, or the repair command, or the optimize command will lock up your tables when it does it’s work and there were 19 web/data instances running on this box that I didn’t want to bother.  The check script is smart enough to handle this situation with the –databases flag.  Simply add the names of the instances that you want the check after that flag and you’re off.

Obviously, you want to do your best to stop traffic to the instances you intend to repair.  Can you get away with not doing this?  Well sure, remember however that the check, repair and optimize functions will lock up the tables in question when they do their work.  What does this mean?  Well, it means that none of your inbound querys will be run, they will queue up behind the repair functions until those functions finish.  In a really bad scenario, it’s possible that each of your querys will be bound to an apache instance and your web server will start eating up memory and potentially crash the whole box.  Thus, it’s in your best interest to stop as much traffic as possible before you start.

In my case, the usage of mysqlcheck looked like this:

mysqlcheck -uroot -pxxxxx –auto-repair –optimize –databases broken_db

The output is fantastic, it tells you where it’s at and what is broken or ok.  If you don’t want to see the output, if you intend to run from cron or whatever, there is a –silent option as well.

At any rate, leave it to mysql to simplify your admin a bit.  Run the mysqlcheck without any options and you’ll see all the options.  Takes 5 minutes to read through them.

That’s it, happy mysql’ing.

So as you might guess it was merely coincidence when we discovered it’s functionality.  Here’s the scenario:

1 DRBD/HA Master MySQL Server
1 DRBD/HA Relay Slave
20 Standalone Slaves

We found with one of our applications that it made alot of sense to keep some tables local and pull some from the master.  The tables that were kept locally on the 20 web serving boxes are rolled up every 10 minutes and their data is sent on for processing.  The tables that are replicated are slave to the Relay Slave, our middle man.

So where does black hole fit in?

We actually figured it out on the initial import of data.  The beginning database size was around 2GB.  We did the full import to the master and at the time, the relay slave was configured with InnoDB tables.  In addition, we were doing the dump as data infile.   Using data infile in a replication scenario works as follows.

1. The entire file is imported to the master.
2. Once the file is completed on the master, the relay slave then begins to import the data.
3. Once the relay slave has completed the file, then the slaves begin to load the data.

In our case these files were quite large.  What we discovered was, for each file we were going to have a hella long wait before it every got to the slaves (which is where we actually needed the data).  The solution?  Don’t write the data to the middle man.  Since binloging is enabled on the relay slave, all of the associated SQL commands are being passed from the Master to the Relay Slave.  Since we never planned on using the Relay Slave in any capacity other than passing on data, there’s no need to write the commands to disk.  Hence, Black Hole.  All of the SQL replication commands are executed on the Relay Slave, but the are basically sent to /dev/null so you save the write time.

The utility of this setup really stood out when using data infile because of the size of the files, it was really easy to see the lag in the middle.  It will probably not be as important with standard inserts and updates to the master as those happen rather quickly.  However, in the event of a problem, should we have to restore a table or whatever, we know moving forward we are going to save that write time in the middle and possible spare some downtime.