When you first realize that you have a problem, you’ll have a gut reaction to just delete these files, and to be quite honest, I’m not sure if that really *breaks* anything or not.  MySQL does track these files though.  So it’s possible if you yank a file out from under the server, it might get a little whacky.   The bottom line is though, they are merely a binary representation of the sql that you intend to send to another server (typically), and they have nothing to do with the integrity of the data in the database.

So if you want to do it the *correct* way, mysql has a command set to deal with these logs.  First, to get a look at the files and their sizes, you can do a show binary logs . This will give you the breakdown of all the bin logs the server knows about.

It may be prudent, if you are running in a master/slave environment to take a look at your slave status so that you know you are not behind.  basically, log into your slave and run show slave status\G . Make sure that Slave_IO_Running: Yes and Slave_SQL_Running: Yes exist, and check Seconds_Behind_Master: 0.   These 3 things will tell you that your slave is running and up to date.  While you’re here, make note of the Master_Log_File: .   This should be the most recent log listed in your show binary logs output on the master server.

Now that you know where you are in your binary log world, all you need to do is tell the master to purge the files up to the next to the last file, or less if you’re paranoid.  If you’re most recent log file is mysql-bin.104 then you might want to issue the following command:  PURGE BINARY LOGS TO ‘mysql-bin.103′.  This will leave log 103 and 104 on the server and remove the rest.

That’s about it, Happy Purging!!

Recently, as I was migrating a set of DBs from an aquired company onto new hardware in a new location, I ran across a DB that seemed to consistently become corrupt.  It was usually the larger tables and typically on the slave system.  At first, it was bad enough that I blamed the hardware.  So the box was replaced and things went well for a while.  Recently the head developer for this system came to me with corrupted tables.  He was seeing the output in his code and was feeding me the tables so that I could repair them.  We did them one by one and it became clear that I really needed to check the whole DB.

It would have been relatively simple to throw together a script to do this, but I was apparenlty feeling lazy and did a google search instead.  Low an behold, I ran across a utility that I have read about countless times in the past, and I’m sure at some point I had actually run.   mysqlcheck.  I believe this nice little script has been packaged with mysql since the late 3.x strains.

At any rate, I think I’m documenting it here so that I won’t forget 6 months from now when I need the same functionality again.

So, on the system in question, I had about 20 instances and only one of them needed to be checked.  This is a big deal as the check script, or the repair command, or the optimize command will lock up your tables when it does it’s work and there were 19 web/data instances running on this box that I didn’t want to bother.  The check script is smart enough to handle this situation with the –databases flag.  Simply add the names of the instances that you want the check after that flag and you’re off.

Obviously, you want to do your best to stop traffic to the instances you intend to repair.  Can you get away with not doing this?  Well sure, remember however that the check, repair and optimize functions will lock up the tables in question when they do their work.  What does this mean?  Well, it means that none of your inbound querys will be run, they will queue up behind the repair functions until those functions finish.  In a really bad scenario, it’s possible that each of your querys will be bound to an apache instance and your web server will start eating up memory and potentially crash the whole box.  Thus, it’s in your best interest to stop as much traffic as possible before you start.

In my case, the usage of mysqlcheck looked like this:

mysqlcheck -uroot -pxxxxx –auto-repair –optimize –databases broken_db

The output is fantastic, it tells you where it’s at and what is broken or ok.  If you don’t want to see the output, if you intend to run from cron or whatever, there is a –silent option as well.

At any rate, leave it to mysql to simplify your admin a bit.  Run the mysqlcheck without any options and you’ll see all the options.  Takes 5 minutes to read through them.

That’s it, happy mysql’ing.

Here’s the error:

Nov  8 07:21:24 localhost named: none:0: open: /etc/named.conf: permission denied
Nov  8 07:21:26 localhost setroubleshoot: SELinux is preventing the named-checkconf from using potentially mislabeled files (./named.conf). For complete SELinux messages. run sealert -l 82a8e3a9-7073-472c-b232-bec1d8e30374

To make it go away, we do this:

echo 0 > /selinux/enforce

If you want to bring it back:

echo 1 > /selinux/enforce

To make it permanent:

If you want it gone for good (after reboot), edit /etc/selinux/config and change this:
SELINUX=enforcing
to this:
SELINUX=permissive

So as you might guess it was merely coincidence when we discovered it’s functionality.  Here’s the scenario:

1 DRBD/HA Master MySQL Server
1 DRBD/HA Relay Slave
20 Standalone Slaves

We found with one of our applications that it made alot of sense to keep some tables local and pull some from the master.  The tables that were kept locally on the 20 web serving boxes are rolled up every 10 minutes and their data is sent on for processing.  The tables that are replicated are slave to the Relay Slave, our middle man.

So where does black hole fit in?

We actually figured it out on the initial import of data.  The beginning database size was around 2GB.  We did the full import to the master and at the time, the relay slave was configured with InnoDB tables.  In addition, we were doing the dump as data infile.   Using data infile in a replication scenario works as follows.

1. The entire file is imported to the master.
2. Once the file is completed on the master, the relay slave then begins to import the data.
3. Once the relay slave has completed the file, then the slaves begin to load the data.

In our case these files were quite large.  What we discovered was, for each file we were going to have a hella long wait before it every got to the slaves (which is where we actually needed the data).  The solution?  Don’t write the data to the middle man.  Since binloging is enabled on the relay slave, all of the associated SQL commands are being passed from the Master to the Relay Slave.  Since we never planned on using the Relay Slave in any capacity other than passing on data, there’s no need to write the commands to disk.  Hence, Black Hole.  All of the SQL replication commands are executed on the Relay Slave, but the are basically sent to /dev/null so you save the write time.

The utility of this setup really stood out when using data infile because of the size of the files, it was really easy to see the lag in the middle.  It will probably not be as important with standard inserts and updates to the master as those happen rather quickly.  However, in the event of a problem, should we have to restore a table or whatever, we know moving forward we are going to save that write time in the middle and possible spare some downtime.

yum -y install net-snmp

Then copy over my default snmpd.conf file and fire it up.

In these instances however, snmpd either refused to start or gave the following error in /var/log/messages:

Apr 16 18:14:35 msx-db01 kernel: audit(1208384075.250:6): avc: denied { read } for pid=27712 comm="snmpd" name="snmpd.conf" dev=sda3 ino=7241736 scontext=root:system_r:snmpd_t tcontext=root:object_r:tmp_t tclass=file
Apr 16 18:14:35 msx-db01 kernel: audit(1208384075.265:7): avc: denied { read } for pid=27712 comm="snmpd" name="snmpd.conf" dev=sda3 ino=7241736 scontext=root:system_r:snmpd_t tcontext=root:object_r:tmp_t tclass=file
Apr 16 18:14:35 msx-db01 snmpd[27712]: Warning: no access control information configured. It's unlikely this agent can serve any useful purpose in this state. Run "snmpconf -g basic_setup" to help you configure the snmpd.conf file for this agent.
Apr 16 18:14:35 msx-db01 snmpd[27712]: NET-SNMP version 5.1.2


I tried to use snmpconf as suggested even though I knew my snmpd.conf file was fine. snmpconf was not on the system so I didn’t waste any time trying to find out how to get it.

So the fix is to install the net-snmp-libs package, not sure what it updates but it works.

yum -y install net-snmp-libs

The catch after that is (at least in all 3 of my situations) the following error on the yum install:

Transaction Check Error: file /usr/share/man/man8/ext2online.8.gz from install of e2fsprogs-1.35-12.11.el4_6.1 conflicts with file from package e2fsprogs-1.35-12.4.EL4

So, back to the hack admin package manager of choice *yum* :

yum update e2fsprogs

Once it’s updated, the net-snmp-libs goes in as expected.

So what’s the deal with all this? I dunno, what the hell do utilities for ext2 have to do with my ability to run snmp monitoring? Who gives a shit and if you know, you’re probably on the wrong site. At any rate, my monitoring works and I’ll leave the rest to the server fairies.

Moral of the story: If your shit works, do you really need to understand it?