Basically, I wanted sendmail to take all inbound email and immediately relay it to another host. In my situation it makes sense because my servers have no direct access to the outside world (security thing). But in reality with the complexities involved in sending email these days, why bother setting up every host or every NAT to be capable of dependably delivering email.

So the solution is, setup one host that you know is a good mail host, and one host that you can monitor for throughput, then point all your other hosts at it for delivery. Piece of cake right? Odd that it took me 30 minutes and then a final trip to the sendmail documentation to find it.

In the end we’re talking about 1 line in your sendmail.mc (I’ve consistently found that most of life’s problems can be solved with 5 to 50 characters in the right text file).

At any rate, the directive you are looking for is:

define(`SMART_HOST', `smart.host.com')dnl

Just replace “smart.host.com” with the smtp server you intend to use as the relay.

For those of you who don’t use the sendmail.mc, you need to modify the DS variable in your sendmail.cf (but really, find something better to do with your time than edit that file).

For all you sendmail.mc users, once that line is in place, run a “make” in your /etc/mail directory and then restart sendmail.

Actually, I was using this mess of a line to test:

touch /tmp/mail.txt; /etc/init.d/sendmail restart ; mail -s "Test2" me@myemail.com < /tmp/mail.txt ; tail -f /var/log/maillog

You’ll come up in a tail of maillog and you should watch for the relay host accepting your message. If it doesn’t get delivered, it’s time to look at the relay server.

That is all, have fun!

So as you might guess it was merely coincidence when we discovered it’s functionality.  Here’s the scenario:

1 DRBD/HA Master MySQL Server
1 DRBD/HA Relay Slave
20 Standalone Slaves

We found with one of our applications that it made alot of sense to keep some tables local and pull some from the master.  The tables that were kept locally on the 20 web serving boxes are rolled up every 10 minutes and their data is sent on for processing.  The tables that are replicated are slave to the Relay Slave, our middle man.

So where does black hole fit in?

We actually figured it out on the initial import of data.  The beginning database size was around 2GB.  We did the full import to the master and at the time, the relay slave was configured with InnoDB tables.  In addition, we were doing the dump as data infile.   Using data infile in a replication scenario works as follows.

1. The entire file is imported to the master.
2. Once the file is completed on the master, the relay slave then begins to import the data.
3. Once the relay slave has completed the file, then the slaves begin to load the data.

In our case these files were quite large.  What we discovered was, for each file we were going to have a hella long wait before it every got to the slaves (which is where we actually needed the data).  The solution?  Don’t write the data to the middle man.  Since binloging is enabled on the relay slave, all of the associated SQL commands are being passed from the Master to the Relay Slave.  Since we never planned on using the Relay Slave in any capacity other than passing on data, there’s no need to write the commands to disk.  Hence, Black Hole.  All of the SQL replication commands are executed on the Relay Slave, but the are basically sent to /dev/null so you save the write time.

The utility of this setup really stood out when using data infile because of the size of the files, it was really easy to see the lag in the middle.  It will probably not be as important with standard inserts and updates to the master as those happen rather quickly.  However, in the event of a problem, should we have to restore a table or whatever, we know moving forward we are going to save that write time in the middle and possible spare some downtime.