Backup Often… But Not Too Often!

This past weekend was a busy one.  It all started on Saturday night.  I came home to find an e-mail in my inbox alerting me to a problem with my dedicated server.  It seems as though TechyDad.com had somehow filled up the partition by growing to 44GB large!

Now, TechyDad.com (along with TheAngelForever and the other sites I run) don’t usually take up that much room.  Not nearly that much.  I began performing some backups and looking at some old databases that might be taking up the extra space.  At first, I couldn’t find anything.

Then, I located it.  The directory where I backed up my mySQL databases was about 39GB large.  As I looked through the directory, I realized what went wrong.  The previous day, I had noticed that my daily backup routine was failing to e-mail me the file.  Thinking that it might be the size of the backups, I altered which databases were backed up.  Then, realizing that I’d need to wait a day to see the results, I set the backup to occur every 15 minutes.

Then, I made my big mistake: I got distracted.  Other matters came up and I forgot all about testing the backups.  But the backups didn’t forget.  They kept going.  Every 15 minutes they would complete another backup.  Every hour, 4 backups would be done.  In no time at all, megabytes of backups turned into gigabytes until the entire disk was flooded.

Thankfully, once the server was running properly again (thanks to plenty of help from a friend of mine), fixing the root cause was easy.  I set the backups to run once a day and I deleted the extra backup files.  Before long, we were back to normal operations.

There are three lessons to be learned here.  First, everyone messes up, including Techy Dads.  Second, backup often… but not too often.  And third, don’t get distracted when you’re doing important work!