Stop Trusting Hardware: A Sysadmin’s Guide to Automated Disaster Recovery

It’s 3:00 AM on a Tuesday. The phone rings. It’s your lead developer screaming that the database is gone. Not corrupted—gone. The file system checks are failing, and the RAID controller on your legacy dedicated server decided to commit suicide, taking the array with it.

If you don’t have an automated, off-site backup strategy in place right now, you are playing Russian Roulette with your business. I have seen grown men cry in server rooms because they thought RAID-10 was a backup strategy. It isn't. RAID provides redundancy; backups provide recovery.

In the Norwegian hosting market, where we pride ourselves on stability and adherence to the Personopplysningsloven (Personal Data Act), losing customer data isn't just an IT failure—it's a legal nightmare involving Datatilsynet. Today, we are going to fix this. No expensive enterprise software, just proven tools: Bash, rsync, and Cron.

The 3-2-1 Rule: Still the Gold Standard

Before we touch the command line, understand the architecture. The 3-2-1 rule is non-negotiable for any serious sysadmin managing a VPS in Norway or anywhere else in Europe:

• 3 Copies of Data: The production data plus two backups.
• 2 Different Media Types: e.g., Server disk and an external storage mount.
• 1 Off-site Copy: If the datacenter in Oslo burns down, your data needs to be safe in a facility in Bergen or Frankfurt.

Pro Tip: At CoolVDS, our Xen-based virtualization platform sits on high-performance Enterprise SSD arrays in RAID-10. This protects you from a single drive failure, but it does not protect you from rm -rf / or a hacked CMS. You need snapshots and file-level backups.

Step 1: The Database (MySQL/MariaDB)

Backing up files is easy. Backing up a running database without locking tables for hours is the trick. If you are running a high-traffic site on a CoolVDS instance, you can't afford downtime.

For InnoDB tables (which you should be using instead of MyISAM in 2012), use the --single-transaction flag. This ensures data consistency without locking the entire table during the dump.

#!/bin/bash

# Database Credentials
USER="root"
PASSWORD="StrongPassword123"
OUTPUT="/backup/mysql"
DATE=$(date +%Y-%m-%d_%H%M)

# Ensure backup dir exists
mkdir -p $OUTPUT

# Dump all databases
mysqldump -u $USER -p$PASSWORD --all-databases --single-transaction --quick --lock-tables=false > $OUTPUT/full_backup_$DATE.sql

# Compress it immediately to save expensive disk space
gzip $OUTPUT/full_backup_$DATE.sql

This script is simple but effective. The --quick option is vital for large tables as it forces mysqldump to retrieve rows from the server a row at a time rather than retrieving the whole row set and buffering it in memory before writing it out.

Step 2: The Filesystem (Rsync)

For file assets (images, configuration files in /etc, web roots in /var/www), rsync is the king of bandwidth efficiency. Unlike FTP, which transfers the whole file every time, rsync only moves the deltas (changes).

When moving data between two Linux servers (e.g., from your CoolVDS production node to a backup storage VPS), use SSH keys for password-less authentication.

rsync -avz -e "ssh -p 22" /var/www/html/ remote_user@backup.coolvds.com:/home/backup/site_copy/

Understanding the Flags:

Step 3: Automating with Cron

A manual backup is a forgotten backup. We need this to run every night when traffic is lowest (usually around 04:00 Oslo time). Edit your crontab:

crontab -e

Add the following line to execute your master backup script:

0 4 * * * /bin/bash /root/scripts/nightly_backup.sh >> /var/log/backup.log 2>&1

This runs at 4:00 AM daily and logs all output (including errors) to a log file. Never run silent cron jobs; you need to know if the disk was full or the network was down.

Retention Policy: Cleaning Up

Disk space costs money. You don't need backups from 2009. Implement a retention policy to delete local backups older than 7 or 14 days. Add this to your bash script:

# Delete backups older than 7 days
find /backup/mysql -type f -name "*.gz" -mtime +7 -exec rm {} \;

Why Infrastructure Matters

Scripts are great, but the underlying hardware dictates restore speed. In 2012, many providers are still pushing spinning SATA disks for VPS hosting. Restoring a 50GB database dump on a 7200 RPM drive takes forever due to IOPS limitations.

This is why CoolVDS utilizes Pure SSD Storage. When you need to recover, you are limited by your network throughput, not your disk seek time. Whether you are running CentOS 5, 6, or the new Ubuntu 12.04 LTS (Precise Pangolin) dropping tomorrow, I/O performance is the bottleneck you want to eliminate.

Summary Checklist for the Paranoid Sysadmin

1. Verify: Test your backups. Restore them to a development Virtual Machine to ensure the SQL file isn't empty.
2. Security: Encrypt your backup files if they are leaving your trusted network using gpg or openssl.
3. Redundancy: Use a CoolVDS secondary storage instance in a different geographic zone if available.

Don't wait for a hardware failure to test your mettle. Script your recovery plan today. If you need a reliable, high-speed environment to host your critical applications with low latency to NIX, deploy a high-performance SSD VPS on CoolVDS now.