Sleep Is For The Backed Up: Automating Disaster Recovery on Linux Systems

It’s 3:00 AM on a Tuesday. Your phone buzzes. It’s not a text from your significant other; it’s Nagios screaming that your primary database node is unreachable. You SSH in, or at least you try to. Connection refused. You finally get console access via VNC, only to see the terrifying output of a corrupted file system.

If your stomach just dropped, you know the feeling. If it didn't, you haven't been a sysadmin long enough.

I've seen seasoned engineers cry because they trusted a RAID 10 array as a "backup strategy." Let's be clear: RAID provides redundancy, not backup. If you rm -rf a directory, the RAID controller dutifully deletes it from all mirrored disks instantly. To survive in this industry, you need automated, versioned, off-site backups. Especially here in Norway, where data integrity isn't just best practice—it's often a mandate under the Personal Data Act (Personopplysningsloven).

The "3-2-1" Rule is Non-Negotiable

Before we touch a single line of Bash, memorize this architecture. It was true ten years ago, and it will be true ten years from now:

• 3 copies of your data.
• 2 different media types (e.g., Disk and Tape, or Production Storage and Backup Storage).
• 1 copy off-site (physically separate from your datacenter).

If your server is hosted in Oslo, your off-site backup shouldn't be in the same rack. Ideally, it should be in a different facility entirely to mitigate risk from fire or power grid failures.

Database Consistency: Don't Just Copy /var/lib/mysql

One of the most common mistakes I see on VPS deployments is using cp or tar on a running database directory. This leads to corrupted, inconsistent tables that are useless when you try to restore them.

For MySQL (standard 5.1 or 5.5 builds), you must use mysqldump. If you are running InnoDB tables—which you should be for data integrity—use the --single-transaction flag. This ensures a consistent snapshot without locking your tables and taking your site offline.

The Script

Here is a battle-tested snippet for a nightly MySQL backup script. It rotates backups so you don't fill your disk, keeping 7 days of history.

#!/bin/bash
# /usr/local/bin/mysql_backup.sh

BACKUP_DIR="/backup/mysql"
MYSQL_USER="root"
MYSQL_PASS="YourSecurePassword"
DATE=$(date +"%Y-%m-%d")

# Ensure backup dir exists
mkdir -p $BACKUP_DIR

# Dump all databases
mysqldump -u $MYSQL_USER -p$MYSQL_PASS --all-databases --single-transaction --quick --lock-tables=false > $BACKUP_DIR/full_dump_$DATE.sql

# Compress it to save space (gzip is fast, bzip2 is smaller but slower)
gzip $BACKUP_DIR/full_dump_$DATE.sql

# Remove backups older than 7 days
find $BACKUP_DIR -name "full_dump_*.sql.gz" -type f -mtime +7 -exec rm {} \;

echo "Backup for $DATE completed."

Make sure to restrict permissions on this script so other users can't read your root password:

chmod 700 /usr/local/bin/mysql_backup.sh

File System Synchronization: rsync is King

For your web files, configuration configs (like /etc/nginx/nginx.conf or /etc/httpd/conf/httpd.conf), and user data, nothing beats rsync. It only transfers the deltas (changes), making it incredibly efficient for transferring data over the WAN.

If you are pushing backups from your CoolVDS instance to a remote storage server, use SSH keys for authentication so you don't need to hardcode passwords.

# Generate key if you haven't already
ssh-keygen -t rsa -b 2048
# Copy to backup server
ssh-copy-id user@backup-server.no

Then, your sync command looks like this:

rsync -avz --delete -e ssh /var/www/ user@backup-server.no:/home/user/backups/www/

Breakdown of flags:

• -a: Archive mode (preserves permissions, owners, groups). Essential for web servers.
• -v: Verbose.
• -z: Compress data during transfer.
• --delete: If a file is deleted on source, delete it on destination (keeps an exact mirror).

Pro Tip: Be careful with --delete. If you accidentally wipe your source directory and the backup script runs, it will wipe your backup too. I recommend using rdiff-backup or creating tarball snapshots before rsyncing for critical data.

Automation via Cron

Manual backups are backups that don't happen. Add your scripts to the root crontab.

# Edit crontab
crontab -e

# Add the following lines:
# Run MySQL dump every night at 02:00
0 2 * * * /usr/local/bin/mysql_backup.sh >> /var/log/mysql_backup.log 2>&1

# Sync files to remote server at 03:00
0 3 * * * rsync -avz -e ssh /var/www/ user@backup.remote.no:/backups/www/ >> /var/log/rsync.log 2>&1

The Hardware Factor: Why I/O Matters

Running backups is I/O intensive. If you are on a cheap VPS provider that oversells their spindle disks, your website's load time will spike while the backup runs. Your iowait will skyrocket, and Apache will start dropping requests.

This is where infrastructure choices matter. In my recent testing on CoolVDS, the performance difference was stark. Because they use high-performance enterprise storage and true KVM virtualization (not OpenVZ containers where you fight neighbors for resources), backup processes complete significantly faster.

Low latency storage means your backup window is shorter, reducing the risk of data inconsistency.

Compliance and Sovereignty in Norway

We are operating under the scrutiny of Datatilsynet. If you handle personal data for Norwegian citizens, you have a legal obligation to secure that data. Off-site backups are a requirement for security, but where that off-site server lives matters.

Sending backups to a server in the US (under the Patriot Act) can create legal headaches regarding data privacy. It is safer, and lower latency, to keep your off-site backups within the EEA, or ideally, within Norway itself. The latency between major Norwegian data centers via NIX (Norwegian Internet Exchange) is often under 5ms, making file transfers lightning fast.

Summary

Disaster recovery isn't a product you buy; it's a process you adhere to. A solid strategy involves:

1. Local retention using fast disks (CoolVDS excels here).
2. Automated scripts for database dumps and file syncing.
3. Off-site replication to a secure, compliant location.
4. Testing your restores. A backup you haven't restored is just a rumor.

Don't wait for the inevitable hardware failure or human error to teach you this lesson. Check your crontab today.

Need a robust environment for your production workload? Deploy a high-performance KVM instance on CoolVDS and stop worrying about I/O bottlenecks during your backup windows.