Disaster Recovery in the Fjord: Why RAID Is Not A Backup Strategy

It was 3:00 AM on a Tuesday when the phone rang. A client's e-commerce platform, hosting a massive seasonal sale for winter gear, had vanished. The database server was unresponsive. The culprit? A corrupted file system that mirrored perfectly across their RAID 1 array. Because they treated redundancy as a backup, they lost everything.

In the world of Server Administration, confusing availability with recovery is a fatal sin. We see this constantly here in Norway. Businesses assume that because they have a VPS in a secure Oslo datacenter with redundant power and RAID storage, their data is immortal. It isn't.

Let’s get real about Disaster Recovery (DR). We aren't talking about expensive SAN appliances or proprietary enterprise software licenses. We are talking about battle-tested, standard Linux tools that saved my skin more times than I can count. If you are running a critical stack on CentOS 6 or Ubuntu 12.04 LTS, this guide is for you.

The 3-2-1 Rule Still Rules

Before we touch the command line, memorize this architecture. It is the gold standard for a reason:

• 3 copies of your data.
• 2 different media types (e.g., your live SSD RAID and a remote storage server).
• 1 copy offsite (If your server is in Oslo, your backup should be in Bergen, London, or at least a different fire zone).

At CoolVDS, we engineer our infrastructure on KVM (Kernel-based Virtual Machine) rather than OpenVZ. Why? Because KVM gives you a true hardware abstraction. If the host kernel panics, your file system doesn't necessarily get taken down with it. However, even KVM cannot save you from rm -rf /. That is where your DR plan starts.

Step 1: The Database Snapshot

If you are running MySQL 5.5 (which you should be, if not 5.1), you cannot just copy the /var/lib/mysql directory while the server is running. You will end up with corrupted tables. You need a consistent dump.

Here is a robust Bash script wrapper for mysqldump. It locks tables specifically for the dump (using --single-transaction for InnoDB to avoid downtime) and rotates old backups.

#!/bin/bash
# /root/scripts/db_backup.sh

BACKUP_DIR="/var/backups/mysql"
MYSQL_USER="root"
MYSQL_PASS="YourSecurePassword"
DATE=$(date +%Y-%m-%d_%H%M)
RETENTION=7

# Ensure backup directory exists
mkdir -p $BACKUP_DIR

# Perform the dump
echo "Starting backup for all databases..."
mysqldump -u$MYSQL_USER -p$MYSQL_PASS --all-databases --single-transaction --quick --lock-tables=false > $BACKUP_DIR/full_backup_$DATE.sql

# Compress it to save space (gzip is fast, bzip2 is smaller but slower)
gzip $BACKUP_DIR/full_backup_$DATE.sql

# Permissions are critical
chmod 600 $BACKUP_DIR/full_backup_$DATE.sql.gz

# Clean up old backups (older than 7 days)
find $BACKUP_DIR -type f -name "*.sql.gz" -mtime +$RETENTION -exec rm {} \;

echo "Backup completed: full_backup_$DATE.sql.gz"

Pro Tip: Never store the backup script inside your web root. I once audited a server where the `backup.sh` was accessible via `http://domain.com/backup.sh`, exposing the root MySQL password to the world. Keep it in `/root/scripts/`.

Step 2: Offsite Transport via Rsync

Local backups protect against accidental deletion. They do not protect against hardware meltdown or datacenter fires. You need to move that data. In 2013, rsync is still the king of bandwidth efficiency.

You don't need FTP (which is insecure) or heavy VPNs. You need SSH keys and a cron job. Suppose you have a secondary CoolVDS storage instance.

First, generate a key pair on your source server:

ssh-keygen -t rsa -b 4096
# Hit enter to save to default location, do NOT set a passphrase for automation
ssh-copy-id -i ~/.ssh/id_rsa.pub user@backup-server.coolvds.net

Now, automate the transfer. This script pushes your MySQL dumps and your web content (e.g., /var/www/html) to the remote server.

#!/bin/bash
# /root/scripts/remote_sync.sh

SOURCE_DIRS="/var/backups/mysql /var/www/html"
REMOTE_HOST="user@backup-server.coolvds.net"
REMOTE_DIR="/home/user/backups/server1/"

# -a: archive mode (preserves permissions, times, etc.)
# -v: verbose
# -z: compress during transfer
# --delete: remove files on remote that are gone on source (Careful with this!)

rsync -avz -e "ssh -p 22" $SOURCE_DIRS $REMOTE_HOST:$REMOTE_DIR

Step 3: High Availability with Heartbeat

Backups cover data loss. High Availability (HA) covers downtime. If you cannot afford the 4 hours it takes to restore from a backup, you need a failover cluster.

On Linux, the classic tool for this is Heartbeat (part of the Linux-HA project). It monitors the health of your servers and moves a "Floating IP" address from the primary to the secondary server if the primary dies.

Install it on both nodes (CentOS example):

yum install heartbeat

Configure /etc/ha.d/ha.cf on the primary node:

logfacility     local0
keepalive       2
deadtime        30
warntime        10
initdead        120
udpport         694

# The nodes in the cluster
node            primary.coolvds.local
node            secondary.coolvds.local

# Interface for heartbeat signals
bcast           eth1

Then configure /etc/ha.d/haresources to define the floating IP:

primary.coolvds.local 192.168.1.100 httpd mysqld

This tells Heartbeat: "If the primary node is alive, it holds IP 192.168.1.100 and runs httpd and mysqld. If it dies, the secondary node takes over the IP and starts those services."

The Datatilsynet Factor

Operating in Norway means respecting the Personopplysningsloven (Personal Data Act). The Data Inspectorate (Datatilsynet) is strict about where customer data lives. When you dump your database and ship it offsite, you are processing personal data.

This is where hosting choice becomes a compliance issue. If your backup target is a cheap bucket in the US, you are navigating the complex Safe Harbor framework. It is infinitely simpler to keep your primary and disaster recovery sites within Norwegian borders or at least within the EEA.

Why CoolVDS Works for DR Scenarios

We built CoolVDS to solve the specific pains of systems administrators who are tired of noisy neighbors and opaque resource limits.

1. Storage Speed: Restoring a 50GB database dump on a standard SATA drive takes forever. CoolVDS uses pure SSD arrays. The IOPS difference means your mysql < dump.sql command finishes in minutes, not hours.
2. Private Networking: We offer unmetered private backend networks. You can run your Heartbeat pulse or DRBD replication over a private interface without eating into your public bandwidth bandwidth or exposing traffic to the internet.
3. KVM Isolation: Unlike OpenVZ, our KVM instances allow you to load your own kernel modules, which is essential if you are doing advanced block-level replication or custom firewalling.

Disaster recovery isn't about pessimism; it's about professionalism. Hardware fails. Admins make typos. Power grids fluctuate (even here). The only variable you control is how fast you recover.

Don't wait for the kernel panic. Spin up a secondary KVM instance on CoolVDS today and test your rsync scripts before you actually need them.