Disaster Recovery in 2014: Why RAID-10 Won't Save You When the Kernel Panics

Let’s be honest with each other. If you are reading this, you probably have a crontab entry somewhere that dumps your database to a local directory, tars it up, and calls it a "strategy." Maybe you are relying entirely on your hosting provider's RAID-10 array, assuming that redundancy equals safety. I have been in this industry long enough to tell you that redundancy is merely a buffer against hardware failure, not a cure for corruption, human error, or catastrophic facility loss.

Following the chaos of the Heartbleed bug back in April, sysadmins across Europe have been scrambling to patch OpenSSL and regenerate keys. It was a stark reminder of how fragile our digital ecosystem is. But while we patch security holes, we often neglect the physical reality: servers melt, fiber gets cut, and rm -rf / happens. As we sit here in mid-2014, with the Norwegian Data Protection Authority (Datatilsynet) tightening its grip on how we handle personal data, relying on luck is no longer a professional option.

In this guide, I’m going to walk you through a pragmatic, battle-hardened approach to Disaster Recovery (DR) suitable for the Norwegian market. We will look at tools available right now in CentOS 6 and Ubuntu 14.04 LTS, and why the infrastructure you choose—like the KVM-based architecture we use at CoolVDS—is the foundation of survival.

The Myth of "The Cloud" and RAID

Marketing teams love to throw around the word "Cloud" as if it magically abstracts away disk failures. It doesn't. Underneath that virtualization layer, there is spinning rust or, if you are lucky, solid-state drives. RAID-10 is standard for performance and single-disk redundancy, but it replicates stupidity instantly. If you accidentally drop a production table, the RAID controller dutifully mirrors that deletion to the other disks. Congratulations, your data is now redundantly gone.

Real DR requires Cold Storage and Off-site Replication. If your primary data center is in Oslo, your backup shouldn't be in the same rack.

Step 1: MySQL Point-in-Time Recovery

Most devs I talk to rely on mysqldump. While useful, it’s heavy on I/O and creates a lock if you aren't careful with transactions. For a high-traffic e-commerce site running on a VPS, you need binary logs enabled. This allows for Point-in-Time Recovery (PITR).

Open your /etc/my.cnf (or /etc/mysql/my.cnf on Debian/Ubuntu) and ensure you have the following configuration. This is crucial for replication and granular recovery:

[mysqld]
# Enable Binary Logging for Replication & Recovery
log_bin             = /var/log/mysql/mysql-bin.log
expire_logs_days    = 10
max_binlog_size     = 100M

# Unique Server ID is mandatory for replication
server_id           = 1

# For InnoDB performance on SSDs (CoolVDS standard)
innodb_flush_log_at_trx_commit = 1
sync_binlog = 1

With log_bin enabled, you can use the mysqlbinlog utility to replay transactions up to the exact second before a disaster occurred. This is far superior to restoring a 24-hour old dump file and explaining to your client why they lost a day's worth of orders.

Step 2: The "Poor Man's" Geo-Redundancy with Rsync

You don't need expensive enterprise software to move data. rsync is still the king of synchronization. It’s efficient, it resumes broken transfers, and it works over SSH.

Here is a robust script pattern I use for clients. It creates incremental backups using hard links, saving massive amounts of space while giving you multiple history points. We run this from a secondary VPS instance, ideally located in a different datacenter.

#!/bin/bash
# /usr/local/sbin/backup-snapshot.sh

SOURCE_USER="root"
SOURCE_HOST="192.168.1.100" # Your Production IP
SOURCE_DIR="/var/www/html/"
DEST_BASE="/backup/snapshots"
DATE=$(date +%F_%H-%M-%S)

# Ensure last backup exists for linking
LATEST_LINK="$DEST_BASE/latest"

mkdir -p $DEST_BASE/$DATE

# Rsync with hard links against the previous backup
rsync -avz --delete --link-dest=$LATEST_LINK \
    -e "ssh -i /root/.ssh/id_rsa_backup" \
    $SOURCE_USER@$SOURCE_HOST:$SOURCE_DIR \
    $DEST_BASE/$DATE/

# Update pointer to latest
rm -f $LATEST_LINK
ln -s $DEST_BASE/$DATE $LATEST_LINK

# Retention policy: Delete backups older than 7 days
find $DEST_BASE/* -maxdepth 0 -type d -mtime +7 -exec rm -rf {} \;

Set this up in your crontab to run nightly. It leverages the file system to only store the differences between backups, but every folder looks like a full backup. It is simple, elegant, and failsafe.

The Latency Factor: Oslo to Europe

When replicating data, latency is the enemy. In 2014, we are seeing a massive improvement in connectivity, but physics still applies. If you are hosting in Norway, you want your primary users to reach you via NIX (Norwegian Internet Exchange) for minimal hops.

However, for your DR site, you need a balance. Too close, and a power grid failure affects both. Too far, and replication lags. Hosting your primary site on a CoolVDS instance in Oslo provides sub-millisecond latency to local users. For the DR site, picking a location like Amsterdam or Frankfurt offers a safe distance while maintaining decent ping times (typically 20-30ms) for data synchronization.

Why KVM Virtualization Matters for Recovery

Not all VPS technologies are created equal. Many budget providers use OpenVZ, which is essentially a container sharing the host kernel. It’s efficient, but it limits your control. You cannot load your own kernel modules, and more importantly, block-level snapshots are difficult.

At CoolVDS, we exclusively use KVM (Kernel-based Virtual Machine). KVM treats your VPS as a true machine. This allows us to perform block-level snapshots of your virtual disk. In a worst-case scenario where your OS file system is corrupted beyond repair by a bad update (we've all seen a yum update go wrong), we can roll back the entire virtual machine state to an hour ago. You cannot do that easily with container-based hosting.

Pro Tip: Always mount your database partition with the noatime flag in /etc/fstab. It stops the server from writing a timestamp every time a file is read, which significantly reduces I/O overhead on high-traffic MySQL servers.

The Compliance Headache: Datatilsynet and Safe Harbor

We need to talk about the legal elephant in the room. The Snowden revelations last year shook trust in US-based hosting. While the "Safe Harbor" framework currently allows data transfer to the US, the political climate is shifting. The EU is discussing major data protection reforms (the proposed General Data Protection Regulation) that could rewrite the rules.

For Norwegian businesses, the safest bet is compliance with the Personopplysningsloven (Personal Data Act). This means keeping your customer data within national borders or the EEA. Hosting on CoolVDS ensures your physical data resides on servers in Norway, under Norwegian law, keeping you clear of the messy jurisdiction battles happening across the Atlantic.

Final Thoughts

Disaster recovery isn't about buying the most expensive enterprise software; it's about architecture. It's about combining the raw speed of SSD storage, the isolation of KVM, and the reliability of simple tools like rsync.

Don't wait for a disk to fail or a hacker to exploit a vulnerability. Verify your backups today. If you are tired of wondering if your current host's "unlimited" storage is actually reliable, it’s time to move to infrastructure built for professionals.

Spin up a KVM instance on CoolVDS today and experience the stability of a true Norwegian datacenter.