When RAID Fails: A Battle-Hardened Guide to Disaster Recovery in Norway

There is a specific kind of silence that haunts a server room. It isn’t the lack of fan noise; it’s the sudden cessation of HDD activity LEDs blinking, followed by the frantic buzzing of a phone. If you have been in this industry long enough, you know the feeling. RAID-10 is not a backup. Snapshots are not a strategy. And relying on hope is professional suicide.

In the wake of the recent Snowden leaks (PRISM), the conversation in Oslo has shifted. It is no longer just about mechanical failure; it is about data sovereignty. If your disaster recovery plan involves shipping encrypted blobs to an S3 bucket in Virginia, you might be violating the Norwegian Personal Data Act (Personopplysningsloven). We need to talk about keeping data alive, keeping it local, and keeping it recoverable within minutes, not days.

The "3-2-1" Rule in the Context of NIX

The standard industry axiom remains true in 2013: Keep 3 copies of your data, on 2 different media types, with 1 offsite. However, "offsite" for a Norwegian business doesn't mean a datacenter in Frankfurt or Amsterdam. Latency matters. Data jurisdiction matters.

When we architect solutions at CoolVDS, we emphasize the proximity to the Norwegian Internet Exchange (NIX) in Oslo. If your primary VPS is melting down, your hot-spare needs to be reachable via low latency peering, not routed through a transatlantic cable.

The Meat: Scripting Your Own Survival

Forget expensive enterprise GUI backup suites that eat up your RAM. If you are running CentOS 6 or Debian 7, you have everything you need built-in: rsync, mysqldump, and cron.

1. The Database Dump (MySQL)

Stopping the database to copy files is for amateurs. Use mysqldump with the --single-transaction flag to ensure data consistency without locking tables (InnoDB only).

#!/bin/bash
# /root/scripts/db_backup.sh

TIMESTAMP=$(date +"%F")
BACKUP_DIR="/backup/mysql"
MYSQL_USER="backup_user"
MYSQL_PASS="ComplexPass123!"

# Create dir if not exists
mkdir -p $BACKUP_DIR

# Dump with consistency
mysqldump -u$MYSQL_USER -p$MYSQL_PASS --all-databases --single-transaction --quick --lock-tables=false > $BACKUP_DIR/full_dump_$TIMESTAMP.sql

# Compress immediately to save SSD space
gzip $BACKUP_DIR/full_dump_$TIMESTAMP.sql

# Remove backups older than 7 days
find $BACKUP_DIR -type f -name "*.gz" -mtime +7 -exec rm {} \;

2. The Offsite Push

Once you have the dump, get it off the server. If the physical host catches fire, your local backup burns with it. We use rsync over SSH.

#!/bin/bash
# /root/scripts/sync_offsite.sh

REMOTE_HOST="backup.coolvds-offsite.no"
REMOTE_USER="storage_user"
LOCAL_DIR="/backup/"

rsync -avz -e "ssh -p 2222" $LOCAL_DIR $REMOTE_USER@$REMOTE_HOST:/home/storage_user/backups/

Pro Tip: Always run your SSH on a non-standard port (like 2222) to reduce log noise from script-kiddie brute force attacks. It doesn't add real security, but it keeps /var/log/secure clean so you can see actual threats.

Real-Time Replication: Beyond Backups

Backups save you from history; replication saves you from downtime. For high-availability setups on CoolVDS, we recommend standard MySQL Master-Slave replication. It is robust, well-documented, and works perfectly on our KVM infrastructure.

Here is the critical configuration for your my.cnf (Master):

[mysqld]
server-id        = 1
log_bin          = /var/log/mysql/mysql-bin.log
expire_logs_days = 10
max_binlog_size  = 100M
binlog_do_db     = production_db
innodb_flush_log_at_trx_commit = 1
sync_binlog      = 1

And for the Slave (the recovery node):

[mysqld]
server-id        = 2
relay_log        = /var/log/mysql/mysql-relay-bin.log
log_bin          = /var/log/mysql/mysql-bin.log
read_only        = 1

Warning: The sync_binlog = 1 setting is the safest for data durability but hits disk I/O hard. This is where hardware selection becomes critical. On spinning rust (HDD), this setting kills performance. On Enterprise SSD storage (which we use exclusively for our high-performance tiers), the latency penalty is negligible.

The Legal Factor: Datatilsynet is Watching

In Norway, compliance is not optional. The Datatilsynet (Data Inspectorate) takes a dim view of personal data leaving the EEA without adequate protection. With Safe Harbor under increasing scrutiny following the NSA revelations this summer, hosting your disaster recovery node within Norwegian borders is the only way to guarantee compliance with the Personal Data Act.

Many "cloud" providers oversell their resources, banking on the fact that you won't use all your CPU cycles. This is dangerous during a recovery scenario when you need 100% of that power immediately to rebuild indexes.

Why KVM Beats OpenVZ for DR

You will see many budget hosts offering OpenVZ containers. Run away. In a disaster recovery scenario, you often need to modify kernel parameters (sysctl) or mount specific file systems (EXT4/XFS) to recover corrupted data.

• OpenVZ: Shared kernel. If the host kernel panics, everyone goes down. You cannot load custom modules.
• KVM (CoolVDS Standard): Full hardware virtualization. You run your own kernel. If you need to upgrade from Linux 2.6 to 3.x for better FS support, you can do it yourself.

The Final Check

A disaster recovery plan that hasn't been tested is just a theoretical document. Scheduled downtime is better than unscheduled panic. This Friday night, run a drill. Kill your primary web service.

1. Can you switch DNS pointers to the backup IP fast enough?
2. Does the Slave database promote to Master cleanly?
3. Are your rsync logs actually showing "success"?

If the answer to any of these is "I think so," you are not ready. Infrastructure is unforgiving. CoolVDS provides the raw, unthrottled I/O and strict KVM isolation you need to build a fortress, but the architecture is up to you. Don't let a failed drive be the end of your business.

Need a sandbox to test your replication topology? Deploy a KVM instance in Oslo with pure SSD storage in under 55 seconds.