Surviving the Apocalypse: Disaster Recovery Strategies for Norwegian Infrastructure
Let’s be honest. If you are reading this, you’ve probably felt the cold sweat of a silent pager. The eerie calm before the realization hits: ping: timeout. Servers die. Hard drives degrade. Even the most robust data centers in Oslo can suffer power fluctuations. But in April 2016, we aren't just fighting entropy; we are fighting a shifting legal landscape.
With the Safe Harbor agreement invalidated last October, relying on US-based cloud giants for your primary disaster recovery (DR) strategy is no longer just a technical risk; it's a legal minefield. If you are handling Norwegian customer data, Datatilsynet is watching. You need a local fortress.
The "3-2-1" Rule is Dead (If You Don't Test)
Most VPS providers will sell you "managed backups" and call it a DR plan. It isn't. A backup is a copy; Disaster Recovery is a workflow. I recently audited a Magento setup for a retailer in Bergen. They had backups. Daily ones. But when their primary SQL node corrupted its tablespace during a traffic spike, they realized their Recovery Time Objective (RTO) was 14 hours. In e-commerce, that’s bankruptcy.
Your goal is an RTO of under 15 minutes. To achieve this, we don't just copy files; we replicate state.
Step 1: Real-Time Data Replication
For a typical LAMP stack running on CoolVDS NVMe instances, `rsync` via cron is too slow. You lose data between the cron ticks. We need continuous synchronization. For the filesystem, I prefer lsyncd (Live Syncing Daemon), which watches the kernel inotify events.
Configuring Lsyncd for Failover
Install lsyncd on your primary server. It will push changes to your CoolVDS DR instance in real-time.
settings {
logfile = "/var/log/lsyncd/lsyncd.log",
statusFile = "/var/log/lsyncd/lsyncd.status"
}
sync {
default.rsync,
source = "/var/www/html/",
target = "dr-user@10.0.0.5:/var/www/html/",
rsync = {
compress = true,
archive = true,
verbose = true,
rsh = "/usr/bin/ssh -p 22 -o StrictHostKeyChecking=no"
}
}
Pro Tip: Always run your replication traffic over a private interface if possible. CoolVDS offers private networking which reduces latency and keeps your sync traffic off the public interface, saving your bandwidth for actual users.
Step 2: Database Replication (The Hard Part)
Files are easy. Databases are where dreams die. For MySQL 5.7 (which you should be using over 5.5 by now), Global Transaction Identifiers (GTIDs) make failover much less painful than the old log-position method.
Here is the my.cnf configuration for your Master node. We focus on durability here. innodb_flush_log_at_trx_commit=1 is non-negotiable for ACID compliance, even if it costs a few IOPS.
Master Configuration
[mysqld]
server-id = 1
log_bin = /var/log/mysql/mysql-bin.log
binlog_format = ROW
gtid_mode = ON
enforce_gtid_consistency = ON
log_slave_updates = ON
innodb_flush_log_at_trx_commit = 1
sync_binlog = 1
On the Slave (DR) node, the config is similar, but with a unique server-id and read-only mode to prevent accidental writes during standby.
Slave Configuration
[mysqld]
server-id = 2
relay_log = /var/log/mysql/mysql-relay-bin.log
gtid_mode = ON
enforce_gtid_consistency = ON
read_only = 1
Once configured, standard replication initiation in 2016 looks like this:
CHANGE MASTER TO
MASTER_HOST='10.0.0.1',
MASTER_USER='repl_user',
MASTER_PASSWORD='StrongPassword123!',
MASTER_AUTO_POSITION = 1;
START SLAVE;
Step 3: The Switchover Script
When the primary node goes dark, you don't want to be fumbling with SSH keys. You need a script pre-loaded on your DR node to promote it to Master. This script stops the slave process, turns off read-only mode, and updates the application config.
#!/bin/bash
# promote_dr.sh - Promote Slave to Master
echo "Stopping Slave..."
mysql -e "STOP SLAVE; RESET SLAVE ALL;"
echo "Disabling Read-Only Mode..."
mysql -e "SET GLOBAL read_only = OFF;"
echo "Updating App Config..."
sed -i 's/10.0.0.1/127.0.0.1/g' /var/www/html/app/etc/local.xml
echo "DR Node is now ACTIVE MASTER."
Why Infrastructure Choice Matters
You can script replication all day, but if the underlying hypervisor is noisy or unstable, your sync will lag. This is where the "noisy neighbor" effect of container-based virtualization (like OpenVZ) kills you.
We build on KVM at CoolVDS. This provides full kernel isolation. Why does this matter for DR? Because when you are syncing terabytes of data, you need consistent I/O performance. Our storage backend uses NVMe (still a rarity in the Nordic market), which drastically reduces the "catch-up" time for slave databases. If your replication lag exceeds 1 second, you are risking data loss.
Network Latency and NIX
If your primary audience is in Oslo or Stavanger, latency matters. Hosting your DR site in Frankfurt adds 20-30ms. Hosting it in the US adds 100ms+. By keeping your DR node within Norway (connected via NIX), you ensure that if a failover happens, your users won't notice the degradation. Plus, you stay compliant with the strictest interpretations of the Personal Data Act.
The Final Check
Don't wait for a crash. Run this test this Friday:
- Modify your local hosts file to point to the DR IP.
- Run the promotion script.
- Attempt to place an order or write data.
- Verify data integrity.
If you sweat doing this, your DR plan is a hallucination. If you need a sandbox to test this safely, spin up a CoolVDS instance. It takes less than a minute, and with our hourly billing, it costs less than a cup of coffee to prove your architecture works.
Don't let a power outage define your career. Build resilient systems.