Surviving the Apocalypse: A Nordic Sysadmin’s Guide to Disaster Recovery

If you are reading this after a catastrophic server failure, I cannot help you. Close this tab and go find your tape drives. If you are reading this while your infrastructure is humming along green, listen closely: your current backup strategy is likely a hallucination. I have seen seasoned engineers cry because they trusted a RAID controller as a backup solution. I have seen entire companies paralyzed because their restoration time from Amazon S3 Glacier took four days longer than their SLA allowed.

In the Norwegian market, we face a dual challenge. We demand the low latency of local routing via NIX (Norwegian Internet Exchange), but we are also bound by some of the strictest data privacy implementations in the world. When the Datatilsynet comes knocking after a breach or data loss event, "the cloud ate it" is not a legal defense. This is how you build a Disaster Recovery (DR) plan that actually works in 2020, focusing on Recovery Time Objective (RTO) and Recovery Point Objective (RPO).

The Fallacy of "The Snapshot"

Many VPS providers sell you "daily snapshots" as a backup solution. This is a convenience feature, not a DR strategy. If the underlying storage array of your provider fails, your snapshot dies with your live data. Furthermore, trusting a snapshot for database consistency without quiescing the filesystem is Russian Roulette.

At CoolVDS, we utilize KVM (Kernel-based Virtual Machine) technology. Unlike container-based virtualization (OpenVZ/LXC), KVM allows for true block-level isolation. However, even on our NVMe-accelerated infrastructure, we advise clients to treat the hypervisor as a single point of failure. Your data must exist in three places: Production (Hot), On-Site Backup (Warm), and Off-Site Immutable Storage (Cold).

The Tooling: BorgBackup & Encryption

In 2020, `rsync` is still useful, but for serious deduplicated, encrypted backups, borgbackup is the industry standard for Linux systems. It handles encryption client-side, meaning your data is unreadable before it ever leaves your server—a mandatory requirement for GDPR compliance when storing customer data off-site.

Here is a battle-tested implementation for backing up a web server to a remote storage box (or a secondary CoolVDS storage instance in a different availability zone).

1. Initializing the Repository

# Install Borg (Debian/Ubuntu) apt-get install borgbackup  # Initialize the encrypted repo export BORG_PASSPHRASE='CorrectHorseBatteryStaple_No_ReUse!' borg init --encryption=repokey user@backup-server.coolvds.net:/var/backups/repo

2. The Backup Script

Do not rely on manual execution. Place this in /usr/local/bin/run-backup.sh and cron it. Note the usage of compression; on CoolVDS NVMe instances, CPU overhead for LZ4 compression is negligible compared to the network bandwidth saved.

#!/bin/bash  # Configuration LOG="/var/log/borg-backup.log" REPO="user@backup-server.coolvds.net:/var/backups/repo" export BORG_PASSPHRASE='CorrectHorseBatteryStaple_No_ReUse!'  # Log Start echo "Starting backup at $(date)" >> $LOG  # Execute Backup borg create                         
    --verbose                       
    --filter AME                    
    --list                          
    --stats                         
    --show-rc                       
    --compression lz4               
    --exclude-caches                
    --exclude '/var/lib/docker'     
    $REPO::{hostname}-{now}         
    /etc                            
    /home                           
    /var/www                        
    /var/lib/automysqlbackup         >> $LOG 2>&1  # Prune older archives to save space borg prune                          
    --list                          
    --prefix '{hostname}-'          
    --show-rc                       
    --keep-daily    7               
    --keep-weekly   4               
    --keep-monthly  6               
    $REPO                           >> $LOG 2>&1

Pro Tip: Notice I excluded /var/lib/docker. Never backup the container runtime overlay filesystem directly. Instead, backup your docker-compose.yml files and your persistent volumes.

Database Consistency: The Silent Killer

Files are easy. Databases are hard. If you backup /var/lib/mysql while the database is running, you will get a corrupted InnoDB tablespace. You have two options for zero-downtime backups in 2020.

Option A: Percona XtraBackup

For large datasets (50GB+), logical dumps are too slow to restore. Use Percona XtraBackup to create physical backups of the binary files.

Option B: Logical Dumps (The mysqldump way)

For most standard deployments, a properly flagged mysqldump is sufficient. The critical flag here is --single-transaction, which ensures a consistent snapshot of InnoDB tables without locking the database.

mysqldump -u root -p 
  --single-transaction 
  --quick 
  --lock-tables=false 
  --all-databases 
  --triggers 
  --routines 
  --events 
  | gzip > /var/lib/automysqlbackup/full_dump_$(date +%F).sql.gz

Store this dump in the directory targeted by your Borg script.

The Geographic Advantage: Why Norway?

Data sovereignty is the hot topic of 2020. With the EU-US Privacy Shield framework under constant legal scrutiny, relying on US-based hyperscalers (AWS, Google Cloud) for storing sensitive Norwegian citizen data carries inherent risk. If a subpoena is issued under the US CLOUD Act, your data could be accessed regardless of where the server technically resides.

By hosting your primary infrastructure and your DR site on CoolVDS, you ensure that your data remains within Norwegian jurisdiction, protected by Norwegian law. Furthermore, the latency between our Oslo data center and major Norwegian ISPs is typically under 5ms. If you need to restore 500GB of data, doing so over a local peering connection at 1Gbps is drastically faster than pulling it from a Frankfurt or Dublin availability zone.

Recovery Time Objective (RTO) Reality Check

Your boss asks: "If the server melts, when are we back online?"
If your answer is "I don't know," you are failing.
If you rely on budget HDDs (Spinning Rust), your restore speed is capped by IOPS. Restoring a 100GB backup on a standard SATA drive can take hours due to I/O wait times.

This is why CoolVDS utilizes Enterprise NVMe storage. We see sustained sequential write speeds vastly superior to standard SSDs. In a DR scenario, disk I/O is almost always the bottleneck. High-performance storage isn't just about loading your website fast; it's about recovering from disaster before your customers tweet about it.

Final verification

A backup that hasn't been restored is not a backup; it's a hope and a prayer. Once a quarter, spin up a fresh CoolVDS instance and attempt a full restore from your Borg repository. If you encounter errors, fix your documentation. If it works, you sleep soundly.