The 3:00 AM Wake-Up Call
Two weeks ago, a client's "enterprise-grade" server (hosted by a budget provider who shall remain nameless) decided to eat its own RAID controller. The array didn't just degrade; it vanished. The RAID 5 rebuild failed—as they almost always do with large SATA drives due to unrecoverable read errors (URE). If you think RAID is a backup, you are wrong. RAID is for uptime; backups are for when the hardware gods inevitably strike you down.
In the Norwegian hosting market, where we pride ourselves on stability and adherence to strict regulations like Personopplysningsloven, losing customer data isn't just an inconvenience. It's a visit from Datatilsynet (The Data Inspectorate) waiting to happen. If you are running a VPS in Oslo or Stavanger without a grimly pessimistic backup strategy, you are negligent.
I've spent the last decade cleaning up after failed hard drives. Here is how we configure automated, bulletproof backups at the system level using tools available right now in 2010.
The Golden Rule: 3-2-1
Before we touch a single line of Bash, memorize this. It is the only thing standing between you and unemployment:
- 3 copies of your data.
- 2 different media types (e.g., Disk on server, remote tape/disk).
- 1 copy off-site.
For a standard Linux VPS, "media types" usually translates to: the live data, a local snapshot, and a remote mirror. If your data lives solely on the server, you don't own it; chance does.
The Toolset: Keep It Simple, Stupid
Forget complex commercial backup agents that eat up your RAM. In 2010, the most robust tools are already installed on your Linux distribution: rsync, mysqldump, and cron. They are lightweight, scriptable, and they don't crash.
1. Database Consistency is King
You cannot just copy the /var/lib/mysql directory while the server is running. You will get corrupted tables. You need a consistent dump. Here is the snippet I drop into /usr/local/bin/backup.sh on every CentOS 5 or Debian Lenny box I manage:
#!/bin/bash
# Define variables
BACKUP_DIR="/backup/mysql"
DATE=$(date +%Y-%m-%d)
# Ensure dir exists
mkdir -p $BACKUP_DIR
# Dump all databases with compression
# --single-transaction is crucial for InnoDB tables to avoid locking the site
mysqldump -u root -p'YourComplexPass' --all-databases --single-transaction --quick | gzip > $BACKUP_DIR/db_dump_$DATE.sql.gz
# Delete dumps older than 7 days to save space
find $BACKUP_DIR -type f -mtime +7 -name "*.gz" -deleteNote the --single-transaction flag. Without it, your MySQL dump locks tables, causing your website to hang during the backup window. On a high-traffic site, that latency is unacceptable.
2. The Off-Site Push via Rsync
Local backups are useless if the data center catches fire or the host node suffers catastrophic hardware failure. You need to push data out. At CoolVDS, we see savvy admins pushing data from our Oslo nodes to secondary locations constantly. We recommend using SSH keys for password-less transfers.
# Sync the web root and the DB dumps to a remote server
rsync -avz -e "ssh -p 22" /var/www/html /backup/mysql [email protected]:/home/user/backups/Pro Tip: Use the -z flag for compression. Bandwidth in Norway is good, thanks to NIX (Norwegian Internet Exchange), but there is no reason to waste transit capacity transferring uncompressed text files.Automation: Set It and Forget It (But Check Logs)
Manual backups are failed backups. Add this to your root crontab (crontab -e):
0 3 * * * /bin/bash /usr/local/bin/backup.sh >> /var/log/backup.log 2>&1This runs at 3:00 AM local time, when traffic is lowest. The output redirection ensures you have a log file to audit if something breaks.
The Hardware Factor: Why Infrastructure Matters
Software automation is only half the battle. The underlying metal dictates your recovery speed. Many VPS providers oversell their nodes, packing hundreds of containers onto slow 7.2k RPM SATA drives. When you try to run a backup (high I/O operation), the "noisy neighbor" effect kicks in, and your server load spikes to 50+.
This is where architecture choices matter. At CoolVDS, we utilize RAID-10 SAS storage. We don't use RAID 5 (slow writes) and we don't use single drives. RAID-10 gives you the striping speed of RAID 0 with the mirroring redundancy of RAID 1. When you run rsync on a CoolVDS instance, you aren't fighting for disk IOPS. It just works.
Testing Your Restore
A backup is Schrödinger's data until you restore it. Once a month, spin up a secondary VPS on CoolVDS (it takes about 60 seconds to provision). Try to restore your data from your remote backup. If it takes you more than 2 hours to get the site back online, your documentation is bad, or your scripts are broken.
Data loss is a question of when, not if. Don't let a hardware failure become a business failure. Secure your data on high-performance infrastructure that respects the 3-2-1 rule.