The Art of Not Waking Up at 3 AM: Infrastructure Monitoring at Scale

There is a specific kind of silence that every DevOps engineer dreads. It’s not the silence of a calm night; it’s the silence of a Slack channel that should be flooding with alerts but isn’t, because the monitoring server itself has crashed. I’ve been there. It was 2019, and a runaway log file on a legacy system choked our centralized logging node. We flew blind for six hours.

Most VPS providers sell you on CPU cores and RAM. They lie by omission. They rarely mention the metric that actually keeps your monitoring stack alive: Disk I/O.

If you are building infrastructure in 2023, specifically targeting the Norwegian or broader European market, you face two distinct enemies: Latency and Legislation. This guide ignores the buzzwords. We aren’t talking about "observability pipelines." We are talking about how to configure Prometheus so it doesn’t melt your disk, and why hosting this on CoolVDS inside Norway isn't just a performance choice—it's a compliance necessity.

The Hidden Killer: TSDB Write Amplification

Time Series Databases (TSDBs) like Prometheus are write-heavy. Extremely write-heavy. Every time your `node_exporter` scrapes a target, it’s dumping blocks of data to the disk. On a standard HDD or a cheap, oversold VPS with "SSD caching," your write latency spikes as soon as you scale past 50 nodes.

When write latency spikes, Prometheus misses scrapes. When it misses scrapes, your graphs have gaps. You start trusting your data less. And then, you stop checking.

Pro Tip: Never run a production Prometheus instance on shared storage without guaranteed IOPS. If `iowait` exceeds 10% on your monitoring node, your alerting is already compromised.

We use CoolVDS NVMe instances for our monitoring clusters specifically because the I/O throughput is dedicated. KVM virtualization ensures that a neighbor compiling the Linux kernel doesn't steal the cycles your Alertmanager needs to wake you up.

The Compliance Minefield: Schrems II and Datatilsynet

Here is the reality for Norwegian businesses in late 2023. If you are piping server logs—which often contain IP addresses (PII)—to a US-owned SaaS monitoring platform, you are walking a legal tightrope. The Schrems II ruling made this incredibly difficult.

The pragmatic solution? Data Sovereignty.

Keep the monitoring stack on Norwegian soil. By hosting your ELK (Elasticsearch, Logstash, Kibana) or LGTM (Loki, Grafana, Tempo, Mimir) stack on a VPS in Oslo, you bypass the data transfer headache entirely. The data stays under Norwegian jurisdiction.

Implementation: The "No-Nonsense" Stack

Let’s look at a reference architecture for monitoring 100+ nodes. We will use the standard exporter model, but optimized for stability.

1. The Node Exporter Service

Don't run this in Docker if you need raw hardware stats; the abstraction layer can muddy the water for network metrics. Run it as a binary managed by systemd.

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter \
    --collector.systemd \
    --collector.processes \
    --no-collector.wifi \
    --web.listen-address=:9100

[Install]
WantedBy=multi-user.target

2. Optimizing Prometheus Configuration

The default prometheus.yml is fine for a laptop. It is garbage for production. Here is how we tune the scraping to avoid overwhelming the network, especially if you are monitoring across different regions (e.g., scraping a server in Bergen from a monitor in Oslo).

global:
  scrape_interval: 15s 
  evaluation_interval: 15s 
  scrape_timeout: 10s

scrape_configs:
  - job_name: 'coolvds_infrastructure'
    scrape_interval: 10s
    metrics_path: /metrics
    scheme: http
    static_configs:
      - targets: 
        - '10.20.30.40:9100'  # Database Master
        - '10.20.30.41:9100'  # Redis Cache
    
    # Drop heavy metrics to save disk space
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: 'node_systemd_unit_state'
        action: drop

Notice the metric_relabel_configs block? That is crucial. Some exporters produce "high cardinality" data—metrics with thousands of unique label combinations. If you don't drop the noise, your RAM usage will explode. This is why having root access on a KVM VPS is superior to managed solutions; you have total control over what you ingest.

3. The Alert That Matters

Stop alerting on CPU usage. A CPU at 100% is fine if the request latency is low—it just means you are getting your money's worth. Alert on saturation and errors.

groups:
- name: host_level
  rules:
  # Alert if disk fills up in 4 hours based on prediction
  - alert: DiskWillFillIn4Hours
    expr: predict_linear(node_filesystem_free_bytes{fstype!~"tmpfs"}[1h], 4 * 3600) < 0
    for: 5m
    labels:
      severity: page
    annotations:
      summary: "Disk is filling up fast on {{ $labels.instance }}"

Network Latency: The NIX Factor

If your users are in Norway, your monitoring should be too. Round-trip time (RTT) from Oslo to Frankfurt is approx 20-30ms. RTT from Oslo to Oslo (via NIX - Norwegian Internet Exchange) is often sub-2ms.

When you are debugging a microsecond-level delay in a database query, network jitter matters. CoolVDS infrastructure is peered directly at major Nordic exchanges. This means when you run `traceroute`, you aren't hopping through three different countries just to reach a server across the street.

Using eBPF for Deep Inspection

Since we are operating in late 2023, eBPF (Extended Berkeley Packet Filter) has matured significantly. Standard exporters tell you that the server is slow. eBPF tells you why.

On a CoolVDS instance, because you have a full kernel (unlike limited container VPSs), you can install `bcc-tools` and run `biolatency` to see disk I/O latency distributions in real-time:

sudo apt-get install bpfcc-tools linux-headers-$(uname -r)
# Trace block device I/O latency
sudo biolatency-bpfcc -D 10

If you see a multi-modal distribution here, you know you have a contention issue. You can't debug this deep on a restrictive platform.

Final Thoughts: Sleep Better

Building a monitoring stack is about trust. You need to trust that when the pager goes off, it's real. And you need to trust that the data is legal, secure, and accurate.

Don't let slow I/O kill your observability. Don't let GDPR compliance keep you awake. Build your fortress on infrastructure that respects the physics of data.

Ready to own your metrics? Deploy a high-performance NVMe instance on CoolVDS today. With low latency to all major Norwegian ISPs, your data stays close, and your dashboards stay green.