Stop Flying Blind: Building a Sovereign APM Stack on NVMe in 2018

It is late 2018. If you are still relying on a "tail -f" in one terminal and `top` in another to diagnose a production outage, you are already dead in the water. But the alternative—shipping gigabytes of metrics to a US-based SaaS provider—is becoming a logistical and legal nightmare. Between the Datatilsynet ramping up GDPR enforcement and the sheer latency of round-tripping data across the Atlantic, Norwegian DevOps teams are stuck between a rock and a hard place.

I have seen this too many times. A client in Oslo pays premium rates for a hosted APM solution, only to find out their "real-time" alerts have a 3-minute lag because the provider's ingestion queue is choked. When your SQL database locks up during a flash sale, three minutes is an eternity.

The solution isn't to buy more SaaS. It is to own your metrics. Today, we are building a production-grade monitoring stack using Prometheus 2.4 and Grafana 5.3. We will host it on high-IOPS infrastructure because a Time Series Database (TSDB) on a spinning hard drive is useless.

The Hardware Reality: Why IOPS Matter

Prometheus 2.x introduced a new storage engine. It is vastly more efficient than the old 1.x storage, but it is hungry for disk operations. It writes data in 2-hour blocks. If your underlying storage suffers from "noisy neighbor" syndrome—common in cheap OpenVZ containers—your monitoring dashboard will freeze exactly when you need it most: during a high-load event.

Pro Tip: Never put a production TSDB on shared standard storage. The write amplification will kill your performance. This is why we default to KVM virtualization on NVMe at CoolVDS. You need dedicated I/O throughput, not just "burstable" credits.

Step 1: The Foundation

We are using Ubuntu 18.04 LTS (Bionic Beaver). It’s stable, supports the latest Docker CE, and has a kernel new enough to handle heavy container networking without panicking.

First, secure the environment. If you are hosting this in a Norwegian datacenter, you benefit from lower latency to your local users, but you still need to lock down the firewall. We only want port 3000 (Grafana) exposed publicly, and strictly behind a reverse proxy.

# UFW Configuration for a Monitoring Node
ufw default deny incoming
ufw allow ssh
ufw allow 80/tcp
ufw allow 443/tcp
# Do NOT open 9090 (Prometheus) to the world
ufw enable

Step 2: Deploying the Stack with Docker Compose

While you can install binaries via `apt`, Docker allows us to lock in specific versions. Create a workspace:

mkdir -p /opt/monitoring/{prometheus,grafana}
cd /opt/monitoring
touch prometheus/prometheus.yml

Here is the `docker-compose.yml` that gives us persistence and stability. Note the volume mapping; we are mapping the TSDB to the host's NVMe storage for maximum throughput.

version: '3'

services:
  prometheus:
    image: prom/prometheus:v2.4.3
    container_name: prometheus
    volumes:
      - ./prometheus/:/etc/prometheus/
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention=30d'
    ports:
      - 127.0.0.1:9090:9090
    restart: always

  grafana:
    image: grafana/grafana:5.3.2
    container_name: grafana
    depends_on:
      - prometheus
    ports:
      - 3000:3000
    volumes:
      - grafana_data:/var/lib/grafana
    restart: always

volumes:
  prometheus_data: {}
  grafana_data: {}

Step 3: Configuring the Scraper

Prometheus pulls metrics; it doesn't wait for them to be pushed. You need to tell it where your targets are. In `prometheus/prometheus.yml`, we define the scrape interval. A 15-second interval is standard. Going lower (e.g., 5s) significantly increases disk I/O.

global:
  scrape_interval:     15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'coolvds_node'
    static_configs:
      - targets: ['10.8.0.5:9100'] # Internal IP of your app server

War Story: The Silent CPU Steal
Last winter, I debugged a Magento cluster that was sluggish despite showing 40% idle CPU. The culprit? %st (Steal Time). The hosting provider had oversold the physical cores. The VM was waiting for the hypervisor to give it cycles. By adding the node_exporter to our stack, we visualized Steal Time in Grafana. It was spiking to 25% every hour. We migrated that workload to a CoolVDS instance with dedicated CPU allocation, and the "phantom lag" vanished instantly.

Step 4: The Node Exporter (Client Side)

On your application servers (the ones being monitored), you don't need Docker. Just run the binary as a systemd service. It is lighter and more reliable if the Docker daemon crashes.

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target

Data Sovereignty and GDPR

This is the part the "Pragmatic CTOs" care about, but you should too. If you are storing IP addresses or user-identifiable metadata in your logs/metrics, and that data leaves the EEA, you are creating a compliance liability. By hosting your APM stack on a VPS in Norway, you simplify your GDPR record-keeping. The data never leaves the jurisdiction. Datatilsynet is happy, and your legal team sleeps better.

Performance Comparison: SATA vs NVMe for TSDB

We ran a benchmark ingestion test of 50,000 metrics/second on Prometheus 2.4.

Conclusion

Monitoring is not a passive activity. It is the heartbeat of your infrastructure. In 2018, you have the tools to build a system that rivals New Relic or Datadog for a fraction of the price, with total control over your data retention and privacy.

But software is only as good as the hardware it runs on. A Time Series Database requires low-latency I/O to function correctly during spikes. Don't handicap your visibility by running it on legacy hardware.

Ready to own your data? Deploy a high-performance NVMe instance on CoolVDS today and start monitoring with zero latency.