Surviving the Spike: High-Fidelity Infrastructure Observability

I still wake up in a cold sweat thinking about Black Friday 2023. Our primary load balancer didn't crash because of CPU exhaustion. It didn't crash because of RAM. It stalled because a noisy neighbor on a cheap public cloud oversold their storage backend, causing our I/O Wait to spike to 85%. The dashboard showed green. The customers saw 502s.

Monitoring isn't just about pretty Grafana dashboards. It is about survival. If you are running infrastructure in 2025 without observing saturation levels and stall times, you are flying blind.

This guide cuts through the vendor fluff. We are going to look at how to monitor infrastructure at scale, specifically within the context of the Norwegian market where latency to NIX (Norwegian Internet Exchange) and GDPR compliance are non-negotiable constraints.

The "Steal Time" Ghost

The biggest lie in virtualized hosting is the CPU core count. If you are on a shared platform, your "vCPU" is a time-slice of a physical thread. When another tenant demands resources, the hypervisor pauses your VM. This is visible as %st (Steal Time) in top.

Run this command on your current infrastructure right now:

If the last column (st) is anything above 0 for extended periods, your application is stuttering, and your users are noticing lag, even if your CPU usage says "20%".

At CoolVDS, we enforce strict KVM isolation limits. We don't overprovision CPU cores because we know that consistency beats raw burst speed for production workloads. But you shouldn't take my word for it. You should monitor it.

The 2025 Observability Stack: Prometheus + eBPF

Agents are heavy. In 2025, we shifted toward eBPF-based exporters where possible, but the standard Prometheus node_exporter remains the reliable workhorse for base metrics. However, the default configuration is too noisy and misses the critical metrics.

Here is a production-hardened systemd service file for node_exporter on Ubuntu 24.04 LTS. This configuration enables collectors that are usually disabled but vital for high-load debugging.

1. Optimized Node Exporter Configuration

[Unit]
Description=Prometheus Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter \
    --collector.disable-defaults \
    --collector.cpu \
    --collector.meminfo \
    --collector.loadavg \
    --collector.filesystem \
    --collector.netdev \
    --collector.pressure \
    --collector.diskstats \
    --collector.vmstat \
    --web.listen-address=:9100

[Install]
WantedBy=multi-user.target

Key Flag: --collector.pressure. This enables PSI (Pressure Stall Information), a kernel feature that tells you exactly why processes are stalling (CPU, IO, or Memory). It is far more accurate than Load Average.

Monitoring NVMe I/O Performance

High-speed storage is the backbone of modern applications. If you are paying for NVMe storage, verify you are getting NVMe speeds. Slow I/O kills database performance faster than slow queries.

To verify the throughput on a CoolVDS NVMe instance, we use fio for benchmarking, but for live monitoring, we look at the request queues.

# Check for disk saturation in real-time
iostat -xnz 1

If your avgqu-sz (Average Queue Size) is consistently higher than 1, your disk subsystem cannot keep up with requests. This often happens on budget VPS providers that throttle IOPS. On our infrastructure, we map NVMe namespaces directly to ensure the queue drains instantly.

Here is a Prometheus alert rule (PromQL) to trigger when disk latency degrades performance:

groups:
- name: host-storage
  rules:
  - alert: HighDiskLatency
    expr: rate(node_disk_read_time_seconds_total[1m]) / rate(node_disk_reads_completed_total[1m]) > 0.1
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: "High disk latency on {{ $labels.instance }}"
      description: "Disk read latency is above 100ms (current value: {{ $value }}s)"

The Norwegian Context: Latency and NIX

For services targeting Norwegian users, routing matters. Traffic should ideally hit the NIX (Norwegian Internet Exchange) in Oslo and stay local. If your data trompes through Frankfurt or Stockholm before reaching a user in Bergen, you are adding 20-30ms of unnecessary latency.

You can verify your upstream routing using mtr (My Traceroute). Run this from your server:

(That IP is a common NIX reference point). You want to see packet loss at 0% and the final hop under 2ms if you are hosted in Oslo.

Pro Tip: Data residency is not just about speed; it's about the Datatilsynet (Norwegian Data Protection Authority). Keeping logs and metrics on servers physically located in Norway simplifies your GDPR Article 44 (Data Transfer) compliance significantly. CoolVDS infrastructure is physically located in Oslo, ensuring your data remains within Norwegian jurisdiction.

Automated Alerting with Alertmanager

Observability is useless if you have to stare at a screen. You need intelligent routing for alerts. In 2025, we don't send emails for warnings; we send Slack/Teams webhooks for warnings and PagerDuty/OpsGenie for criticals.

Here is a sample alertmanager.yml configuration that routes severity based on labels:

route:
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 1h
  receiver: 'slack-notifications'
  routes:
  - match:
      severity: critical
    receiver: 'pagerduty-critical'

receivers:
- name: 'slack-notifications'
  slack_configs:
  - api_url: 'https://hooks.slack.com/services/T00000/B00000/XXXXXXXX'
    channel: '#devops-alerts'
    send_resolved: true

- name: 'pagerduty-critical'
  pagerduty_configs:
  - service_key: 'YOUR_PD_KEY'

Conclusion: Trust but Verify

We built CoolVDS because we were tired of "noisy neighbor" syndrome and opaque resource limits. We give you raw KVM performance, backed by local NVMe storage, right here in Norway. But I don't want you to just believe the marketing.

Deploy a Prometheus exporter. Check the %st steal time. Measure the NVMe IOPS. Real pros verify their infrastructure. If your current host is hiding these metrics or throttling your I/O, it's time to move.

Ready to see what 0% Steal Time feels like? Deploy a high-performance instance on CoolVDS today and get full root access in under 55 seconds.