Why "Status: OK" is often a Lie

It’s 3:00 AM. Your phone buzzes. You check Nagios. All checks are green. CPU is at 40%. RAM is fine. Disk space is plentiful. You go back to sleep.

At 8:00 AM, you wake up to a furious email from the CTO. The checkout page has been timing out for six hours. Customers are leaving. Revenue is bleeding. But your dashboard still says everything is "OK."

This is the failure of traditional monitoring. We have spent the last decade building checks for known failure modes (Is the process running? Is the port open?), but modern infrastructure fails in unknown ways. With the release of Prometheus 1.0 just last month and the maturation of the ELK stack, we are finally moving from "Monitoring" (checking if the lights are on) to "Introspection" (checking if the wiring is melting).

In this deep dive, we are going to configure Nginx for machine-readable logging, set up the new Prometheus time-series database, and discuss why your storage backend—specifically NVMe—is likely the reason your log aggregation is failing.

The Shift: Blackbox vs. Whitebox Monitoring

The Google SRE book, which has been making rounds in the community this year, distinguishes between these two concepts clearly:

• Blackbox (Nagios/Zabbix): Pings the server from the outside. "Can I reach port 80?" It detects symptoms but not causes.
• Whitebox (Prometheus/StatsD): metrics emitted from inside the application. "How long is the garbage collection pause?" "What is the depth of the MySQL write queue?"

To survive a high-traffic launch, you need Whitebox visibility.

Step 1: Stop Parsing Text Logs with Regex

If you are still using `grep` or complex regex in Logstash to parse default Nginx logs, you are wasting CPU cycles. Configure Nginx to output JSON directly. This makes ingestion into the ELK (Elasticsearch, Logstash, Kibana) stack trivial.

Edit your /etc/nginx/nginx.conf inside the http block:

http {
    log_format json_combined escape=json
      '{ "time_local": "$time_local", '
      '"remote_addr": "$remote_addr", '
      '"remote_user": "$remote_user", '
      '"request": "$request", '
      '"status": "$status", '
      '"body_bytes_sent": "$body_bytes_sent", '
      '"request_time": "$request_time", '
      '"upstream_response_time": "$upstream_response_time", '
      '"http_referrer": "$http_referrer", '
      '"http_user_agent": "$http_user_agent" }';

    access_log /var/log/nginx/access_json.log json_combined;
}

Pro Tip: The $request_time vs $upstream_response_time is critical. If request time is high but upstream is low, the latency is in Nginx (likely buffering or slow client). If both are high, your PHP-FPM or backend application is the bottleneck.

Step 2: Implementing Prometheus 1.0

Prometheus 1.0 was released in July (2016), and it handles high-cardinality metrics better than Graphite. Unlike push-based systems, Prometheus pulls metrics.

First, grab the node_exporter. This exposes kernel-level metrics. On your CoolVDS instance (running Ubuntu 16.04):

wget https://github.com/prometheus/node_exporter/releases/download/v0.12.0/node_exporter-0.12.0.linux-amd64.tar.gz
tar xvfz node_exporter-0.12.0.linux-amd64.tar.gz
cd node_exporter-0.12.0.linux-amd64
./node_exporter &

Now, run the Prometheus server itself. While you can install from source, Docker is cleaner for tooling:

docker run -d -p 9090:9090 -v /tmp/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus:v1.0.1

Your prometheus.yml should look like this:

global:
  scrape_interval:     15s

scrape_configs:
  - job_name: 'node'
    static_configs:
      - targets: ['localhost:9100']

The Hardware Bottleneck: Why I/O Kills Observability

Here is the trade-off nobody talks about: Observability is expensive.

When you turn on debug logging, detailed metrics, and push thousands of events per second to Elasticsearch, you create a massive I/O load. I recently consulted for a client in Oslo whose monitoring stack crashed their production DB. Why? Because they put the ELK stack on the same spinning HDD RAID array as their MySQL database. The logs saturated the IOPS, causing the database to lock up.

The CoolVDS Reality Check:
If you are running Elasticsearch or Prometheus on standard VPS hosting with spinning disks or network-throttled storage (SAN), you will experience gaps in your data. High-ingest logging requires NVMe.

We benchmarked this on our CoolVDS NVMe instances versus standard SSD VPS providers. When ingesting 5,000 log lines/second:

Advanced MySQL Introspection

Don't just check if MySQL is running. Check what it is doing. If you have root access (which you do on CoolVDS), use the Percona Toolkit or run raw queries to find lock contentions.

-- Find the top transactions holding locks
SELECT 
  r.trx_id waiting_trx_id,
  r.trx_mysql_thread_id waiting_thread,
  r.trx_query waiting_query,
  b.trx_id blocking_trx_id,
  b.trx_mysql_thread_id blocking_thread,
  b.trx_query blocking_query
FROM information_schema.innodb_lock_waits w
INNER JOIN information_schema.innodb_trx b 
  ON b.trx_id = w.blocking_trx_id
INNER JOIN information_schema.innodb_trx r 
  ON r.trx_id = w.requesting_trx_id;

If this query returns rows often, your application has a race condition, and no amount of CPU scaling will fix it.

The Norwegian Context: Data Sovereignty

We are seeing stricter enforcement from Datatilsynet regarding where user logs are stored. IP addresses in access logs are considered PII (Personally Identifiable Information). If you are shipping your logs to a US-based SaaS monitoring tool, you are navigating a legal minefield.

Hosting your own Prometheus and ELK stack on a CoolVDS instance in Oslo ensures that your introspection data never leaves Norwegian legal jurisdiction. You get better latency, full control, and compliance without the headache.

Final Thoughts

Green lights on a dashboard are comforting, but they are often deceptive. To truly own your infrastructure in 2016, you need to peel back the layers.

1. Switch Nginx to JSON logging.
2. Deploy Prometheus to capture time-series trends.
3. Ensure your underlying storage can handle the write-heavy load of detailed logging.

Don't let slow I/O be the reason you can't see why your server is crashing. Spin up a CoolVDS NVMe instance today and get the IOPS you need to run a proper introspection stack.