Unmasking Latency: A Early Adopter's Guide to OpenTelemetry on High-Performance VPS

It is 3:00 AM. Your pager is screaming. The checkout service in your Oslo region is timing out, but the CPU load is sitting comfortably at 20%. You grep the Nginx logs, and everything looks fine. You check the database; queries are fast. Yet, customers are hitting 504 errors.

This is the distributed systems murder mystery. In the old days of monoliths, we just looked at a single stack trace. Now, a single request hits a load balancer, an authentication service, a cart service, a payment gateway, and an inventory checker. If one of those hops stutters, the whole request dies.

This is why OpenTelemetry (OTel) is currently the most exciting project in the CNCF landscape, merging the best of OpenTracing and OpenCensus. Although it is still in Beta (as of April 2020), it is stable enough for those of us tired of debugging blind. But be warned: observability comes with a tax. It eats RAM and CPU. If you try to run a full observability stack on oversold, budget hosting, your monitoring tool will become the bottleneck you are trying to find.

The Architecture of Visibility

Observability relies on three pillars: Metrics (what happened?), Logs (why did it happen?), and Traces (where did it happen?). OpenTelemetry provides a unified standard to collect these.

In a typical Norwegian setup, where data residency is critical due to GDPR (and the watchful eye of Datatilsynet), you cannot just pipe your traces to a US-based SaaS without a headache. You need to host the collector and the backend yourself. This keeps data inside the country and reduces latency penalties.

The Setup

We are going to deploy the OpenTelemetry Collector. It sits as a middleware, receiving telemetry from your apps and pushing it to backends like Jaeger (for tracing) and Prometheus (for metrics).

Here is a battle-tested configuration for an otel-collector-config.yaml. We are keeping it simple: receiving data via OTLP and exporting to a local Jaeger instance.

receivers:
  otlp:
    protocols:
      grpc:
      http:

exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"
    namespace: "coolvds_metrics"

  jaeger:
    endpoint: "jaeger-all-in-one:14250"
    insecure: true

processors:
  batch:

service:
  extensions: [pprof, zpages, health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [jaeger]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus]

This configuration assumes you have a backend ready to accept the data. If you are running this on a CoolVDS instance, you can spin up the backends using Docker Compose. Note the memory limits—Java-based components like Elasticsearch (often used with Jaeger) love to devour RAM.

version: "3"
services:
  # Jaeger for Trace Visualization
  jaeger-all-in-one:
    image: jaegertracing/all-in-one:1.17
    environment:
      - COLLECTOR_ZIPKIN_HTTP_PORT=9411
    ports:
      - "16686:16686"
      - "14268:14268"
      - "14250:14250"
    deploy:
      resources:
        limits:
          memory: 1G

  # The OTel Collector
  otel-collector:
    image: otel/opentelemetry-collector:0.2.0
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "1888:1888"   # pprof extension
      - "8888:8888"   # Prometheus metrics exposed by the collector
      - "8889:8889"   # Prometheus exporter metrics
      - "13133:13133" # health_check extension
      - "4317:4317"   # OTLP gRPC receiver
      - "55679:55679" # zpages extension
    depends_on:
      - jaeger-all-in-one

The Hidden Cost: Context Switching & I/O

This is where things get technical. When you instrument an application to emit traces, you are essentially asking it to write a diary entry every time it performs an action. "I started a SQL query," "I finished the SQL query," "I called the Redis cache."

This generates a massive amount of small I/O operations and network packets. On a standard, oversold VPS, this leads to CPU Steal. The hypervisor pauses your VM to let another neighbor run their PHP script. Your application halts for 20ms. In a microservice chain of 10 calls, that 20ms delay compounds. Suddenly, your "performance monitoring" tool is causing the performance issue.

Pro Tip: Check your CPU steal time using sar -u 1 5. If the %steal column is consistently above 0.5%, your host is overselling. Move to a platform with dedicated cores like CoolVDS immediately. You cannot debug latency if your infrastructure is introducing random noise.

Instrumentation: The Application Layer

Let's look at how we actually generate this data. As of early 2020, the Go libraries for OpenTelemetry are in alpha/beta, but usable. Here is how you initialize a tracer that talks to our Collector running on the localhost.

package main

import (
	"context"
	"log"
	"time"

	"go.opentelemetry.io/otel/api/global"
	"go.opentelemetry.io/otel/exporters/otlp"
	"go.opentelemetry.io/otel/sdk/resource"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	"google.golang.org/grpc"
)

func initTracer() func() {
	ctx := context.Background()

    // Connect to the Collector running on the CoolVDS instance
	driver := otlp.NewGrpcDriver(
		otlp.WithInsecure(),
		otlp.WithAddress("localhost:4317"),
	)

	exporter, err := otlp.NewExporter(ctx, driver)
	if err != nil {
		log.Fatalf("failed to create exporter: %v", err)
	}

	tp, err := sdktrace.NewProvider(
		sdktrace.WithConfig(sdktrace.Config{DefaultSampler: sdktrace.AlwaysSample()}),
		sdktrace.WithSyncer(exporter),
		sdktrace.WithResource(resource.New(key.String("service.name", "payment-service"))),
	)
	if err != nil {
		log.Fatalf("failed to create trace provider: %v", err)
	}

	global.SetTraceProvider(tp)

	return func() {
		_ = exporter.Shutdown(ctx)
	}
}

Notice the sdktrace.AlwaysSample(). In a production environment with high traffic, this is dangerous. It means every single request is traced, serialized, and sent over the network. This is where the hardware matters.

Why NVMe is Non-Negotiable

When you dump these traces into a backend like Elasticsearch or Cassandra (backing Jaeger), you are performing heavy write operations. Mechanical HDDs cannot keep up with the IOPS required for real-time tracing of a high-traffic application.

At CoolVDS, we standardized on NVMe storage not just for speed, but for queue depth. NVMe drives can handle thousands of parallel requests. When your observability stack is flushing buffers to disk, you don't want your main database to lock up because the I/O channel is choked.

Latency and The Norwegian Context

If your users are in Oslo or Bergen, hosting your monitoring stack in Frankfurt or Amsterdam adds 20-30ms of round-trip time (RTT). While that sounds small, for a "sidecar" collector pushing UDP packets, distance equals packet loss.

By deploying your OTel collector on a VPS Norway instance provided by CoolVDS, you benefit from direct peering at NIX (Norwegian Internet Exchange). Your traces reach the collector instantly. This accuracy is vital when you are trying to debug a race condition that only happens under load.

Conclusion: Light in the Dark

OpenTelemetry is still evolving. APIs are changing. But the visibility it grants is worth the bleeding edge pain. Just remember that observability is data-heavy. It requires CPU to serialize objects and I/O to store them.

Don't let your infrastructure be the reason your monitoring fails. For a stack that can handle the overhead of full-system tracing without sweating, deploy your OTel setup on CoolVDS.

Ready to see what your code is actually doing? Spin up a high-performance CoolVDS KVM instance in 55 seconds and stop debugging in the dark.