Microservices in Production: Service Discovery, Latency, and the Infrastructure Trap

Everyone wants to be Netflix. But let’s be honest: your e-commerce platform serving the Norwegian market isn't Netflix, and treating it like one without the requisite operational maturity is a suicide mission. In 2016, the migration from monolithic architectures to microservices is the prevailing wind in our industry, but I see too many engineering teams shipwrecked on the rocks of network latency and distributed complexity.

Splitting an application into twenty Docker containers doesn't make it decoupled; often, it just turns function calls into network calls. And network calls can fail. In this deep dive, we are going to look at the architectural patterns necessary to survive this transition—specifically Service Discovery and API Gateways—and why the underlying metal (or virtual metal) matters more than your code quality.

The Latency Tax: Why Infrastructure is Architecture

Before we touch a single config file, understand this: Microservices amplify infrastructure weaknesses. In a monolith, components communicate via memory. It is instantaneous. In a distributed system, they communicate via the network.

If you host your cluster on a budget provider where neighbors are stealing your CPU cycles (high %st in top), or where the hypervisor is oversubscribed, your internal API latency spikes. If Service A calls Service B, which calls Service C, and each hop adds 50ms of jitter due to poor I/O wait times, your user is staring at a white screen.

Pro Tip: Always run iostat -x 1 when diagnosing slow microservices. If your %util is hitting 100% on disk while your traffic is low, your provider is throttling your IOPS. This is why we standardize on KVM and NVMe at CoolVDS; shared limitations are unacceptable for distributed systems.

Pattern 1: The API Gateway (NGINX)

Do not let clients talk to microservices directly. It is a security nightmare and makes refactoring impossible. You need a gatekeeper. In 2016, NGINX remains the undisputed king here, though HAProxy is a valid alternative. The API Gateway handles SSL termination, rate limiting, and routing.

Here is a battle-tested nginx.conf snippet for routing traffic to different backend pools based on URI. This setup assumes you are using private networking between your VPS instances to avoid public bandwidth costs and reduce latency.

http {
    upstream auth_service {
        # weighted round-robin
        server 10.10.0.5:4000 weight=3;
        server 10.10.0.6:4000;
        keepalive 32;
    }

    upstream inventory_service {
        server 10.10.0.7:5000;
        server 10.10.0.8:5000;
    }

    server {
        listen 80;
        server_name api.norway-shop.no;

        location /auth/ {
            proxy_pass http://auth_service/;
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            proxy_set_header X-Real-IP $remote_addr;
        }

        location /inventory/ {
            proxy_pass http://inventory_service/;
            # Timeout aggressive to fail fast
            proxy_read_timeout 3s;
        }
    }
}

Notice the keepalive 32. Without this, NGINX opens a new TCP connection for every request to the backend. In a high-traffic environment, you will exhaust your ephemeral port range and hit TIME_WAIT limits on the OS kernel.

Pattern 2: Dynamic Service Discovery with Consul

Hardcoding IP addresses in NGINX works for three servers. It breaks at thirty. If a node dies and you spin up a replacement, you shouldn't need to manually edit config files and reload NGINX. We need Service Discovery.

We use HashiCorp's Consul. It provides DNS-based discovery and health checking. If a node becomes unhealthy, Consul removes it from the DNS pool immediately.

Installing the Consul Agent

On a CentOS 7 instance (common in enterprise environments), the setup is straightforward. We run the agent in client mode on every worker node.

# Download Consul 0.6.4 (Stable as of mid-2016)
wget https://releases.hashicorp.com/consul/0.6.4/consul_0.6.4_linux_amd64.zip
unzip consul_0.6.4_linux_amd64.zip
sudo mv consul /usr/local/bin/

# Start agent joining the cluster
consul agent -data-dir /tmp/consul -node=worker-1 -bind=10.10.0.5 -join=10.10.0.1

Registering a Service

Instead of manually configuring NGINX upstreams, services register themselves. Create a definition file /etc/consul.d/inventory.json:

{
  "service": {
    "name": "inventory",
    "tags": ["production", "norway-region"],
    "port": 5000,
    "check": {
      "script": "curl localhost:5000/health",
      "interval": "10s"
    }
  }
}

Now, your applications can reach the inventory service simply by pinging inventory.service.consul. No more hardcoded IPs.

The Data Persistence Headache

Microservices are stateless; data is not. Running databases in Docker is still a contentious topic this year. For production, I strongly advise running your data stores (PostgreSQL, MongoDB, Redis) on the host OS or dedicated instances, not inside containers. The I/O overhead of the union filesystem can be brutal for write-heavy workloads.

Furthermore, data sovereignty is becoming critical. With the invalidation of Safe Harbor and the new Privacy Shield framework adopted this July, plus the looming GDPR regulations from the EU, where your data sits physically is a legal matter. Hosting on US-controlled clouds adds a layer of legal complexity that many Norwegian CTOs prefer to avoid.

Using a provider with a datacenter physically located in Oslo or the broader Nordic region ensures you fall under Datatilsynet's jurisdiction and keeps latency to local users minimal. A ping from Oslo to a Frankfurt datacenter is ~25ms. A ping to a local CoolVDS instance is <2ms. In a microservice call chain of 10 requests, that difference is the difference between "snappy" and "broken."

Monitoring: The ELK Stack

When you had one server, you could just tail -f /var/log/syslog. With distributed systems, logs are scattered everywhere. You need centralized logging. The ELK Stack (Elasticsearch, Logstash, Kibana) is the standard in 2016.

However, Elasticsearch is a memory beast. It relies heavily on the Java Heap and filesystem cache.

# JVM Heap setting for Elasticsearch in /etc/default/elasticsearch
ES_HEAP_SIZE=4g

Do not run this on a standard spinning disk VPS. Indexing logs requires high IOPS. If the disk is slow, Logstash backs up, and you lose visibility. We recently benchmarked CoolVDS NVMe storage against standard SSD VPS providers; the re-indexing time for a 50GB log cluster was 4x faster on NVMe. Speed allows for faster incident resolution.

Conclusion

Microservices offer agility, but they demand rigorous architecture. You need robust patterns for routing (NGINX) and discovery (Consul), but above all, you need infrastructure that respects the laws of physics. Latency is cumulative. Jitter is fatal.

Whether you are deploying Docker Swarm or managing processes with Systemd, the underlying hardware determines your stability. Don't build a Ferrari engine and put it in a go-kart.

Ready to architect for performance? Deploy a high-frequency KVM instance with NVMe storage in our Oslo datacenter today. Experience low latency that keeps your microservices in sync.