502 Bad Gateway? It's Not Your App, It's Your Config.

It’s 2:00 AM. Your monitoring dashboard is bleeding red with 502 errors. You check your backend microservices—Node.js, Go, Python—they’re idling at 5% CPU. The database is napping. So why is the frontend timing out?

Welcome to the bottleneck. In the rush to adopt microservices this year, many teams in Oslo and across Europe forgot that the API Gateway (usually NGINX, Kong, or HAProxy) is a single point of failure if treated like a standard web server. Default configurations are designed for compatibility, not for handling 10,000 concurrent connections per second.

I’ve spent the last month debugging a high-traffic fintech deployment here in Norway. We learned the hard way that without kernel-level tuning and specific NGINX directives, even the most optimized code fails. Here is the exact roadmap we used to fix it, using technologies available right now in late 2018.

1. The OS is Lying to You (Kernel Tuning)

Before you even touch NGINX, you have to look at the Linux kernel. By default, most distributions (even our beloved Ubuntu 18.04 LTS) are conservative. They assume you are running a desktop, not a high-throughput gateway.

The first silent killer is the connection backlog. When a TCP packet arrives, it sits in a queue. If that queue is full, the kernel drops the packet. Your client sees a timeout. You see nothing in the app logs.

Edit your /etc/sysctl.conf. These are the values we use on production CoolVDS instances to ensure the network stack doesn't collapse under load:

# Increase the maximum number of open file descriptors
fs.file-max = 2097152

# Maximize the backlog of incoming connections
net.core.somaxconn = 65535
net.core.netdev_max_backlog = 65535

# Reuse sockets in TIME_WAIT state for new connections (critical for API gateways)
net.ipv4.tcp_tw_reuse = 1

# Increase the range of ephemeral ports
net.ipv4.ip_local_port_range = 1024 65535

# Increase TCP buffer sizes for 10Gbps+ links
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216

Apply these with sysctl -p. If you don't enable tcp_tw_reuse, your gateway will run out of ephemeral ports when talking to upstream services, resulting in the dreaded "Cannot assign requested address" error.

2. NGINX Configuration: The "Upstream" Trap

Most tutorials tell you to set worker_processes auto; and walk away. That is insufficient for an API Gateway.

The biggest mistake I see in 2018 is failing to enable upstream keepalives. By default, NGINX opens a new connection to your backend service for every single request, does the handshake, sends data, and closes it. This burns CPU and adds latency.

Here is a battle-tested snippet for nginx.conf targeting high-performance API routing:

worker_processes auto;
worker_rlimit_nofile 100000;

events {
    worker_connections 4096;
    use epoll;
    multi_accept on;
}

http {
    # ... logs and basics ...

    # OPTIMIZATION 1: Disable disk buffering for proxying
    # We want data to stream to the client, not wait for the disk.
    proxy_buffering off;

    # OPTIMIZATION 2: Upstream Keepalive
    upstream backend_service {
        server 10.0.0.5:8080;
        keepalive 64; # Keep 64 idle connections open
    }

    server {
        listen 80;
        listen 443 ssl http2;
        
        location /api/ {
            proxy_pass http://backend_service;
            
            # REQUIRED for keepalive to work
            proxy_http_version 1.1;
            proxy_set_header Connection "";
        }
    }
}

Pro Tip: Notice proxy_buffering off;? If your clients are mobile apps on 4G networks, turn this on. If your clients are other servers (server-to-server APIs), turn it off to lower latency. Knowing your client profile is half the battle.

3. TLS 1.3 is Here (Use It)

With OpenSSL 1.1.1 finally hitting stable, and the RFC 8446 published just last month (August 2018), we can finally use TLS 1.3. It reduces the handshake from two round-trips to one.

If you are hosting in Norway and serving clients in Oslo, latency is already low. But if you have users connecting from the US or Asia, that handshake reduction is perceptible. Ensure your CoolVDS instance is running the latest OpenSSL and add this:

4. Hardware: The "Noisy Neighbor" Problem

Software tuning only goes so far. In a virtualized environment, "Steal Time" (CPU steal) is the enemy of consistent latency. If your hosting provider oversubscribes their physical cores, your epoll loop waits while another tenant processes their WordPress backups.

For an API Gateway, disk I/O is often the secondary bottleneck, specifically for logging. Every request generates an access log. On standard SATA SSDs, high concurrency writes can block the worker process.

Why Architecture Matters

This is why we built CoolVDS on KVM with strict resource isolation and pure NVMe storage. We benchmarked standard VPS offerings against our NVMe setup using wrk.

The 99th percentile (p99) is what your users actually feel. A p99 of 320ms is unacceptable for a modern API.

5. The Norwegian Context: GDPR and Latency

Since GDPR enforcement began in May, data residency has moved from "nice to have" to "legal necessity." Routing your internal API traffic through a load balancer in Frankfurt or London adds 20-40ms of latency and potential compliance headaches if data inadvertently crosses borders.

By hosting your API Gateway directly in Oslo, you achieve two things:

1. Compliance: Data stays within Norwegian jurisdiction (or EEA), satisfying the Datatilsynet's strict requirements.
2. Speed: Latency from NIX (Norwegian Internet Exchange) to local ISPs like Telenor and Altibox is often under 2ms.

Summary: Don't let default settings kill your project

Building a microservices architecture in 2018 is complex enough without fighting your own infrastructure. Tune your kernel, enable upstream keepalives, and adopt TLS 1.3.

But most importantly, build on a foundation that respects your need for raw I/O and CPU stability. You can implement all the configs above, but if the underlying disk is slow, you will still time out.

Ready to test your API performance? Deploy a CoolVDS NVMe instance in 55 seconds and run your own wrk benchmark. The numbers won't lie.