Crushing Latency: Tuning NGINX as an API Gateway for High-Load Microservices

It is 2017. The monolith is dying, and we are all rushing to slice our applications into microservices. It sounds perfect on paper until you realize you’ve just traded function calls for network calls. Suddenly, your frontend isn't talking to a backend; it's talking to an API Gateway that routes traffic to ten different services.

And that Gateway is choking.

I recently audited a Norwegian e-commerce platform that collapsed during the Romjula sales. Their backend services were written in Go and incredibly efficient. The database was fine. The problem? Their NGINX gateway was default-configured, running on a budget VPS with magnetic storage. Every request added 150ms of overhead. That is unacceptable. In the Nordic market, where fiber penetration is high and users expect instant loads, latency is a business killer.

The Hidden Cost of the "Microservices Tax"

When you place a reverse proxy or API gateway in front of your services, you introduce a hop. If that hop isn't optimized, you introduce jitter. In high-frequency trading or real-time bidding, we obsess over microseconds. For a standard REST API serving JSON, we should at least obsess over consistency.

Most developers install NGINX, leave the worker_connections at 768, and wonder why 502 Bad Gateway errors spike when traffic hits. Let's fix that.

Step 1: The OS Is Your Foundation

Before touching the web server config, you must tune the Linux kernel. Default Linux distros like Ubuntu 16.04 LTS are tuned for general-purpose desktop or light server usage, not for handling 50,000 concurrent connections.

You need to widen the TCP pipe. Open your /etc/sysctl.conf and apply these settings. These are aggressive, designed for a dedicated API gateway handling bursty traffic.

# /etc/sysctl.conf

# Maximize the number of open file descriptors
fs.file-max = 2097152

# Increase the read/write buffer sizes for TCP
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216

# Increase the number of incoming connections backlog
net.core.somaxconn = 32768

# Allow reusing sockets in TIME_WAIT state for new connections
net.ipv4.tcp_tw_reuse = 1

# Protection against SYN flood
net.ipv4.tcp_syncookies = 1

# Increase the range of ephemeral ports
net.ipv4.ip_local_port_range = 1024 65535

After saving, run sysctl -p. Without this, NGINX will hit the OS ceiling long before it hits its own limits.

Pro Tip: Check your `ulimit`. If you are running NGINX as the `www-data` user, ensure `/etc/security/limits.conf` allows enough open files. A setting of `nofile 65535` is a minimum starting point for production gateways.

Step 2: NGINX Configuration for Throughput

Now, let's look at the gateway configuration. Whether you are using raw NGINX, OpenResty, or Kong (which is gaining traction quickly here in Europe), the underlying NGINX directives remain the same.

The goal is to keep connections alive to the upstream backends. Establishing a TCP handshake and negotiating SSL for every single API call between the Gateway and the Microservice is CPU suicide.

Here is a reference `nginx.conf` structure for high-performance proxying:

worker_processes auto;
worker_rlimit_nofile 65535;

events {
    worker_connections 16384;
    use epoll;
    multi_accept on;
}

http {
    # ... logging and other settings ...

    # Upstream configuration with Keepalive
    upstream backend_service {
        server 10.0.0.5:8080;
        server 10.0.0.6:8080;
        
        # CRITICAL: Keep idle connections open to the backend
        keepalive 64;
    }

    server {
        listen 80;
        listen 443 ssl http2;
        
        location /api/v1/ {
            proxy_pass http://backend_service;
            
            # Use HTTP/1.1 for keepalive support
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            
            # Buffer tuning
            proxy_buffers 16 16k;
            proxy_buffer_size 32k;
        }
    }
}

Why `keepalive` matters

Without keepalive 64; and proxy_http_version 1.1;, NGINX closes the connection to your Go or Node.js service after every request. This forces your internal network to churn through ephemeral ports and wastes CPU cycles on handshakes.

Step 3: The Hardware Reality (The CoolVDS Factor)

You can have the best config in the world, but if your underlying infrastructure is noisy, your P99 latency will suffer. In 2017, many VPS providers are still pushing magnetic spinning disks (HDD) or overselling their CPU cores using older OpenVZ containers.

For an API Gateway, I/O wait is the enemy. Every API request generates logs (access logs, error logs). If your disk cannot write those logs fast enough, the NGINX worker process blocks. This is where NVMe storage becomes non-negotiable.

At CoolVDS, we use KVM virtualization. This ensures that a neighbor compiling a massive C++ project doesn't steal the CPU cycles your gateway needs for SSL termination. When you are terminating TLS for thousands of concurrent users, you need every cycle of that AVX instruction set.

Step 4: Benchmarking Your Success

Do not guess. Measure. In 2017, ab (Apache Bench) is showing its age. I recommend using wrk for a more modern approach to generating significant load.

Run this from a separate machine (latency to the test target matters, so try to test from a server in the same region, e.g., Oslo):

# Run for 30 seconds, using 12 threads and 400 open connections
wrk -t12 -c400 -d30s https://api.yourdomain.no/v1/status

Look at your Latency Distribution. Average latency is a vanity metric. Pay attention to the 99% percentile. If your 99% is under 50ms, you are doing well. If it spikes to 1s, check your CPU steal time or disk I/O.

Data Privacy and Geography

With the GDPR implementation date looming next year (2018), and Datatilsynet becoming stricter about where data lives, hosting your API Gateway inside Norway is a strategic advantage. It reduces round-trip time (RTT) for your local users and simplifies your compliance posture regarding data transit.

Final Thoughts

Performance tuning is an iterative process. Start with the kernel, secure the NGINX configuration, and ensure your hardware isn't fighting against you. If you are tired of debugging latency spikes caused by "noisy neighbors" on budget hosting, it is time to upgrade.

Don't let slow I/O kill your application's reputation. Deploy a test instance on CoolVDS today and see what KVM and NVMe can do for your throughput.