API Gateway Tuning: Scaling Nginx & OpenResty for Low-Latency Loads

It is 3:00 AM. Your monitoring dashboard is bleeding red. Your API Gateway—supposedly the robust front door to your microservices—is choking on 10,000 concurrent connections. The latency has spiked from 30ms to 800ms, and your customers in Oslo are seeing timeouts.

The problem isn't your code. It’s rarely the code. It’s the default Linux network stack and a vanilla Nginx configuration that assumes you're serving static cat pictures, not high-frequency JSON payloads. If you are running serious infrastructure in 2016, you cannot rely on defaults.

I have spent the last week debugging a high-throughput API for a Norwegian fintech client. They were migrating from a monolith to microservices and used a default Nginx reverse proxy. The result? 502 Bad Gateway errors during peak trading hours. Here is how we fixed it, and how you can tune your gateway to handle the thundering herd.

1. The OS Layer: Tuning the Linux Kernel (4.x)

Before touching Nginx, look at the kernel. The default TCP stack in Ubuntu 16.04 is conservative. It saves memory but sacrifices throughput. For an API gateway, we need to open the floodgates.

The bottleneck usually lies in the ephemeral port range and how quickly the kernel recycles TCP connections. When you have thousands of short-lived connections (common in REST APIs), you run out of ports.

Edit /etc/sysctl.conf. These settings assume you are on a KVM-based system like CoolVDS, where you have actual control over the network stack (unlike container-based hosting where these flags are often locked).

# /etc/sysctl.conf

# Increase the range of ephemeral ports
net.ipv4.ip_local_port_range = 1024 65535

# Allow reuse of sockets in TIME_WAIT state for new connections
# Critical for high-throughput API gateways
net.ipv4.tcp_tw_reuse = 1

# Increase the maximum number of open files (file descriptors)
fs.file-max = 2097152

# Max backlog of connection requests
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535

# Decrease the time default value for tcp_fin_timeout connection
net.ipv4.tcp_fin_timeout = 15

Pro Tip: Do not enable net.ipv4.tcp_tw_recycle. In 2016, with the prevalence of NAT (Network Address Translation) on mobile networks, recycling can cause dropped packets for users behind the same IP. Stick to tcp_tw_reuse.

Apply these changes with sysctl -p. If you are hosted on a budget VPS provider that uses OpenVZ, these commands might fail because you share the kernel with noisy neighbors. This is why for production APIs, we strictly use KVM virtualization at CoolVDS.

2. Nginx Configuration: Beyond the Basics

Most tutorials tell you to set worker_processes auto; and walk away. That is insufficient. For an API Gateway, we need to manage keepalives aggressively to avoid the overhead of SSL handshakes and TCP setups for every single request.

The Worker Setup

In your nginx.conf, we need to increase the file descriptor limits to match our kernel tuning.

worker_processes auto;
worker_rlimit_nofile 65535;

events {
    worker_connections 16384;
    use epoll;
    multi_accept on;
}

Upstream Keepalive

This is the most common mistake I see. Nginx, by default, speaks HTTP/1.0 to upstreams and closes the connection. This destroys performance when talking to backend microservices (Node.js, Go, etc.). You must enable keepalive to your upstream servers.

http {
    upstream backend_api {
        server 10.0.0.5:8080;
        server 10.0.0.6:8080;
        
        # Keep 64 idle connections open to the upstream
        keepalive 64;
    }

    server {
        listen 443 ssl http2;
        server_name api.yourservice.no;

        location / {
            proxy_pass http://backend_api;
            
            # Required for keepalive to work
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
        }
    }
}

Notice the http2 directive in the listen block? HTTP/2 is a game-changer for API performance over high-latency mobile networks. If your clients support it (and most modern libraries do), enable it immediately.

3. SSL/TLS Termination: AES-NI is Mandatory

Encryption is expensive. In 2016, with Let's Encrypt becoming standard, every API should be HTTPS. However, doing the TLS handshake burns CPU cycles.

Ensure your VPS supports the AES-NI instruction set. This allows the processor to offload encryption math. Without it, your "4 vCPU" instance will bottleneck on soft-interrupts (si) long before it hits network limits.

4. Local Context: The Nordic Latency Advantage

If your user base is in Scandinavia, hosting in Frankfurt or London adds 20-30ms of round-trip time (RTT). For a real-time trading API or a high-frequency bidding system, that is an eternity.

Routing traffic through NIX (Norwegian Internet Exchange) in Oslo ensures that local traffic stays local. Furthermore, with the recent Privacy Shield framework replacing Safe Harbor this July, data sovereignty is becoming a boardroom discussion. Keeping your API termination point within Norwegian jurisdiction simplifies compliance with the Personal Data Act (Personopplysningsloven) and prepares you for the tighter EU regulations looming on the horizon.

5. Monitoring and Verification

You cannot improve what you do not measure. Use stub_status in Nginx to watch your active connections.

location /nginx_status {
    stub_status on;
    allow 127.0.0.1;
    deny all;
}

Test your tuning with a tool like wrk (a modern HTTP benchmarking tool available on GitHub). Do not use ApacheBench (ab); it is single-threaded and cannot saturate a tuned CoolVDS link.

# Run a benchmark with 12 threads and 400 connections for 30 seconds
wrk -t12 -c400 -d30s https://api.yourservice.no/v1/ping

Summary

Tuning an API Gateway is an exercise in removing bottlenecks one by one. First, the file descriptors. Second, the TCP stack. Third, the SSL handshake overhead.

But software tuning only gets you so far. If the underlying disk I/O is choking because your host oversold their storage array, no amount of sysctl magic will save you. API Gateways logging access logs to disk are notorious for high I/O wait (iowait). This is why we deploy exclusively on NVMe storage at CoolVDS.

Ready to lower your Time-to-First-Byte? Deploy a CoolVDS instance in Oslo today and test your latency against the competition.