Scaling Nginx as an API Gateway: Tuning for Sub-10ms Latency

Your API is the nervous system of your application. If the gateway lags, the entire user experience collapses. I recently audited a mobile backend deployed for a retail client in Oslo. Their architecture was sound—Microservices, RESTful endpoints—but their latency was spiking to 400ms during flash sales. The culprit? A default Nginx configuration running on a shared, oversold VPS hosted outside the EEA.

With the recent invalidation of the Safe Harbor agreement, hosting data within Norway or Northern Europe isn't just about latency anymore; it's a compliance necessity. But moving to a local datacenter won't save you if your TCP stack is choking on connections.

Here is how we tune an API Gateway to handle thousands of requests per second without sweating, using technology available right now in 2016.

1. The OS Layer: Breaking the Limits

Before touching Nginx, look at the kernel. Most distributions like CentOS 7 ship with conservative defaults intended for desktop usage, not high-performance packet switching.

When you act as an API Gateway, you burn through file descriptors and ephemeral ports rapidly. If you see Too many open files in your logs, your OS is killing your traffic.

Kernel Tuning for High Concurrency

Edit your /etc/sysctl.conf. We need to widen the TCP port range and allow faster recycling of sockets in the TIME_WAIT state. This is critical for REST APIs where short-lived connections are the norm.

# /etc/sysctl.conf

# Increase system file descriptor limit
fs.file-max = 2097152

# Widen the port range
net.ipv4.ip_local_port_range = 1024 65535

# Recycle zombies
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 15

# Protect against SYN floods while allowing burst traffic
net.ipv4.tcp_max_syn_backlog = 4096
net.ipv4.tcp_syncookies = 1

# Increase the connection tracking table (Critical for firewalls)
net.netfilter.nf_conntrack_max = 1048576

Apply these with sysctl -p.

Pro Tip: Many budget VPS providers use OpenVZ or LXC containers that share a kernel with the host. In those environments, you often cannot modify nf_conntrack_max or tcp_tw_reuse. This is why CoolVDS relies strictly on KVM virtualization. You get your own kernel. You get full control. If you need to tune the stack, we don't stand in your way.

2. Nginx Configuration: Beyond the Defaults

The standard nginx.conf is designed to be safe, not fast. For an API Gateway, we need to minimize disk I/O (logging) and maximize network throughput.

Worker Processes and Connections

Ensure Nginx is utilizing every core available. In 2016, with the rise of multi-core virtual instances, hardcoding this is obsolete. Use auto.

worker_processes auto;
worker_rlimit_nofile 100000; # Must exceed worker_connections

events {
    worker_connections 4096;
    use epoll;
    multi_accept on;
}

Upstream Keepalives

This is the most common mistake I see. Nginx speaks HTTP/1.1 to the client, but often drops to HTTP/1.0 when talking to your backend app servers (Node.js, Python, PHP-FPM), closing the connection after every request. This adds massive overhead.

You must force HTTP/1.1 and define a connection pool for the upstream.

http {
    upstream backend_api {
        server 10.0.0.5:8080;
        server 10.0.0.6:8080;
        
        # Keep 64 idle connections open to the backend
        keepalive 64;
    }

    server {
        location /api/ {
            proxy_pass http://backend_api;
            
            # essential for keepalive to work
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            
            # Pass IP info accurately
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }
    }
}

3. The SSL Handshake Bottleneck

With Google now ranking HTTPS sites higher and HTTP/2 requiring encryption, SSL is mandatory. However, the initial handshake is CPU intensive. If you are terminating SSL at the gateway, you must cache sessions.

ssl_session_cache shared:SSL:50m;
ssl_session_timeout 1d;
ssl_session_tickets off;

# Modern cipher suite (2016 standard)
ssl_protocols TLSv1.1 TLSv1.2;
ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:kEDH+AESGCM';
ssl_prefer_server_ciphers on;

A 50MB cache can store roughly 200,000 sessions. This prevents the server from renegotiating the handshake for every recurring API call from a mobile client.

4. Hardware Matters: The I/O Trap

You can tune software all day, but you cannot tune away bad physics. API Gateways logging access requests to disk can become I/O bound instantly.

Buffer your logs:

access_log /var/log/nginx/access.log main buffer=16k flush=2s;

Better yet, ensure the disk underneath is fast. Standard spinning HDDs (SAS/SATA) cannot handle the random write patterns of high-traffic logging. Even standard SSDs can choke if the