Crushing Latency: Tuning Nginx as an API Gateway on Linux

Let's be honest: your API isn't slow because your Node.js or Go code is inefficient. It's slow because your TCP stack is garbage and your gateway configuration is stuck in 2012. I recently audited a high-traffic mobile backend hosted in Oslo that was bleeding 200ms on every request solely due to SSL handshakes and connection overhead. In the world of microservices, that latency compounds. If you have five internal service calls, you just killed the user experience.

In April 2016, we have the tools to fix this. HTTP/2 is finally gaining traction, Nginx 1.10 is stable, and NVMe storage is starting to appear in enterprise VPS tiers. But none of that matters if you stick with default settings.

The "Default Config" Trap

Most VPS providers hand you a Linux image optimized for compatibility, not throughput. They use generic kernels and standard file descriptor limits. When you use Nginx as a reverse proxy or API gateway, it acts as a funnel. If that funnel is narrow, it doesn't matter how wide your backend pipe is.

Here is the reality of the Norwegian infrastructure landscape: We have excellent connectivity via NIX (Norwegian Internet Exchange), but if your server performs a disk write to a spinning HDD for every access log, or if your CPU is stolen by a "noisy neighbor" on an oversold OpenVZ node, your 99th percentile latency (p99) will spike unpredictably.

1. Kernel Tuning: Open the Floodgates

Before touching Nginx, we must tune the Linux kernel. The default TCP backlog is often set to 128. For a high-performance gateway, this causes dropped packets during traffic bursts.

Edit your /etc/sysctl.conf. These settings are aggressive but necessary for handling thousands of concurrent connections without sweating.

# /etc/sysctl.conf

# Increase system-wide file descriptors
fs.file-max = 2097152

# Increase the backlog for incoming connections
net.core.somaxconn = 65535
net.core.netdev_max_backlog = 65535

# Reuse sockets in TIME_WAIT state for new connections
# Critical for API gateways making many short-lived requests
net.ipv4.tcp_tw_reuse = 1

# Increase ephemeral port range
net.ipv4.ip_local_port_range = 1024 65535

Apply these with sysctl -p. If you are on a restrictive container system (like older Virtuozzo/OpenVZ setups common in cheap hosting), these commands might fail because you share a kernel. This is why CoolVDS uses KVM virtualization—you get your own kernel to tune. You cannot optimize what you do not control.

2. Nginx: The Gateway Configuration

In 2016, Nginx is the de-facto standard for API Gateways, edging out HAProxy when you need advanced caching or Lua scripting capabilities (via OpenResty). However, the `proxy_pass` directive has a hidden cost: by default, Nginx opens a new connection to your backend (Node/PHP-FPM/Python) for every single request.

This "connection churn" eats CPU cycles and ephemeral ports.

The Keepalive Solution

You must configure an upstream block with keepalives. This keeps the TCP pipe open between Nginx and your application.

# nginx.conf inside http block

upstream backend_api {
    server 127.0.0.1:8080;
    # Keep 64 idle connections open to the backend
    keepalive 64;
}

server {
    listen 80;
    listen 443 ssl http2; # Enable HTTP/2 if you are on Nginx 1.9.5+

    location /api/ {
        proxy_pass http://backend_api;
        
        # REQUIRED for keepalive to work
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        
        # Pass headers
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
    }
}

Pro Tip: Setting proxy_set_header Connection ""; is mandatory. If you forget this, Nginx forwards the client's "Connection: close" header to the backend, killing the keepalive tunnel you just tried to create.

3. SSL/TLS Optimization: The "Tax" on Security

With the recent invalidation of the Safe Harbor agreement (thanks to the Schrems ruling in late 2015) and the rise of Let's Encrypt (out of beta as of roughly now), encryption is non-negotiable. But SSL handshakes are CPU heavy.

To reduce the handshake latency:

1. Session Caching: Allow clients to reuse SSL parameters.
2. OCSP Stapling: Have your server verify the certificate status, sparing the client a DNS lookup to the CA.

# SSL Optimization in Nginx
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;

# OCSP Stapling
ssl_stapling on;
ssl_stapling_verify on;
resolver 8.8.8.8 8.8.4.4 valid=300s;
resolver_timeout 5s;

The Hardware Reality: Why I/O Waits Kill APIs

You can tune software all day, but hardware eventually dictates the ceiling. An API Gateway logs everything. Access logs, error logs, audit trails. On standard SATA SSDs (or worse, HDDs), high concurrency forces the CPU to wait for the disk to finish writing that log line before moving on.

This is "iowait," the silent killer of API performance.

At CoolVDS, we deploy NVMe storage as standard. When your gateway is handling 10,000 requests per second, writing logs to NVMe means the CPU never stalls. Additionally, running in Norway (outside US jurisdiction) is becoming critical for compliance strategies following the Datatilsynet's tightening stance on data sovereignty this year.

Final Thoughts

Building a high-performance API gateway in 2016 requires a holistic view. You need the kernel to handle the connections, Nginx to manage the protocol efficiently, and the underlying hardware to clear the I/O path. If you are still hosting on shared, noisy infrastructure, your `nginx.conf` optimizations are just lipstick on a pig.

Don't let slow I/O or bad peering kill your project. Deploy a KVM-based, NVMe-powered instance on CoolVDS today and see what your API is actually capable of.