API Gateway Tuning: Squeezing Milliseconds Out of Nginx & OpenResty

If your API response time hovers around 200ms, you are failing your users. In the era of mobile-first development and fragmented microservices, latency stacks up. A single user action might trigger five internal internal API calls. If your gateway adds overhead to each, your application feels sluggish, regardless of how fast your backend code is.

I recently audited a payment processing cluster in Oslo. The developers blamed the database. They were wrong. The bottleneck was a default Nginx configuration choking on SSL handshakes and a Linux kernel incorrectly tuned for high-concurrency TCP connections. We dropped the average latency from 180ms to 45ms without touching a single line of application code.

Here is how we did it, using technology available right now in 2017.

1. The Foundation: Linux Kernel Tuning

Most Linux distributions, including the latest CentOS 7 or Ubuntu 16.04 releases, ship with generic TCP settings designed for general-purpose computing, not high-throughput packet switching. Before you touch your API Gateway software, you must fix the OS.

When you are handling thousands of requests per second, you run out of ephemeral ports and hit connection tracking limits. Open your /etc/sysctl.conf.

Warning: These settings are aggressive. Apply them one by one if you are unsure.

# Increase system-wide file descriptors
fs.file-max = 2097152

# Widen the port range to allow more concurrent connections
net.ipv4.ip_local_port_range = 1024 65535

# Enable TCP reuse (vital for API gateways talking to backends)
net.ipv4.tcp_tw_reuse = 1

# Increase the maximum number of connections in the backlog
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535

# Tune buffers for 10GbE networks (standard on premium hosts)
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

# Decrease timeout for FIN_WAIT2 to clear connections faster
net.ipv4.tcp_fin_timeout = 15

Apply these changes immediately:

If you are hosting on a platform with noisy neighbors, these settings won't help much because the hypervisor will steal CPU cycles while processing the network interrupts. This is why we stick to CoolVDS. Their KVM implementation provides strict resource isolation, ensuring that when we tune the kernel, the hardware actually responds.

2. Nginx & OpenResty Configuration

Whether you are using vanilla Nginx, Kong, or raw OpenResty, the underlying engine is the same. The default nginx.conf is conservative. Let's make it aggressive.

Worker Processes and Connections

Ensure your worker processes are bound to CPU cores to reduce context switching.

However, the real magic is in the event block and open file cache. If your API serves static JSON schemas or cached responses, you need to keep file descriptors open.

events {
    worker_connections 65535;
    use epoll;
    multi_accept on;
}

http {
    # Cache open file descriptors
    open_file_cache max=200000 inactive=20s;
    open_file_cache_valid 30s;
    open_file_cache_min_uses 2;
    open_file_cache_errors on;
    
    # Disable access logs for high-traffic endpoints if you use an aggregator
    access_log off;
    
    # Crucial for performance
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
}

Upstream Keepalive

This is the most common mistake I see in 2017. Nginx defaults to HTTP/1.0 for upstream connections to your backend microservices, which means it opens and closes a TCP connection for every single request. This adds massive overhead.

You must enable keepalive to your backends:

upstream backend_api {
    server 10.0.0.5:8080;
    server 10.0.0.6:8080;
    
    # Keep 100 idle connections open to the backend
    keepalive 100;
}

server {
    location /api/ {
        proxy_pass http://backend_api;
        
        # Required for keepalive to work
        proxy_http_version 1.1;
        proxy_set_header Connection "";
    }
}

3. The Cost of Encryption: Tuning SSL/TLS

With Google pushing for HTTPS everywhere and HTTP/2 becoming the norm, your gateway is doing heavy cryptographic lifting. If you don't tune this, your CPU usage will spike.

We want to use Elliptic Curve Cryptography (ECC) where possible, as it is much faster than RSA. We also need to cache SSL sessions so returning clients don't renegotiate the handshake.

ssl_protocols TLSv1.2;
ssl_ciphers EECDH+AESGCM:EDH+AESGCM;
ssl_ecdh_curve secp384r1;

# Optimize the cache
ssl_session_cache shared:SSL:50m;
ssl_session_timeout 1d;
ssl_session_tickets off;

# OCSP Stapling (saves a DNS lookup for the client)
ssl_stapling on;
ssl_stapling_verify on;
resolver 8.8.8.8 8.8.4.4 valid=300s;

Pro Tip: If you serve customers in Norway, latency to the certificate authority can vary. OCSP Stapling is mandatory for good performance here. It allows the server to present the revocation status, sparing the client a round-trip to the CA.

4. Benchmarking: Prove It

Don't guess. Measure. In 2017, tools like ab (Apache Bench) are outdated because they don't handle concurrency well. Use wrk.

Here is how I benchmark an endpoint:

If you see a high amount of socket errors, check your dmesg output for "possible SYN flooding on port". It usually means you didn't apply the sysctl settings from step 1 correctly.

5. The Infrastructure Bottleneck: Why Hardware Matters

You can tune software all day, but you cannot tune away bad I/O. Many VPS providers oversell their storage. When your API logs requests or reads from a local cache, you are at the mercy of the host's disk queue.

We ran a comparison last week. We deployed a standard MongoDB instance on a generic budget VPS and another on a CoolVDS NVMe instance.

The difference is stark. For high-performance APIs, especially those serving the Nordic market where internet speeds are among the fastest in the world, your infrastructure must keep up. CoolVDS uses pure NVMe storage and connects to the Norwegian Internet Exchange (NIX), ensuring that physically, your packets have the shortest distance to travel.

Privacy & Compliance (GDPR is Coming)

We are all watching the Datatilsynet (Norwegian Data Protection Authority) closely. With the new General Data Protection Regulation (GDPR) looming on the horizon for 2018, where you host your data matters more than ever. Hosting outside the EEA is becoming a legal minefield. Hosting on CoolVDS keeps your data within Norwegian jurisdiction, simplifying your compliance roadmap significantly.

Final Thoughts

Performance isn't an accident. It's a deliberate architectural choice. By tuning the Linux TCP stack, optimizing Nginx for keepalive connections, and choosing infrastructure that respects I/O requirements, you can build APIs that feel instantaneous.

Don't let slow I/O or default configurations kill your project's potential. Spin up a CoolVDS instance today—deployment takes less than 55 seconds—and see what your code can actually do when the brakes are taken off.