Stop Accepting Latency: The Art of API Gateway Tuning

If your API response time starts with a digit other than zero, we need to talk. In the world of microservices, the API Gateway is the grand central station. If it jams, the trains don't just stop; they derail. I see too many developers pushing optimized Go or Rust code into a production environment strangled by default Linux kernel settings and an unconfigured Nginx reverse proxy.

It is December 2020. We have NVMe storage. We have 10Gbps uplinks. There is no excuse for slow I/O.

In this guide, I’m putting on my "Performance Obsessive" hat. We aren't just going to apt-get install Nginx; we are going to strip it down and tune it for raw speed, specifically targeting the Norwegian infrastructure context where milliseconds to Oslo or Trondheim matter.

The Hardware Reality: NVMe or Nothing

Before we touch a single config file, look at your infrastructure. Are you running on spinning rust? Or worse, a "cloud SSD" that is actually network-attached storage throttled by a noisy neighbor?

API Gateways are I/O intensive, especially if you are logging heavily or caching payloads. High I/O Wait (iowait) kills throughput. In benchmarks we ran earlier this year using fio, local NVMe storage consistently outperformed network-attached block storage by a factor of 10x in random read/write operations.

Pro Tip: Check your CPU Steal Time. Run top and look at the %st value. If it's above 0.0, your host node is oversold. This is why CoolVDS enforces strict KVM isolation—you don't share CPU time with a crypto-mining neighbor.

Step 1: The Linux Kernel is Not Ready for You

Default Linux distributions (even the excellent Ubuntu 20.04 LTS) are tuned for general-purpose computing, not for handling 50,000 concurrent connections. We need to tell the kernel to open the floodgates.

You cannot do this on shared kernel hosting (like OpenVZ or LXC). You need a KVM instance where you have root access to the kernel parameters.

Edit your /etc/sysctl.conf. These settings optimize the TCP stack for high throughput and fast recycling of connections.

# /etc/sysctl.conf

# Increase system-wide file descriptor limits
fs.file-max = 2097152

# Increase the size of the receive queue
net.core.somaxconn = 65535
net.core.netdev_max_backlog = 65535

# Increase the read/write buffers for TCP
net.ipv4.tcp_rmem = 4096 87380 67108864
net.ipv4.tcp_wmem = 4096 65536 67108864

# Allow reusing sockets in TIME_WAIT state for new connections
# Critical for high-traffic API gateways
net.ipv4.tcp_tw_reuse = 1

# Protection against SYN flood
net.ipv4.tcp_syncookies = 1

# Port range expansion
net.ipv4.ip_local_port_range = 1024 65535

Apply these with sysctl -p. If you are on a platform that forbids this, migrate immediately.

Step 2: Nginx as the Gateway

Nginx 1.18 (Stable) is our weapon of choice. It handles concurrency better than Apache and is lighter than Java-based gateways. However, the default nginx.conf is conservative.

Worker Processes and File Descriptors

The most common error I see in logs during traffic spikes: 24: Too many open files.

Nginx needs permission to open massive amounts of sockets. Here is the configuration strategy:

# /etc/nginx/nginx.conf

user www-data;
# Auto matches number of CPU cores. 
# On a 4-core CoolVDS instance, this spawns 4 workers.
worker_processes auto;

# The limit on maximum open files for worker processes.
# Must be larger than worker_connections.
worker_rlimit_nofile 65535;

events {
    # High concurrency handling
    worker_connections 16384;
    use epoll;
    multi_accept on;
}

Keepalive to Upstream

This is where performance dies. By default, Nginx opens a new connection to your backend service (Node.js, Python, PHP) for every single request. The TCP handshake overhead adds latency.

Configure an upstream block with keepalive connections. This keeps the pipe open.

http {
    upstream backend_api {
        server 127.0.0.1:8080;
        # Keep 64 idle connections open to the backend
        keepalive 64;
    }

    server {
        location /api/ {
            proxy_pass http://backend_api;
            
            # HTTP 1.1 is required for keepalive
            proxy_http_version 1.1;
            
            # Clear the Connection header to persist the link
            proxy_set_header Connection "";
            
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
        }
    }
}

Step 3: TLS 1.3 and Latency

Security is not optional, but it shouldn't be slow. With the release of OpenSSL 1.1.1, we have widespread support for TLS 1.3. It reduces the handshake from two round-trips to one.

Ensure your Nginx build supports it (run nginx -V). This is standard on our Ubuntu 20.04 images.

ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256;
ssl_prefer_server_ciphers on;

# OCSP Stapling allows the server to send the certificate status 
# instead of the browser checking with the CA.
ssl_stapling on;
ssl_stapling_verify on;
resolver 1.1.1.1 8.8.8.8 valid=300s;

The "Schrems II" Reality Check

Technical tuning is useless if your architecture is illegal. In July 2020, the CJEU invalidated the Privacy Shield framework (Schrems II). If you are terminating SSL on an API Gateway hosted on a US-controlled cloud provider, you are traversing a legal minefield regarding GDPR.

Hosting locally in Norway or within the EEA isn't just about latency to the NIX (Norwegian Internet Exchange); it's about compliance. Data sovereignty is the new uptime.

Benchmark: The Proof

We ran a load test using wrk against two setups. Both 4 vCPU, 8GB RAM. Target: A simple JSON echo API.

The difference is not code. It's configuration.

Conclusion

Optimizing an API gateway is an exercise in removing limits. You remove the limits of the file system, the TCP stack, and the SSL handshake. But you cannot remove the limits of bad hardware.

If you need a testing ground that gives you raw NVMe I/O and full KVM kernel control to apply these settings, spin up an instance. Don't let your infrastructure be the reason your users bounce.

Ready to drop your latency? Deploy a high-performance VPS Norway instance on CoolVDS today.