Scaling Nginx as an API Gateway: Kernel Tuning & Architecture for Sub-Millisecond Latency

The monolith is dying, but the replacement isn't painless. If you have spent the last year decoupling your application into microservices, you have likely run into the new enemy: latency accumulation. In 2015, the API Gateway is no longer just a reverse proxy; it is the single most critical component in your infrastructure. It handles authentication, rate limiting, and routing, often becoming the choke point that turns a snappy app into a sluggish mess.

I recently audited a setup for a fintech client in Oslo. They were pushing 15,000 requests per second (RPS) through a default Nginx install on a budget VPS. The result? 502 Bad Gateways during peak trading hours and a latency tail that looked like a horror movie. The CPU wasn't maxed out; the software interrupt queues were.

Here is how we fixed it, and how you can tune your stack to handle high-concurrency API traffic without melting your servers.

1. The "Safe Harbor" Elephant in the Room

Before we touch `sysctl.conf`, let's address the legal landscape. With the ECJ's recent invalidation of the Safe Harbor agreement (Schrems I) last month, hosting API data on US-controlled clouds is now a massive liability. If you are processing data for Norwegian users, latency isn't your only concern—so is Datatilsynet.

Moving your API Gateway to a sovereign Norwegian host isn't just about millisecond gains to the NIX (Norwegian Internet Exchange); it is about compliance. However, moving away from massive public clouds means you need to manage your own performance. You don't have an Elastic Load Balancer to hide behind anymore.

2. Nginx: The Gateway Configuration

Nginx is the de facto standard for API gateways today, beating out HAProxy in versatility due to the OpenResty (Lua) ecosystem. But the default `nginx.conf` is optimized for serving static files, not high-throughput JSON piping.

Upstream Keepalives are Mandatory

By default, Nginx opens a new connection to your backend service for every single request. In a microservices architecture, this TCP handshake overhead is fatal. You must enable keepalives to the upstream.

upstream backend_api {
    server 10.0.0.5:8080;
    server 10.0.0.6:8080;
    
    # The Critical Setting
    keepalive 64;
}

server {
    location /api/ {
        proxy_pass http://backend_api;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
    }
}

Pro Tip: Without `proxy_set_header Connection "";`, Nginx will forward the "close" header to the backend, rendering the keepalive useless. I see this mistake in 90% of the audits I perform.

Enable SO_REUSEPORT

If you are running a modern kernel (Linux 3.9+) and Nginx 1.9.1+ (which you should be), use `SO_REUSEPORT`. This allows multiple worker processes to bind to the same port, letting the kernel distribute incoming connections directly to the workers. This reduces lock contention on the accept mutex.

listen 80 reuseport;
listen 443 ssl reuseport;

3. Kernel Tuning for API Workloads

Your application is only as fast as the OS allows it to be. When handling thousands of small API requests, the Linux networking stack often becomes the bottleneck before the application logic does.

Edit your /etc/sysctl.conf to widen the TCP highway. We need to allow more open files and handle connections in the TIME_WAIT state faster.

# Max open files
fs.file-max = 2097152

# Increase the TCP backlog queue
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535

# Reduce time spent in TIME_WAIT
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_tw_reuse = 1

# Increase port range for outgoing connections (critical for gateways)
net.ipv4.ip_local_port_range = 1024 65535

Apply these with sysctl -p. If you ignore `ip_local_port_range` on a high-traffic gateway, you will run out of ephemeral ports and start dropping connections to your backend services.

4. The Hardware Reality: Why Virtualization Matters

You can tune software all day, but if your underlying disk I/O is fighting for scraps, your API latency will spike unpredictably. This is the "Noisy Neighbor" problem common with budget VPS providers using container-based virtualization like OpenVZ.

In an OpenVZ environment, the kernel is shared. If another tenant on the physical node decides to run a heavy database import, your API gateway suffers because the disk I/O wait times skyrocket.

At CoolVDS, we exclusively use KVM (Kernel-based Virtual Machine) virtualization. This provides true hardware isolation. When you run an API gateway on our infrastructure, the RAM and CPU cycles are yours. We also utilize enterprise-grade SSDs in RAID 10 arrays. For an API gateway logging thousands of access requests per second, spinning rust (HDDs) is a death sentence in 2015.

5. SSL Termination and the CPU Cost

With Google now using HTTPS as a ranking signal, SSL is mandatory. However, the handshake is CPU intensive. To maintain low latency:

• Use ECDHE ciphers: They are faster and more secure than traditional RSA.
• Session Resumption: Enable `ssl_session_cache`. This allows clients to reuse SSL parameters, skipping the heavy handshake on subsequent requests.
• OCSP Stapling: Saves the client a DNS lookup and connection to the Certificate Authority.

ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
ssl_stapling on;
ssl_stapling_verify on;

Conclusion

Building a high-performance API gateway is a balancing act between kernel limits, Nginx configuration, and hardware capabilities. Do not let the "default settings" complacency kill your user experience. With the shifting legal sands of European data privacy, hosting locally in Norway on robust KVM infrastructure is the smartest move for stability and compliance.

If you need a testing ground that mimics a production environment with dedicated resources, don't gamble on oversold containers. Deploy a KVM instance on CoolVDS today and see what your API is actually capable of when the brakes are taken off.