Scaling API Gateways: Nginx Tuning & Kernel Optimization for High-Load Systems

It’s 3:00 AM. Your monitoring dashboard—probably Nagios or Zabbix if you have taste—is lighting up like a Christmas tree. Your mobile app just got featured, traffic is spiking, and your API gateway is choking. The backend servers are bored, sitting at 5% CPU, but the gateway is dropping connections.

I’ve been there. Last month, I audited a setup for a major Oslo-based e-commerce platform. They were running a standard LEMP stack on a budget VPS. The moment they hit 2,000 concurrent connections, the handshake overhead killed them. It wasn't the code; it was the config.

In 2015, with mobile traffic overtaking desktop, latency is the new downtime. If you aren't tuning your gateway for milliseconds, you're losing users. Here is how we fix it, using Nginx, Linux kernel tweaks, and the right infrastructure.

The Bottleneck is Usually You (Or Your Config)

Most default Linux distributions are tuned for general-purpose computing, not for handling 10,000 simultaneous API requests. When you use Nginx as a reverse proxy or API gateway, you hit limits on file descriptors and TCP stack buffering long before you run out of RAM.

1. The Kernel: Open the Floodgates

Before touching Nginx, we need to tell the OS it's allowed to work hard. Open your /etc/sysctl.conf. We need to tweak how the kernel handles TCP connections, specifically the TIME_WAIT state. Without this, your server runs out of ephemeral ports.

# /etc/sysctl.conf

# Increase system-wide file descriptor limits
fs.file-max = 2097152

# Widen the port range
net.ipv4.ip_local_port_range = 1024 65535

# Recycle zombies. Vital for high-throughput API gateways.
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 15

# Increase the backlog for incoming connections
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535

Run sysctl -p to apply. If you don't do this, Nginx will log "Too many open files" or simple connection timeouts regardless of your upstream capacity.

2. Nginx: The Engine Room

Using Nginx 1.8 or the newer 1.9 branch (if you are brave enough for the experimental HTTP/2 support coming down the pipe) requires moving away from defaults. The directives below focus on keeping connections alive to the backend to avoid the expensive TCP handshake overhead.

worker_processes auto;
worker_rlimit_nofile 100000;

events {
    worker_connections 4096;
    use epoll;
    multi_accept on;
}

http {
    # ... logging and basic settings ...

    # Upstream configuration
    upstream backend_api {
        server 10.0.0.5:8080;
        server 10.0.0.6:8080;
        
        # CRITICAL: Keep connections open to the backend
        keepalive 64;
    }

    server {
        listen 80;
        server_name api.example.no;

        location / {
            proxy_pass http://backend_api;
            
            # Use HTTP/1.1 for keepalive support
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            
            # Buffering tweaks
            proxy_buffers 16 16k;
            proxy_buffer_size 32k;
        }
    }
}

Pro Tip: If you are doing heavy JSON processing or authentication at the gateway level, look into OpenResty. It bundles Nginx with LuaJIT. We use it to handle rate limiting in Redis before the request even hits the PHP/Python application layer.

The "Noisy Neighbor" Problem

You can have the best config in the world, but it means nothing if your CPU creates "wait time" because your neighbor on the physical host is mining Bitcoin or compiling kernels. This is the flaw of container-based virtualization like OpenVZ, which is still sadly common among budget hosts.

In a high-performance API scenario, CPU Steal is the enemy. You need guaranteed cycles.

This is why at CoolVDS, we strictly use KVM (Kernel-based Virtual Machine). It provides true hardware virtualization. If you buy 4 cores, they are your 4 cores. We also swapped out spinning rust for SSDs (and recently NVMe storage where available) because database I/O wait is often the second bottleneck after the network.

Comparison: Virtualization Types

Local Context: Why Hosting in Norway Matters

Latency is physics. If your user base is in Oslo, Bergen, or Trondheim, hosting in a massive data center in Virginia or even Frankfurt adds unavoidable milliseconds. For an API making multiple round-trips per session, that lag compounds.

Furthermore, we have the Personopplysningsloven (Personal Data Act) to consider. While the Safe Harbor agreement is currently being scrutinized by the European courts, keeping data on Norwegian soil—protected by Datatilsynet regulations—is the safest bet for legal compliance and customer trust.

Final Thoughts

Performance isn't an accident. It's a combination of aggressive kernel tuning, intelligent Nginx configuration, and selecting infrastructure that respects your need for dedicated resources. Don't let your API fail because you didn't tweak tcp_tw_reuse or chose a cheap oversold VPS.

Need a sandbox to test these configs? Spin up a KVM instance on CoolVDS. We offer pure SSD storage and direct peering at NIX (Norwegian Internet Exchange) for the lowest possible latency in the region.