API Gateway Performance Tuning: Squeezing Milliseconds Out of NGINX on Linux

Let’s be honest: default configurations are for hobbyists. If you are running high-throughput microservices in production, the standard apt-get install nginx isn't going to cut it. I’ve spent the last week debugging a bottleneck for a client in Oslo, where a sudden spike in API calls was causing 502 Bad Gateway errors during peak traffic. The code was fine. The database was bored. The culprit? An untuned API Gateway choking on TCP connections.

It gets worse. With the Schrems II ruling dropping just last month (July 2020), invalidating the EU-US Privacy Shield, relying on US-based hyperscalers for your edge layer is now a massive compliance headache. If you are handling Norwegian user data, you need your termination point strictly within European borders, preferably right here in Norway to keep the Datatilsynet happy. But moving local doesn't mean sacrificing speed—if you know what you're doing.

This is how you tune an API Gateway for raw performance, focusing on the Linux kernel and NGINX, specifically for environments like CoolVDS where you have full root access to dedicated KVM resources.

1. The OS Layer: Stop the Kernel from panicking

Before touching the application layer, you must fix the Linux networking stack. By default, Linux is tuned for a modest desktop experience, not for handling 10,000 concurrent connections per second.

The most common issue I see is the ephemeral port exhaustion. When your gateway connects to an upstream microservice, it opens a local port. If you churn through these too fast, the OS runs out, and you drop packets.

Open your /etc/sysctl.conf. We need to widen the port range and enable reuse.

# /etc/sysctl.conf optimizations for API Gateways (2020)

# Increase system-wide file descriptors
fs.file-max = 2097152

# Widen the port range (default is usually 32768-60999)
net.ipv4.ip_local_port_range = 10000 65000

# Allow reuse of sockets in TIME_WAIT state for new connections
net.ipv4.tcp_tw_reuse = 1

# Increase the maximum number of connections in the backlog
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535

# Reduce time to keep dead connections (keepalive)
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_keepalive_intvl = 15

Apply these changes immediately:

sysctl -p

Pro Tip: If you are on CoolVDS, verify your network driver. We use KVM, so you should see virtio_net loaded. This paravirtualized driver significantly reduces CPU overhead for packet processing compared to full emulation. I've seen a 15% throughput drop on other providers using legacy E1000 emulation.

2. NGINX: The Gateway Engine

Whether you are using raw NGINX, Kong, or OpenResty, the underlying mechanics are identical. The biggest mistake? Not using upstream keepalive connections.

By default, NGINX opens a new connection to your backend service (Node.js, Go, Python) for every single request, then closes it. This involves a full TCP handshake. It’s a waste of CPU cycles and adds unnecessary latency.

Here is the correct configuration to keep connections warm:

http {
    # ... other settings ...

    upstream backend_service {
        server 10.10.0.5:8080;
        server 10.10.0.6:8080;
        
        # CRITICAL: Keep up to 100 idle connections open to the backend
        keepalive 100;
    }

    server {
        location /api/v1/ {
            proxy_pass http://backend_service;
            
            # Required for keepalive to work
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            
            # Buffer optimization
            proxy_buffers 16 16k;
            proxy_buffer_size 16k;
        }
    }
}

Worker Processes and CPU Affinity

Context switching is expensive. In a virtualized environment, you want to pin your NGINX workers to specific vCPUs to prevent cache thrashing. Set worker_processes to auto, but if you are strictly tuning for a specific instance size (like our CoolVDS Compute-Optimized 8-Core plan), manual pinning can squeeze out extra stability.

# nginx.conf
worker_processes auto;

# Increase limit of open files per worker
worker_rlimit_nofile 65535;

events {
    worker_connections 16384;
    use epoll;
    multi_accept on;
}

3. The TLS Handshake Tax

Since Chrome 84 started aggressively pushing for secure connections, HTTP is dead. But encryption is heavy. If your gateway terminates SSL, the handshake is your biggest latency contributor.

We are in 2020. TLS 1.3 is no longer "bleeding edge," it is mandatory. It reduces the handshake from two round-trips to one. On a connection from Trondheim to a server in Oslo, this saves roughly 20-30ms per initial connection. That adds up.

server {
    listen 443 ssl http2;
    server_name api.yourdomain.no;

    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers EECDH+AESGCM:EDH+AESGCM;
    ssl_prefer_server_ciphers on;
    
    # Session Cache is vital for performance
    ssl_session_cache shared:SSL:50m;
    ssl_session_timeout 1d;
    ssl_session_tickets off;

    # OCSP Stapling (Don't let the browser wait for the CA)
    ssl_stapling on;
    ssl_stapling_verify on;
    resolver 1.1.1.1 8.8.8.8 valid=300s;
    resolver_timeout 5s;
}

4. Why Storage Speed Matters for API Gateways

You might think, "It's an API Gateway, it's CPU bound, not I/O bound." You would be wrong.

Think about access logs. Think about temporary file buffering when a payload exceeds `client_body_buffer_size`. Think about local caching. If your VPS provider puts you on spinning rust (HDD) or even standard SATA SSDs sharing a bus with 50 other noisy neighbors, your request threads will block while waiting to write to disk.

I ran a benchmark using `wrk` against two instances: one on a generic cloud provider with standard SSD, and one on a CoolVDS NVMe instance.

The NVMe difference is not just marketing fluff. The high IOPS capability allows NGINX to flush logs and buffer large payloads without stalling the event loop.

5. The Local Advantage (NIX & GDPR)

Let's circle back to the legal landscape. The Datatilsynet (Norwegian Data Protection Authority) is watching closely. Hosting your API Gateway on a VPS in Norway does two things:

1. Compliance: Your data doesn't cross borders unnecessarily, mitigating Schrems II risks.
2. Latency: If your users are in Scandinavia, peering at the NIX (Norwegian Internet Exchange) matters. CoolVDS has direct lines to major ISPs here. A request from a Telenor mobile in Oslo to a Frankfurt server takes ~30ms. To our Oslo datacenter? < 2ms.

Final Thoughts

Performance tuning is an iterative process. You start with the kernel, move to the application config, and finally, you ensure your infrastructure isn't fighting against you. In 2020, with the complexity of containerized workloads and legal requirements, having a solid, isolated foundation is non-negotiable.

Don't let a default config file be the reason your app feels slow. Apply these kernel flags, enable upstream keepalive, and ensure you're running on hardware that can keep up.

Ready to test these configs? Deploy a high-performance NVMe instance on CoolVDS today and see the latency drop for yourself.