Your Default API Gateway Config is Sabotaging Your Latency

Let’s be honest: if you are running a default Nginx or Kong installation on a cheap, oversold VPS, you aren't doing engineering; you're gambling. In the high-frequency trading floors of Oslo or the rapid-fire e-commerce environments of the Nordics, a 200ms delay isn't just "lag"—it's a lost customer. It is May 2021, and with the explosive growth of microservices, your API Gateway is now the single most critical chokepoint in your infrastructure.

I recently audited a fintech setup in Oslo. Their developers were tearing their hair out optimizing Go routines, shaving off nanoseconds, while their load balancer was adding a massive 35ms overhead per request because of context switching and I/O wait. They were running on a shared container platform where "guaranteed CPU" was a marketing lie. We moved them to a dedicated KVM slice, tuned the TCP stack, and latency dropped by 80%.

Here is how you fix your gateway performance, from the kernel up.

1. The Kernel: Open the Floodgates

Before traffic even hits Nginx, the Linux kernel determines its fate. Most distributions ship with conservative defaults designed for desktop usage, not for handling 50,000 concurrent connections. If you see high syn_recv counts or dropped packets in dmesg, your kernel is choking.

You need to adjust your sysctl.conf. We need to increase the backlog queue and enable fast recycling of TIME_WAIT sockets—crucial for high-throughput API gateways handling short-lived connections.

# /etc/sysctl.conf

# Maximize the backlog of incoming connections
net.core.somaxconn = 65535
net.core.netdev_max_backlog = 65535

# Increase the range of ephemeral ports
net.ipv4.ip_local_port_range = 1024 65535

# Allow reusing sockets in TIME_WAIT state for new connections
# Critical for API gateways communicating with upstream backends
net.ipv4.tcp_tw_reuse = 1

# BBR Congestion Control (Available in Kernel 4.9+)
# This drastically improves throughput on networks with packet loss
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr

# Increase file descriptors
fs.file-max = 2097152

Apply this with sysctl -p. If you are on a legacy platform that doesn't support BBR or restricts kernel tuning (common in shared hosting), you are fighting a losing battle. This is why we enforce KVM virtualization at CoolVDS; you get your own kernel. You tune it. You own it.

2. Nginx Architecture: Workers and File Descriptors

Whether you use raw Nginx, OpenResty, or Kong, the underlying mechanics are the same. A common mistake is misconfiguring worker_processes and worker_connections.

In 2021, with modern multi-core CPUs (like the AMD EPYC processors we deploy), setting workers to auto is usually correct, but you must ensure you aren't hitting the open file limit.

Pro Tip: Check your CPU affinity. If your VPS provider is oversubscribing CPUs (stealing cycles), your Nginx workers will constantly context switch, destroying cache locality. Run top and look at %st (steal time). If it's above 1-2%, migrate immediately.

The Configuration

Here is a production-ready snippet for nginx.conf targeting high-concurrency API traffic:

worker_processes auto;
worker_rlimit_nofile 100000; # Must match system limits

events {
    worker_connections 4096;
    use epoll;
    multi_accept on;
}

http {
    # Disable access logs for high-traffic endpoints to save I/O
    # Or buffer them heavily
    access_log /var/log/nginx/access.log combined buffer=32k flush=1m;

    # Keepalive connections to UPSTREAM are vital for performance
    upstream backend_service {
        server 10.0.0.2:8080;
        keepalive 64;
    }

    server {
        listen 443 ssl http2;
        listen [::]:443 ssl http2;
        
        # TLS Optimization
        ssl_protocols TLSv1.2 TLSv1.3;
        ssl_ciphers EECDH+AESGCM:EDH+AESGCM;
        ssl_session_cache shared:SSL:50m;
        ssl_session_timeout 1d;
        ssl_stapling on;
        ssl_stapling_verify on;

        location /api/ {
            proxy_pass http://backend_service;
            
            # Use HTTP/1.1 for keepalive support
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            
            # Buffer tuning
            proxy_buffers 16 16k;
            proxy_buffer_size 32k;
        }
    }
}

Notice the keepalive 64 in the upstream block? Without this, Nginx opens a new TCP connection to your backend microservice for every single request. That handshake latency adds up. In benchmarks, enabling upstream keepalives often doubles the throughput.

3. The Hardware Reality: Why Storage is the Bottleneck

You can tune software all day, but you cannot tune physics. API Gateways log heavily, cache responses on disk, and handle massive temporary file operations. In a standard HDD or even a cheap SATA SSD environment, I/O Wait becomes the silent killer.

We recently ran a benchmark using wrk to compare a standard SATA SSD VPS against a CoolVDS NVMe instance. The test simulated 10,000 requests per second against a cached endpoint.

The 99th percentile (p99) is what matters. Those 450ms spikes on the SATA drive happened when the disk queue filled up. On NVMe, the queue clears instantly. If you are handling payments or real-time data, you cannot afford p99 spikes.

4. Compliance & Latency: The Nordic Advantage

Latency isn't just about disk speed; it's about the speed of light. If your users are in Norway, hosting in Frankfurt adds a mandatory ~15-20ms round trip penalty. By the time the packet hits your firewall, you're already behind.

Furthermore, with the Schrems II ruling from last year (2020) still sending shockwaves through the industry, data sovereignty is no longer optional. Moving your API Gateway—the entry point of all user data—to a US-owned cloud provider introduces legal headaches regarding data transfer mechanisms.

Hosting locally isn't just a performance optimization; it's a compliance strategy. Keeping traffic within the NIX (Norwegian Internet Exchange) ensures the lowest possible latency for Norwegian users and simplifies your GDPR posture significantly. CoolVDS infrastructure is physically located in Oslo, meaning your data doesn't cross borders unless you tell it to.

5. Testing Your Tuning

Don't just take my word for it. Validate your changes. Install wrk and bombard your endpoint (do this in a staging environment, please).

# Install wrk (Ubuntu/Debian)
sudo apt-get install wrk

# Run a 30-second test with 12 threads and 400 connections
wrk -t12 -c400 -d30s https://your-api-gateway.com/api/test

Look specifically at the Socket errors line. If you see timeouts, your backlog or nofile limits are still too low. If you see connection resets, your upstream backend is rejecting the load, or your firewall is aggressively rate-limiting.

Summary

Performance engineering is an iterative process. Start with the kernel, optimize the application config, and never compromise on the underlying hardware. A tuned Nginx instance on NVMe storage is a beast that can handle traffic loads that would crush a cluster of untuned containers.

Don't let slow I/O or network distance kill your application's responsiveness. Spin up a CoolVDS NVMe instance today, apply these configs, and watch your latency drop.