API Gateway Latency: Tuning Nginx & Kong for Sub-5ms Response Times in the Post-Schrems II Era

If you represent an API response time in averages, you are already failing. The tail latency—the 99th percentile (p99)—is where your users actually live, and it is where your infrastructure goes to die. In a microservices architecture, the API Gateway is the grand central station. If it lags, the entire mesh creates a cascade of timeouts that no amount of frontend JavaScript can hide.

Following the chaos of the Schrems II ruling in July, many DevOps teams across the Nordics are frantically migrating workloads back to European soil to ensure GDPR compliance. But moving from a hyperscaler to a local VPS provider often exposes hidden performance cracks. You lose the infinite elasticity of the cloud, so you must replace it with raw, optimized efficiency.

I have spent the last week debugging a high-throughput payment processing layer deployed on Ubuntu 20.04. The goal? Stabilize p99 latency under 15ms while handling 10,000 requests per second. Here is the exact tuning playbook we used, focusing on the Linux kernel, Nginx, and the hardware layer.

1. The Kernel: Open the Floodgates

Most default Linux distributions, including the standard Ubuntu 20.04 server image, are tuned for general-purpose computing, not high-concurrency packet switching. When your API gateway gets hit with a traffic spike, the kernel drops packets not because the CPU is maxed, but because the TCP backlog buffers are full.

We need to modify /etc/sysctl.conf to handle thousands of ephemeral connections. The defaults for somaxconn are laughably low (often 128) for a production gateway.

# /etc/sysctl.conf tuning for API Gateways (Sept 2020)

# Increase the maximum number of connections in the backlog queue
net.core.somaxconn = 65535

# Increase the range of ephemeral ports to allow more outgoing connections to upstreams
net.ipv4.ip_local_port_range = 1024 65535

# Enable TCP Fast Open to reduce network latency by one round-trip time (RTT)
net.ipv4.tcp_fastopen = 3

# Reuse sockets in TIME_WAIT state for new connections (critical for high request rates)
net.ipv4.tcp_tw_reuse = 1

# Increase TCP buffer sizes for 10Gbps+ links (common in Nordic datacenters)
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216

Apply these with sysctl -p. If you skip this, your fancy API gateway software will sit idle while the kernel silently discards SYN packets.

2. Nginx & Kong: Stop Closing Connections

Whether you are using raw Nginx or an abstraction layer like Kong (v2.1 is the current stable choice), the mistake is almost always the same: failing to reuse connections to upstream services.

SSL handshakes are expensive. Establishing a TCP connection is expensive. If your gateway opens a new connection to your backend microservice for every single API call, your CPU will burn up in TLS overhead.

The Upstream Keepalive Configuration

In your nginx.conf or Kong template, you must define an upstream block with a keepalive directive. This keeps a cache of open sockets ready for reuse.

upstream backend_service {
    server 10.0.0.5:8080;
    
    # Keep 64 idle connections open to this upstream
    keepalive 64;
}

server {
    location /api/v1/ {
        proxy_pass http://backend_service;
        
        # REQUIRED: HTTP 1.1 is needed for keepalive
        proxy_http_version 1.1;
        
        # Clear the Connection header to prevent the backend from closing it
        proxy_set_header Connection "";
    }
}

Pro Tip: If you are using Kong, check your `nginx_http_upstream_keepalive` setting in `kong.conf`. The default is often too conservative for high-traffic environments. Bump it to at least 60.

3. The Hardware Reality: NVMe or Nothing

This is where the "Pragmatic CTO" meets the "Performance Obsessive". You can tune software all day, but if your underlying storage creates I/O Wait (iowait), your API latency will spike unpredictably. This is particularly true for API gateways that log requests or perform caching.

In 2020, spinning rust (HDD) is obsolete for this workload. Even standard SATA SSDs can choke under heavy concurrent write loads (like logging 5,000 requests/sec). You need NVMe.

When we benchmarked Kong on a standard SATA VPS versus a CoolVDS NVMe instance, the p99 latency dropped from 120ms to 8ms. The bottleneck wasn't the CPU; it was the access logs writing to disk.

4. The "Steal Time" Trap and Local Compliance

Here is the dirty secret of cheap hosting: CPU Steal Time. In a noisy neighbor environment (common with budget providers overselling OpenVZ containers), your VM pauses while the hypervisor services another client. For an asynchronous API gateway, a 50ms pause is a disaster.

We strictly utilize KVM virtualization at CoolVDS. This ensures hardware isolation. Your CPU cycles are yours. Combined with the low latency of the Norwegian Internet Exchange (NIX), you are looking at network round trips within Oslo of under 1ms.

The Schrems II Factor

Since the ECJ invalidated the Privacy Shield framework in July, storing API logs containing PII (Personally Identifiable Information) on US-owned cloud infrastructure is a legal minefield. By hosting your API Gateway on Norwegian soil, owned by a European entity, you simplify your data processing agreements significantly. You get lower latency to your Nordic user base and better sleep for your Data Protection Officer.

5. Verification: Stress Testing with WRK

Don't take my word for it. Verification is part of the job. We use wrk, a modern HTTP benchmarking tool, to smash the gateway and see how it holds up.

# Install wrk (available in Ubuntu 20.04 repos)
sudo apt install wrk

# Run a test: 12 threads, 400 connections, for 30 seconds
wrk -t12 -c400 -d30s --latency http://your-coolvds-ip/api/endpoint

If you see a "Socket errors: connect" message, go back to step 1 and check your somaxconn settings. If you see high latency standard deviation, check your disk I/O.

Final Thoughts

Performance isn't magic; it's physics and configuration. You minimize distance (latency), maximize throughput (bandwidth), and eliminate friction (kernel locks). With the right KVM environment and NVMe storage, your API gateway can handle the heavy lifting of a modern microservices stack without breaking a sweat.

Stop fighting with noisy neighbors and sluggish I/O. Deploy a high-performance NVMe instance on CoolVDS today and see what legitimate raw power does for your p99 metrics.