Squeezing Milliseconds: API Gateway Tuning for High-Throughput Microservices

It’s June 2018. The GDPR panic is finally settling down. You’ve updated your privacy policies, you’ve audited your data processors, and you’re hopefully compliant with Datatilsynet. But while you were busy worrying about Article 28, your API latency crept up. Your mobile app users in Oslo are seeing spinning wheels, and your microservices are choking under load.

I recently audited a fintech setup deployed near the NIX (Norwegian Internet Exchange). Their architecture was sound—Docker containers on Ubuntu 18.04—but their API Gateway (based on Nginx) was hitting a wall at 500 requests per second. The hardware wasn't the problem; the configuration was.

If you treat your API Gateway like a standard web server, you will fail. Here is how we tune the stack for raw throughput, minimizing overhead to the microsecond.

1. Stop killing your Upstream Connections

The single most common mistake I see in microservices architectures involves TCP handshakes. By default, Nginx acts as a courteous proxy: it opens a connection to your backend service, sends the request, receives the response, and closes the connection.

For a monolithic app, this is fine. For microservices where a single user action triggers ten internal API calls, this is suicide. You are wasting CPU cycles on TCP SYN/ACK handshakes and ephemeral port exhaustion.

You must enable keepalive connections to your upstreams.

upstream backend_microservice {
    server 10.0.0.5:8080;

    # The Critical Fix: Keep 64 idle connections open
    keepalive 64;
}

server {
    location /api/v1/ {
        proxy_pass http://backend_microservice;
        
        # Required to make keepalive work
        proxy_http_version 1.1;
        proxy_set_header Connection "";
    }
}

The Result: In our tests, this change alone dropped internal latency by 35ms per request chain.

2. Kernel Tuning: The sysctl.conf Essentials

Linux out-of-the-box is tuned for general desktop usage, not for handling 10,000 concurrent connections. If you are running on a standard VPS, you might be hitting limits defined effectively in the 90s.

Open /etc/sysctl.conf. We need to widen the ephemeral port range and allow faster recycling of TIME_WAIT sockets. Warning: Understand these flags before applying them. `tcp_tw_recycle` is dangerous in NAT environments (like some public clouds), so we stick to `reuse`.

# Increase system file descriptor limit
fs.file-max = 100000

# Allow more connections to be queued
net.core.somaxconn = 4096

# Widen the port range for outgoing connections
net.ipv4.ip_local_port_range = 1024 65535

# Reuse sockets in TIME_WAIT state for new connections
net.ipv4.tcp_tw_reuse = 1

# TCP Fast Open (TFO) reduces handshake round trips
net.ipv4.tcp_fastopen = 3

Apply these with sysctl -p. If you are on a restrictive virtualization platform (like old OpenVZ containers), these might fail. This is why we at CoolVDS use KVM (Kernel-based Virtual Machine). You need your own kernel to do serious tuning.

3. The "I/O Wait" Killer

Logging is expensive. Standard Nginx configurations write access logs to disk for every single request. If you have a traffic spike, and your disk is a spinning HDD (or a cheap, throttled SATA SSD), your Nginx worker processes will block while waiting for the disk to confirm the write.

Pro Tip: If you don't need every single access log for compliance, buffer them. Or better yet, disable access logs for static assets entirely.

http {
    # Buffer logs: write only when 64k is full or flush every 5m
    access_log /var/log/nginx/access.log combined buffer=64k flush=5m;
    
    # Disable logging for favicon/robots to save I/O
    location = /favicon.ico {
        log_not_found off;
        access_log off;
    }
}

However, buffering has a risk: if the server crashes, you lose the last 64k of logs. This is where hardware matters. We standardized on NVMe storage for all CoolVDS instances because the IOPS throughput is massive. With NVMe, blocking on I/O is rarely the bottleneck, allowing you to keep stricter logging without the performance penalty—crucial for GDPR audit trails.

4. SSL Termination: Tuning the Handshake

With Chrome marking HTTP sites as "Not Secure" aggressively this year, SSL is mandatory. But RSA handshakes are CPU heavy.

1. Session Cache: Don't make the client handshake every time. Cache the SSL parameters.
2. OCSP Stapling: Save the client a trip to the Certificate Authority.

ssl_session_cache shared:SSL:50m;
ssl_session_timeout 1d;
ssl_session_tickets off;

# OCSP Stapling
ssl_stapling on;
ssl_stapling_verify on;
resolver 1.1.1.1 8.8.8.8 valid=300s;
resolver_timeout 5s;

Note: We are using Cloudflare's 1.1.1.1 (released just a few months ago) as the resolver here for speed.

5. The Norway Factor: Latency and Law

Physics is stubborn. If your servers are in Frankfurt but your customers are in Bergen, you are adding 20-30ms of round-trip time (RTT) purely on distance. For an API Gateway handling multiple sequential requests, that lag compounds.

Hosting locally in Norway isn't just about speed; it's about Data Sovereignty. With the ink barely dry on GDPR enforcement (May 25th), keeping Norwegian user data within national borders simplifies your compliance posture with Datatilsynet significantly. You don't have to worry about the complexities of Privacy Shield if the data never leaves the jurisdiction.

Final Thoughts

You can have the cleanest code in the world, but if your kernel is choking on TIME_WAIT sockets or your worker processes are waiting on disk I/O, your API will feel sluggish. Tuning requires a holistic view: from the sysctl flags up to the Nginx upstream config, supported by hardware that doesn't blink under load.

Don't let legacy infrastructure throttle your growth. If you need a sandbox to test these configurations without the "noisy neighbor" effect, spin up a high-performance instance with us.

Deploy your API Gateway on a CoolVDS NVMe instance today—provisioned in under 55 seconds.