Taming the API Gateway: Sub-Millisecond Latency and the Coming GDPR Storm

It is March 2018. If you work in IT anywhere in Europe, you are likely tired of hearing the acronym "GDPR." But while the legal team scrambles to define data processors, we on the operations side have a different problem: Latency.

With the new regulations enforced by Datatilsynet coming into effect this May, many Norwegian companies are pulling data back from US-based clouds to local infrastructure. This is great for compliance, but it exposes a brutal truth: your API Gateway configuration, which looked fine when hidden behind 150ms of transatlantic latency, is actually garbage.

I recently audited a setup for a client in Oslo. They migrated their microservices stack (running on Kubernetes 1.9) to a local data center. They expected instant response times. Instead, they got 502 Bad Gateway errors under load. The hardware wasn't the problem; the default Linux network stack was.

Here is how you fix it, using the exact tuning parameters we implement on the reference architecture at CoolVDS.

1. The OS Layer: Stop Choking on File Descriptors

Most default VPS images ship with conservative limits designed for 2010. When your API Gateway (likely NGINX, HAProxy, or Kong) tries to open thousands of upstream connections to your backend services, the kernel panics.

You need to modify /etc/sysctl.conf. These settings aren't suggestions; they are mandatory for high-traffic endpoints handling thousands of requests per second.

# Increase system file descriptor limit
fs.file-max = 2097152

# Widen the port range for outgoing connections
net.ipv4.ip_local_port_range = 10000 65000

# Allow reuse of sockets in TIME_WAIT state for new connections
net.ipv4.tcp_tw_reuse = 1

# Protection against SYN flood, but increased for high concurrency
net.ipv4.tcp_max_syn_backlog = 4096

# Disconnect dead TCP connections faster (default is usually 7200s!)
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_keepalive_intvl = 15

Apply these with sysctl -p. If you are running on a standard cloud provider, you might find that the hypervisor ignores some of these requests. This is a common issue with OpenVZ or older container-based hosting. At CoolVDS, we use KVM, so the kernel you tune is actually yours to control.

2. NGINX: The "Keepalive" Trap

The most common mistake I see in API Gateway configurations is failing to use keepalive connections to the upstream backends. By default, NGINX closes the connection to the backend service after every request.

This means for every single API call, your gateway has to perform a full TCP handshake with your microservice. That is a massive waste of CPU cycles and adds measurable latency.

Here is the correct configuration structure for your upstream block:

upstream backend_api {
    server 10.0.0.5:8080;
    server 10.0.0.6:8080;
    
    # The magic number. Keeps 64 idle connections open to the backend.
    keepalive 64;
}

server {
    location /api/ {
        proxy_pass http://backend_api;
        
        # REQUIRED for keepalive to work
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        
        # Pass the real IP for audit logs (crucial for GDPR logs)
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

Pro Tip: If you are using SSL termination at the gateway (which you should be), enable OCSP Stapling. It allows the server to send the certificate revocation info during the handshake, saving the browser a separate DNS lookup. In nginx.conf: ssl_stapling on; ssl_stapling_verify on;.

3. The Meltdown/Spectre Factor

We cannot discuss performance in 2018 without addressing the elephant in the room: the Meltdown and Spectre patches released in January. These mitigations introduced context-switching overhead that hit I/O-heavy workloads hard.

If your hosting provider is running older hardware or overselling their CPUs, you are likely seeing "stolen CPU" (st) metrics spike in top. This is fatal for API gateways.

4. Local Nuances: The Norwegian Context

Latency is physics. If your users are in Oslo, Bergen, or Trondheim, routing traffic through a data center in Amsterdam or Ireland adds 20-30ms of round-trip time (RTT). For a real-time bidding API or a financial trading platform, that is unacceptable.

Furthermore, the Norwegian Data Protection Authority (Datatilsynet) is taking a hard line on where data is processed. Hosting on CoolVDS servers physically located in Norway not only drops your ping to sub-5ms for local users but also simplifies your Article 30 records of processing activities.

Conclusion

A fast API gateway is a combination of efficient software configuration and unconstrained hardware. You can tune sysctl all day, but if your disk I/O is saturated by a "noisy neighbor" on a shared platform, your 99th percentile latency will suffer.

Don't let legacy infrastructure throttle your code. Deploy a test instance on a CoolVDS NVMe VPS today and run ab -n 10000 -c 100 against it. The results will speak for themselves.