The "Default Config" Trap: Why Your API Gateway is Slow

Let’s be honest: default configurations are for hobbyists. If you apt-get install Nginx on a standard VPS and expect it to handle 10,000 concurrent connections during a Black Friday spike, you’re going to have a bad time. I recently audited a payment processing cluster here in Oslo that was timing out on 5% of requests. The code was fine. The database was indexed. But the API gateway—the front door to their entire infrastructure—was hitting limits hardcoded into the Linux kernel back when 512MB of RAM was considered a luxury.

In the Norwegian market, where users expect near-instant interactions and data privacy (thanks, GDPR) is non-negotiable, you cannot afford latency. If a packet has to travel to Frankfurt and back just to get a handshake, you've already lost milliseconds you can't get back. But even with local hosting, configuration is king.

1. The File Descriptor Bottleneck

Everything in Linux is a file. A socket connection is a file. By default, many distributions ship with a limit of 1024 open files per process. For a high-performance API gateway like Nginx or Kong, this is laughable. When you hit connection 1,025, your gateway simply stops accepting new traffic. You'll see Too many open files in your error logs, but by then, your SLAs are already broken.

Here is how we fix this at the OS level. You need to edit /etc/sysctl.conf to widen the highway:

fs.file-max = 2097152
net.core.somaxconn = 65535
net.core.netdev_max_backlog = 65535
net.ipv4.tcp_max_syn_backlog = 65535
net.ipv4.ip_local_port_range = 1024 65535

Apply these with sysctl -p. But you aren't done. You must specifically tell Nginx to use these descriptors. In your nginx.conf, push the worker limits:

worker_rlimit_nofile 65535;

events {
    worker_connections 20480;
    use epoll;
    multi_accept on;
}

Pro Tip: Don't just blindly copy-paste multi_accept on. In some specific high-load scenarios with limited CPU cores, it can actually decrease performance due to worker contention. Benchmark it. We default to 'off' on smaller CoolVDS instances but 'on' for our dedicated-core plans.

2. Keepalives: The Silent Performance Killer

The most expensive part of an HTTP request is the TCP handshake (and the TLS handshake on top of it). If your API gateway opens a new connection to your backend microservice for every single request, you are wasting CPU cycles and adding massive latency. I've seen 'microservices' architectures where the gateway spent 40% of its time just opening sockets.

You must configure upstream keepalives. This keeps the pipe open between Nginx and your backend (Node.js, Go, Python, etc.).

upstream backend_api {
    server 10.0.0.5:8080;
    # Keep 64 idle connections open to the backend
    keepalive 64;
}

server {
    location /api/ {
        proxy_pass http://backend_api;
        # Required for HTTP/1.1 keepalive to upstream
        proxy_http_version 1.1;
        proxy_set_header Connection "";
    }
}

3. The I/O Wait Trap and the "Noisy Neighbor"

You can tune your kernel all day, but if your underlying disk I/O is garbage, your API will stutter. This is common in shared hosting environments using older container technologies (like OpenVZ) or spinning rust (HDD). When another customer on the same physical host decides to run a massive backup or a database import, your disk I/O wait shoots up. Your CPU sits idle, waiting for data to be read from the disk.

This is why we standardized on NVMe storage for all CoolVDS production instances. The Input/Output Operations Per Second (IOPS) on NVMe are magnitudes higher than SSDs connected via SATA.

Furthermore, virtualization matters. We use KVM (Kernel-based Virtual Machine). Unlike containers that share a kernel, KVM provides better isolation. If a neighbor panics their kernel, your API gateway keeps humming along.

4. TLS 1.3: The New Standard (As of 2018)

With the IETF finally publishing RFC 8446 in August 2018, TLS 1.3 is officially here. It drops the handshake latency significantly by removing a round-trip. If you are serving traffic to mobile devices on 4G networks (common in the Nordics), this latency reduction is noticeable.

To use this today, you need OpenSSL 1.1.1. Check your version:

openssl version

If you are on Ubuntu 18.04, ensure you are updated. Then, configure Nginx to prioritize it:

ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers TLS13-CHACHA20-POLY1305-SHA256:TLS13-AES-256-GCM-SHA384:EECDH+AESGCM:EDH+AESGCM;

5. The GDPR Angle: Location, Location, Location

Since May 25th, the landscape has changed. Datatilsynet (The Norwegian Data Protection Authority) is watching. Hosting your API gateway on a cheap provider in the US or even outside the EEA creates a compliance headache regarding data transfer mechanisms.

Hosting in Norway isn't just about milliseconds (though pinging 1ms to NIX is nice); it's about legal certainty. When you terminate SSL in Oslo, the decrypted data stays within the jurisdiction. It simplifies your Article 30 records of processing activities significantly.

Conclusion

Optimizing an API gateway is a game of millimeters. You tune the kernel to handle the volume, you tune the application to reuse connections, and you ensure the hardware (NVMe and KVM) doesn't betray you. Don't let your infrastructure be the reason your application fails under success.

If you need a test environment that doesn't steal your CPU cycles, spin up a CoolVDS instance. We have the NVMe storage and the low-latency peering already set up. You just bring the code.