The Art of Sub-Millisecond Routing: Tuning NGINX as an API Gateway

Let's be brutally honest: if your API response time drifts above 200ms, your users aren't just annoyed—they're leaving. In the world of high-frequency trading or real-time bidding, we fight for microseconds. Yet, I still see senior engineers deploying stock NGINX configurations on oversold shared hosting and wondering why their 502 Bad Gateway errors spike during traffic surges.

Building a robust API gateway in 2014 requires more than just installing packages. It requires a deep understanding of the Linux TCP stack, file descriptor limits, and the hardware underlying your virtualization. Whether you are serving JSON to mobile apps or XML to legacy banking systems here in Norway, the bottleneck is rarely your application logic—it's how your gateway talks to the kernel.

1. The Foundation: Kernel Tuning

Before touching nginx.conf, we must look at the OS. Default Linux distributions (even the new Ubuntu 14.04 LTS) are tuned for general-purpose desktop usage, not high-throughput packet switching. When acting as an API gateway, your server needs to handle thousands of ephemeral connections.

One of the most common issues we diagnose at CoolVDS is port exhaustion. When NGINX connects to an upstream backend, it uses a local port. If you run out, connections drop.

Here is the baseline /etc/sysctl.conf configuration I deploy for high-load nodes:

# /etc/sysctl.conf

# Maximize the number of open file descriptors
fs.file-max = 2097152

# Allow reusing sockets in TIME_WAIT state for new connections
net.ipv4.tcp_tw_reuse = 1

# Increase the range of ephemeral ports
net.ipv4.ip_local_port_range = 1024 65000

# Increase the maximum backlog of connection requests
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 262144

# Decrease the time default values for tcp_fin_timeout
net.ipv4.tcp_fin_timeout = 15

Apply these with sysctl -p. The tcp_tw_reuse flag is controversial to some, but in a controlled gateway environment where you own both ends of the connection, it is essential for surviving traffic spikes.

2. NGINX 1.6: Configuration for Concurrency

With NGINX 1.6 stable released just last month, we have solid tools for session management. However, the default `worker_connections` setting of 1024 is laughable for production API gateways.

The Worker Process Equation

Your worker processes should generally equal your CPU cores. However, on a CoolVDS KVM instance, you have dedicated cores, meaning you can trust this mapping. On lesser platforms using OpenVZ, "4 cores" might actually mean "4 cores shared with 50 other noisy neighbors," leading to context switching hell.

Pro Tip: Set worker_rlimit_nofile to the limit of your OS file descriptors. If NGINX hits this limit, it stops accepting new connections regardless of your worker capacity.

worker_processes auto;
worker_rlimit_nofile 100000;

events {
    worker_connections 4096;
    use epoll;
    multi_accept on;
}

Upstream Keepalive

This is where most API setups fail. By default, NGINX opens a new connection to your backend (PHP-FPM, Python/Django, or Node.js) for every single request. This adds the overhead of the TCP three-way handshake to every API call. For an API gateway, you must enable keepalive to the upstream.

http {
    upstream backend_api {
        server 10.0.0.5:8080;
        # Keep 100 idle connections open to the backend
        keepalive 100;
    }

    server {
        location /api/ {
            proxy_pass http://backend_api;
            proxy_http_version 1.1;
            proxy_set_header Connection "";
        }
    }
}

Note the proxy_set_header Connection ""; directive. Without this, NGINX forwards the close header to the backend, defeating the purpose of the keepalive.

3. The Hardware Reality: Why I/O Matters

You can tune software all day, but you cannot tune away bad physics. In a recent project migrating a large Norwegian e-commerce retailer (handling post-17. mai sales), we noticed sporadic latency spikes of 500ms+.

The culprit? I/O Wait.

They were hosting on a legacy provider using spinning HDDs in RAID 10. The logs were writing to disk, and the sheer volume of access logs during the sale saturated the drive controller. The CPU was sitting idle, waiting for the disk to confirm the write.

This is why we standardized on Pure SSD storage at CoolVDS. For an API Gateway, logging is intense. If you aren't writing logs asynchronously or to a high-speed SSD, your metrics will suffer.

4. Local Nuances: The Norwegian Context

Hosting outside of Norway introduces latency that physics cannot overcome. A roundtrip from Oslo to a data center in Frankfurt or Amsterdam adds roughly 20-30ms. For a static site, this is negligible. For an API making five internal calls per user action, that adds up to 150ms of dead time.

Furthermore, we must adhere to the Personal Data Act (Personopplysningsloven). Keeping data within Norwegian borders simplifies compliance significantly compared to navigating the complexities of Safe Harbor frameworks with US-based providers. By peering directly at NIX (Norwegian Internet Exchange), CoolVDS ensures your API responses reach Telenor and Altibox fiber users in single-digit milliseconds.

5. Security at Speed

Finally, a fast gateway must be a secure gateway. SSL termination is computationally expensive. If you are using OpenSSL, ensure you are using the latest patched versions (especially after the Heartbleed scare last month).

We recommend enabling OCSP Stapling to speed up SSL handshakes by allowing the server to provide the revocation status, rather than forcing the client to query the CA.

ssl_stapling on;
ssl_stapling_verify on;
resolver 8.8.8.8 8.8.4.4 valid=300s;
resolver_timeout 5s;

Conclusion

Performance isn't an accident; it's an architecture. By tuning your Linux kernel for network throughput, configuring NGINX for upstream persistence, and ensuring your underlying storage infrastructure can handle the I/O pressure, you can achieve an API gateway that scales effortlessly.

Don't let slow I/O or noisy neighbors kill your application's responsiveness. If you are ready to test a platform built for performance obsessives, deploy a CoolVDS SSD instance today. We are live in Oslo, and our KVM virtualization guarantees the resources you pay for are the resources you get.