Scaling Nginx as an API Gateway: Tuning Linux for High Concurrency

Let’s be honest: default Linux distributions are not designed to handle 10,000 concurrent API requests per second. They are designed for general-purpose computing. If you take a vanilla CentOS 6 or Ubuntu 12.04 installation and slap a REST API on it, you are going to hit a wall. I’ve seen it happen too many times—production launches that turn into firefighting drills because the server ran out of file descriptors or ephemeral ports.

In the Nordic hosting market, where we pride ourselves on engineering precision, sloppy configuration is unacceptable. When your users are connecting from Oslo or Bergen, they expect near-instantaneous responses. If you are routing traffic through a generic US-based cloud, you are already fighting a losing battle against the speed of light. But even with local hosting, your software stack needs to be razor-sharp.

This guide cuts through the noise. We aren’t talking about changing button colors; we are talking about the sysctl.conf flags and Nginx directives that actually move the needle on throughput.

1. The Kernel Bottleneck: It Starts at TCP

Before Nginx even touches a request, the Linux kernel has to accept the TCP connection. By default, the kernel is conservative. It tries to save RAM. But on a modern KVM VPS with dedicated resources (like we provide at CoolVDS), you have the memory. Use it.

The first silent killer of API performance is running out of TCP sockets. When a connection closes, it enters a TIME_WAIT state. If you have high churn (lots of short-lived API calls), you will exhaust your port range.

Open your /etc/sysctl.conf and verify these settings. If they aren't there, add them:

# /etc/sysctl.conf

# Allow reuse of sockets in TIME_WAIT state for new connections
net.ipv4.tcp_tw_reuse = 1

# Increase the range of ephemeral ports
net.ipv4.ip_local_port_range = 1024 65535

# Max number of packets in the receive queue
net.core.netdev_max_backlog = 5000

# Max pending connections
net.core.somaxconn = 4096

# Increase TCP buffer sizes for high-speed connectivity
net.ipv4.tcp_rmem = 4096 87380 6291456
net.ipv4.tcp_wmem = 4096 16384 4194304

Apply these changes with sysctl -p. This ensures your OS doesn't choke when the "Slashdot effect" hits your API.

2. Nginx Configuration: Beyond the Defaults

Nginx 1.4+ is a beast, but only if you tame it. As an API Gateway, Nginx is often acting as a reverse proxy to a backend application (like Python/Django, Ruby on Rails, or the rising Node.js v0.10). The most common mistake I see is failing to enable keepalives to the upstream.

Without keepalives, Nginx opens a new connection to your backend for every single request. That is an enormous waste of CPU cycles on TCP handshakes.

The Upstream Keepalive Fix

Configure your upstream block to hold connections open:

upstream backend_api {
    server 127.0.0.1:8080;
    # Maintain 32 idle connections to the backend
    keepalive 32;
}

And inside your server block, you must set the HTTP version to 1.1 and clear the connection header:

location /api/ {
    proxy_pass http://backend_api;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    
    # Buffering tuning
    proxy_buffers 16 16k;
    proxy_buffer_size 32k;
}

Worker Processes and Open Files

Nginx needs permission to open many files (sockets are files in Unix). Check your limits with ulimit -n. If it says 1024, you are in trouble. Increase this at the OS level, then match it in nginx.conf:

worker_processes auto;
worker_rlimit_nofile 65535;

events {
    worker_connections 2048;
    use epoll;
    multi_accept on;
}

Pro Tip: Do not set worker_processes higher than your CPU core count. Context switching kills performance. On a CoolVDS instance, we map virtual cores directly to physical threads to minimize this latency.

3. The SSL/TLS Tax

Security is not optional, especially with the Datatilsynet (Norwegian Data Inspectorate) keeping a close watch on data privacy. However, SSL handshakes are expensive. If you are terminating SSL at the gateway, you must cache the session parameters so returning clients don't renegotiate from scratch.

Add this to your http block:

ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;

# Prioritize fast ciphers (RC4 is dying, prefer AES-GCM if available in your OpenSSL version)
ssl_ciphers HIGH:!aNULL:!MD5;
ssl_prefer_server_ciphers on;

A 10MB cache can hold roughly 40,000 sessions. This drastically reduces CPU usage on your load balancers.

4. Why Your VPS Provider Matters

You can tune your kernel all day, but if your host is overselling the CPU, your API will suffer from "steal time." This is where the architecture of your infrastructure becomes critical.

Many budget providers use container-based virtualization (like OpenVZ) where you share the kernel with every other customer on the node. If a neighbor decides to mine Bitcoins or compile a massive C++ project, your API latency spikes. There is no isolation.

At CoolVDS, we exclusively use KVM (Kernel-based Virtual Machine). This provides hardware-level virtualization. Your RAM is your RAM. Your CPU cycles are reserved for you. Furthermore, for database-heavy APIs, we are pioneering the use of high-performance SSD storage arrays.

The Norwegian Advantage

Latency is determined by physics. If your users are in Oslo and your server is in Amsterdam, you are adding 15-20ms of round-trip time (RTT) before processing even begins. By hosting on VPS Norway infrastructure connected directly to the NIX (Norwegian Internet Exchange), you drop that network latency to under 2ms for local users.

DDoS protection is also standard here. In 2013 and early 2014, we saw a rise in NTP amplification attacks. Having a provider that filters this at the edge, rather than letting it hit your Nginx interface, is mandatory for uptime.

Summary

Building a high-performance API gateway isn't magic; it's engineering. It requires:

• Kernel Tuning: Opening up the TCP stack.
• Nginx Optimization: Using keepalives and efficient buffering.
• Infrastructure: Choosing KVM over OpenVZ and ensuring local presence.

Don't let slow I/O or bad routing kill your project's reputation. If you need a testing ground that mirrors a high-end production environment, spin up a KVM instance with us.

Ready to lower your latency? Deploy a high-performance instance on CoolVDS today and see the difference dedicated resources make.