Scaling Nginx as an API Gateway: Survival Guide for High-Traffic Norwegian Workloads

It starts with a few timeouts. Then the latency graph on your New Relic dashboard starts looking like a hockey stick. Finally, your inbox floods with alerts: 502 Bad Gateway. Your API isn't just slow; it's dead.

If you are running a monolithic backend or experimenting with these new "microservices" everyone is talking about, the bottleneck is rarely the application logic itself—it's the door you walk through. In 2014, that door is almost certainly Nginx.

Most developers apt-get install Nginx, leave the default config, and wonder why their shiny REST API falls over at 300 concurrent connections. Today, we are going to fix that. We are ripping out the defaults and tuning the Linux kernel for raw throughput.

The "Too Many Open Files" Trap

I learned this the hard way during a Black Friday campaign for a retail client in Oslo last year. We had plenty of RAM, but the API Gateway suddenly stopped accepting connections. The logs were screaming worker_connections are not enough.

Linux, by default, is conservative. It limits the number of file descriptors (sockets count as files) a process can open. On a standard CentOS 6 install, this is often set to 1024. For a high-performance gateway, that is a joke.

Step 1: System-Level Limits

Before you touch Nginx, you must tell the kernel it's okay to work hard. Edit /etc/sysctl.conf to widen the highway:

# /etc/sysctl.conf
fs.file-max = 2097152
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 15

Run sysctl -p to apply. This allows the system to handle more open sockets and recycle them faster—crucial when you have thousands of short-lived API requests hitting your gateway.

Nginx Configuration: Beyond the Defaults

Now, let's look at nginx.conf. The goal here is to maximize concurrency without swapping to disk. If you are using CoolVDS, you likely have access to fast SSD storage, but we still want to keep everything in RAM for that sweet sub-millisecond response time.

The Worker Setup

Nginx is event-based. You don't need a thread per connection (looking at you, Apache prefork). However, you do need to configure your workers correctly.

worker_processes auto; # Detects CPU cores
worker_rlimit_nofile 100000; # Allow Nginx to open this many files

events {
    worker_connections 4096;
    use epoll;
    multi_accept on;
}

The multi_accept on directive tells a worker to accept all new connections at once, rather than one at a time. In a high-latency environment this might be bad, but if you are hosting in Norway with low latency to NIX (Norwegian Internet Exchange), this improves throughput significantly.

The Silent Killer: Upstream Keepalives

This is where 90% of setups fail. By default, Nginx acts as a reverse proxy that opens a new connection to your backend (Node.js, PHP-FPM, Java) for every single request. This is expensive. It burns CPU cycles on TCP handshakes.

You need to enable keepalives between Nginx and your backend application.

upstream backend_api {
    server 127.0.0.1:8080;
    keepalive 64;
}

server {
    location /api/ {
        proxy_pass http://backend_api;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
    }
}

Note the empty Connection header. Without clearing this header, Nginx will forward the "close" header from the client to the backend, killing the persistent connection you just tried to establish.

Pro Tip: If you are using SSL (and you should be, especially after the Heartbleed mess earlier this year), enable ssl_session_cache shared:SSL:10m;. This caches SSL handshake parameters, saving massive amounts of CPU on your gateway.

Infrastructure Matters: The Noisy Neighbor Problem

You can tune sysctl until you are blue in the face, but software cannot fix hardware contention. In the VPS market, overselling is the industry standard. Providers pile 50 customers onto a single physical server, hoping they don't all spike at once.

When your neighbor decides to compile a Linux kernel or run a heavy MySQL import, your CPU "steal time" goes up. Your API latency spikes, not because of your code, but because the hypervisor has put your CPU instructions in a queue.

This is why for mission-critical gateways, we prefer CoolVDS. The architecture relies on KVM (Kernel-based Virtual Machine), which offers strict resource isolation compared to older container-based virtualization like OpenVZ. When you buy 2 vCPUs, they are yours.

Data Sovereignty and Latency

For Norwegian businesses, the physical location of your API gateway is not just a technical detail; it is a legal and performance necessity. With the Data Inspectorate (Datatilsynet) keeping a close watch on how personal data is handled under the Personal Data Act, keeping traffic within national borders is a safe bet.

Hosting in Frankfurt or London adds 20-40ms of latency to every round trip. For an API that makes multiple internal calls, that adds up to a sluggish user experience. Hosting locally in Oslo ensures your packets hit the Norwegian Internet Exchange (NIX) and reach your users instantly.

Final Security Check: Disable SSLv3

Since the POODLE vulnerability was disclosed in October, SSLv3 is officially dead. Ensure your Nginx config explicitly excludes it:

ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

Conclusion

Performance is a stack. It starts with your hardware, moves to the kernel, and ends with your Nginx configuration. Don't let default settings strangle your application's potential.

If you need a platform that respects your need for raw I/O and doesn't treat CPU cycles like a shared commodity, it is time to upgrade. Don't let slow I/O kill your SEO.

Deploy a high-performance KVM instance on CoolVDS in 55 seconds and see the difference.