Tuning Nginx as an API Gateway: Surviving the 10k Connection Problem

Let’s be honest: the traditional LAMP stack is struggling to keep up. With the explosion of mobile apps in the last two years, we aren't just serving HTML pages anymore; we are serving thousands of small JSON payloads every second. Your Apache prefork configuration from 2010 isn't going to cut it when your iOS app gets featured on the App Store.

I recently audited a Magento deployment for a client in Oslo. They were running on a standard spinning-disk VPS. During a mild traffic spike, their API latency shot up from 200ms to 4 seconds. The CPU wasn't the bottleneck—I/O wait was. The disk heads were thrashing trying to write access logs while reading session files. We moved them to a high-performance SSD instance and tuned their gateway. The result? 45ms consistent response times.

In this guide, we are going to tear down the default Nginx configuration and rebuild it as a high-performance API gateway suitable for the modern RESTful web. We will look at kernel tuning, Nginx directives, and why hardware choice (specifically SSD vs. HDD) is the single biggest factor in API stability.

The Architecture: Nginx as the Shield

In 2014, the best practice is decoupling. Do not let your heavy application servers (Python/Django, Ruby on Rails, or PHP-FPM) touch the public internet directly. They are too slow at handling connection handshakes. Instead, we use Nginx 1.4+ as a reverse proxy/gateway.

Nginx handles the thousands of concurrent TCP connections, buffers the requests, and hands them off to the backend only when the request is fully received. This prevents the "Slowloris" attack and frees up your backend workers to do what they do best: execute logic.

Pro Tip: If you are hosting in Norway, latency to the Norwegian Internet Exchange (NIX) matters. A gateway located in Frankfurt adds 20-30ms of round-trip time (RTT) for your Oslo users. Hosting locally ensures that your TCP handshake—which requires 3-way communication—is lightning fast.

Step 1: The Operating System (sysctl.conf)

Before touching Nginx, we must fix the Linux kernel defaults. Most distributions like CentOS 6.5 or Ubuntu 12.04 LTS ship with conservative limits designed for desktop usage, not high-throughput servers.

Open /etc/sysctl.conf. We need to increase the range of ephemeral ports and allow faster recycling of TCP connections. Without this, your API gateway will run out of sockets under load (TIME_WAIT state exhaustion).

# /etc/sysctl.conf

# Increase system-wide file descriptor limit
fs.file-max = 2097152

# Widen the port range to allow more concurrent connections
net.ipv4.ip_local_port_range = 1024 65535

# Enable reusing sockets in TIME_WAIT state for new connections
net.ipv4.tcp_tw_reuse = 1

# Reduce the time a connection stays in FIN-WAIT-2
net.ipv4.tcp_fin_timeout = 15

# Increase the maximum backlog of connection requests
net.core.somaxconn = 65535
net.core.netdev_max_backlog = 65535

Apply these changes with sysctl -p. Note: You need root access to do this. This is why shared hosting is dead for serious APIs. You need a VPS or a Dedicated Server where you control the kernel parameters. At CoolVDS, our KVM virtualization ensures you have full isolation and authority to modify these flags.

Step 2: Nginx Configuration for APIs

The default nginx.conf is decent, but it's optimized for general file serving, not high-concurrency JSON shuffling. Here is the reference configuration I use for production API gateways.

Worker Processes and Connections

We want Nginx to utilize all CPU cores and handle as many connections as the file descriptor limit allows.

worker_processes auto; # Requires Nginx 1.3.8+
worker_rlimit_nofile 100000;

events {
    worker_connections 4096;
    use epoll;
    multi_accept on;
}

Buffers and Timeouts

APIs often receive large JSON bodies (POST requests) or return large datasets. If the buffers are too small, Nginx writes to a temporary file on the disk. This is a performance killer if you are on standard spinning rust (HDD).

http {
    # ... basic settings ...

    # Optimize for packet size
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;

    # Buffers - keep data in RAM, avoid disk I/O
    client_body_buffer_size 128k;
    client_max_body_size 10m;
    client_header_buffer_size 1k;
    large_client_header_buffers 4 4k;
    output_buffers 1 32k;
    postpone_output 1460;

    # Timeouts - Fail fast. Don't hold connections open for slow clients.
    client_header_timeout 10s;
    client_body_timeout 10s;
    send_timeout 10s;
    
    # Keepalive is crucial for API clients making multiple requests
    keepalive_timeout 15s;
}

Upstream Keepalive

This is the most common mistake I see. Nginx talks to your backend (e.g., Node.js on port 3000 or PHP-FPM) via HTTP. By default, Nginx opens a new connection to the backend for every single request, then closes it. This adds unnecessary overhead.

Use the keepalive directive in your upstream block to maintain persistent connections to your application.

upstream api_backend {
    server 127.0.0.1:3000;
    # Maintain 64 idle connections to the backend
    keepalive 64;
}

server {
    location /api/ {
        proxy_pass http://api_backend;
        
        # Required for HTTP/1.1 keepalive to backend
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

The Hardware Reality: SSD vs. HDD

You can tune software all day, but physics is physics. API workloads are characterized by random I/O patterns—logging access, reading database rows, updating session stores. Standard 7200 RPM Hard Drives provide roughly 80-120 Input/Output Operations Per Second (IOPS). That is your hard ceiling.

If your API receives 200 concurrent requests, and each request triggers a log write and a database lookup, a mechanical drive will queue operations. This manifests as "iowait" in top, and your users see a spinning wheel.

The solution is Solid State Drives (SSD).

At CoolVDS, we don't upsell SSDs as a "premium" feature; we made it our standard. For high-frequency trading or real-time APIs, the latency reduction is not a luxury—it is a requirement.

Data Privacy in Norway

When operating an API gateway, you are likely logging IP addresses. Under the Norwegian Personal Data Act (Personopplysningsloven), IP addresses are considered personal data. The Data Protection Authority (Datatilsynet) is strict about how long this data is retained.

To stay compliant while maintaining performance, rotate your Nginx logs daily and delete them after 30 days unless strictly necessary for security auditing. You can configure this easily in /etc/logrotate.d/nginx.

/var/log/nginx/*.log {
        daily
        missingok
        rotate 30
        compress
        delaycompress
        notifempty
        create 0640 www-data adm
        sharedscripts
        postrotate
                [ -f /var/run/nginx.pid ] && kill -USR1 `cat /var/run/nginx.pid`
        endscript
}

Conclusion

Scaling an API in 2014 isn't about magic; it's about removing bottlenecks. By tuning the Linux kernel to handle more file descriptors, configuring Nginx to maintain keepalive connections, and—most importantly—running on fast SSD storage, you can serve thousands of concurrent users on a single VPS.

Don't let legacy hardware kill your mobile app's momentum. Spin up a Pure SSD instance on CoolVDS today and see the difference latency makes.