Scaling Nginx: The Art of the Millisecond API Gateway in a Post-Apache World

Let’s be honest: if your API takes 200ms to respond, your mobile app feels broken. In the fast-moving Norwegian tech scene—where 4G LTE adoption is outpacing the rest of Europe—users have zero patience for latency. I recently audited a payment gateway running on a legacy hosting provider in Oslo. Their stack? Apache Prefork on standard spinning rust (HDD). The result? A catastrophic 1.2-second wait time per handshake. It was embarrassing.

The era of the monolithic request handler is dying. We are shifting toward micro-components and SOA (Service Oriented Architecture), and that requires a new breed of gatekeeper. You don't need a heavier server; you need a smarter one. Today, we are tearing down the default configurations and tuning Nginx to handle 10,000 concurrent connections (C10k) without breaking a sweat. And we are doing it on hardware that actually respects I/O physics.

The Hardware Reality: Why Virtualization Type Matters

Before we touch a single config file, we must address the infrastructure. You cannot tune your way out of bad hardware. In 2014, many VPS providers are still overselling OpenVZ containers. The problem? CPU Steal Time. In a containerized environment like OpenVZ, you are sharing the kernel with noisy neighbors. If user 402 decides to compile a kernel, your API gateway stutters.

This is why at CoolVDS, we exclusively deploy on KVM (Kernel-based Virtual Machine). KVM provides hardware virtualization. Your RAM is yours. Your CPU cycles are reserved. More importantly, we map storage directly to Enterprise SSD arrays. When your API logs access tokens or reads session data, you need the random I/O performance of solid-state drives. Spinning disks can simply no longer keep up with the concurrency requirements of modern JSON APIs.

Step 1: The OS Layer (CentOS 7 / Ubuntu 14.04)

Linux, out of the box, is tuned for a desktop experience, not a high-throughput gateway. We need to tell the kernel that it's okay to have thousands of open files.

Tuning sysctl.conf

Open /etc/sysctl.conf. We need to widen the TCP ephemeral port range and allow for faster recycling of sockets to prevent your gateway from running out of connections during traffic spikes.

# /etc/sysctl.conf

# Increase system-wide file descriptor limit
fs.file-max = 2097152

# Widen the port range for outgoing connections
net.ipv4.ip_local_port_range = 1024 65535

# Enable TCP Reuse to recycle sockets in TIME_WAIT state
net.ipv4.tcp_tw_reuse = 1

# Protect against SYN flood attacks (common in 2014)
net.ipv4.tcp_syncookies = 1

# Increase backlog for incoming connections
net.core.somaxconn = 32768
net.ipv4.tcp_max_syn_backlog = 8192

Apply these changes with sysctl -p. Without this, Nginx will hit a wall regardless of its internal settings.

Step 2: Nginx as the API Gateway

Nginx 1.6+ is the gold standard here. Unlike Apache, which spawns a thread/process for every connection (eating RAM like Chrome eats battery), Nginx uses an asynchronous, event-driven architecture. It can handle thousands of keep-alive connections with a few megabytes of RAM.

Here is a battle-tested configuration for an API gateway handling JSON traffic. This setup strips unnecessary headers, enables GZIP for text payloads, and buffers efficiently.

# /etc/nginx/nginx.conf

user nginx;
worker_processes auto; # Detects CPU cores automatically
worker_rlimit_nofile 100000; # Must exceed worker_connections

events {
    worker_connections 4096;
    use epoll; # Essential for Linux 2.6+
    multi_accept on;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    # Optimization for file sending (less context switching)
    sendfile        on;
    tcp_nopush      on;
    tcp_nodelay     on;

    # Keepalive is crucial for API clients making multiple requests
    keepalive_timeout  65;
    keepalive_requests 100;

    # Gzip reduces JSON payload size over the wire
    gzip on;
    gzip_types application/json application/javascript text/css;
    gzip_min_length 1000;

    # Security: Hide Nginx version
    server_tokens off;

    upstream backend_api {
        # Use keepalive to upstream to reduce handshake overhead
        server 127.0.0.1:8080;
        keepalive 32;
    }

    server {
        listen 80;
        server_name api.yourdomain.no;

        location / {
            proxy_pass http://backend_api;
            
            # Proxy Headers
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            
            # Timeouts (Fail fast concept)
            proxy_connect_timeout 5s;
            proxy_send_timeout 10s;
            proxy_read_timeout 10s;
        }
    }
}

Pro Tip: After the Heartbleed and POODLE vulnerabilities this year, ensure you disable SSLv3 in your SSL blocks immediately. Use ssl_protocols TLSv1 TLSv1.1 TLSv1.2; only. Security is not optional.

The Norwegian Context: Latency & Law

Why host this in Norway? Aside from the obvious latency benefits—pinging NIX (Norwegian Internet Exchange) in Oslo from a local CoolVDS instance takes roughly 1-3ms—there is the legal aspect. With the EU Data Protection Reform (the upcoming GDPR framework) currently in draft stages, and the Datatilsynet (Norwegian Data Protection Authority) enforcing strict adherence to the Personal Data Act, data residency is becoming a board-level discussion.

Hosting your API gateway on US-based clouds introduces legal ambiguity regarding Safe Harbor, especially after the recent Snowden revelations. Keeping your data on Norwegian soil, protected by Norwegian law, is the safest bet for any enterprise dealing with sensitive customer data.

Benchmarking the Difference

Don't take my word for it. Install wrk (a modern HTTP benchmarking tool superior to `ab`) and test your endpoint.

# Run a 30-second test with 12 threads and 400 connections
wrk -t12 -c400 -d30s http://api.yourdomain.no/v1/status

On a standard HDD VPS, you might see 200 requests/sec. On a tuned CoolVDS SSD KVM instance with the config above, we regularly benchmark over 5,000 requests/sec on entry-level plans. That is the difference between an app that flies and one that times out.

Conclusion

Performance isn't magic; it's engineering. It's the sum of a tuned kernel, an efficient event loop, and hardware that doesn't choke on I/O. As we move into 2015, the tolerance for slow APIs will hit zero.

Stop fighting your infrastructure. If you need a stack that is pre-optimized for high-performance workloads with local peering to NIX, it's time to upgrade.

Ready to drop your latency? Deploy a high-performance SSD KVM instance on CoolVDS today and test the difference yourself.