API Gateway Survival Guide: Tuning Nginx & Kernel for High Concurrency

It is November 2016. Black Friday is weeks away. Your marketing team is buying ads, your frontend developers are pushing React updates, and you—the sysadmin—are staring at top hoping the load average doesn't spike past the CPU count. I’ve been there. Last year, I watched a poorly configured API gateway capsize under a traffic surge that wasn't even that large. The hardware was fine; the configuration was garbage.

The bottleneck is rarely the CPU speed itself; it's how your OS handles thousands of concurrent connections (C10K problem) and how your reverse proxy manages the handshake. If you are serving users in Oslo or Stockholm, latency is your enemy. A 20ms delay in the gateway becomes 200ms by the time the frontend renders.

In this post, we are tearing down the default settings. We aren't just "optimizing"; we are re-architecting the stack for raw throughput using Nginx 1.10 and the Linux 4.4 kernel.

1. The Foundation: Linux Kernel Tuning

Most VPS providers ship Linux distributions tuned for desktop usage or light web serving. For an API gateway handling thousands of requests per second, the defaults are suffocating. You need to open the floodgates on file descriptors and TCP limits.

First, check your current file descriptor limit. If it returns 1024, you are in trouble:

To fix this permanently for high-load environments, we need to edit /etc/sysctl.conf. We are going to adjust the backlog (how many packets sit in the queue before the kernel drops them) and the ephemeral port range. When Nginx acts as a proxy, it opens a connection to the client and a connection to the upstream backend. You can run out of ports fast.

Here is a battle-tested sysctl.conf configuration used on our high-performance CoolVDS instances:

# /etc/sysctl.conf

# Maximize the number of open file descriptors
fs.file-max = 2097152

# Increase the TCP backlog queue
net.core.somaxconn = 65535
net.core.netdev_max_backlog = 65535

# Widen the port range for outgoing connections (upstream proxies)
net.ipv4.ip_local_port_range = 1024 65535

# Allow reuse of sockets in TIME_WAIT state for new connections
# Critical for API gateways making many short requests to backends
net.ipv4.tcp_tw_reuse = 1

# Speed up TCP window scaling
net.ipv4.tcp_window_scaling = 1

# Protect against SYN flood attacks while maintaining performance
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 65535

Apply these changes immediately without rebooting:

2. Nginx Architecture: Beyond the Defaults

Using apt-get install nginx gives you a safety-first config. It is not built for speed. In 2016, with HTTP/2 gaining traction, we need to ensure our SSL termination is efficient and our worker processes are pinned correctly.

The Worker Process Dilemma

Context switching kills performance. If you have 4 CPU cores, you want exactly 4 worker processes. If you set this higher, the scheduler wastes time moving processes between cores. On a CoolVDS KVM instance, you have dedicated cores, so you can trust the hardware mapping.

Here is the optimized nginx.conf structure:

user www-data;
worker_processes auto;
worker_rlimit_nofile 65535;
pcre_jit on;

events {
    worker_connections 8192;
    use epoll;
    multi_accept on;
}

http {
    ##
    # Basic Settings
    ##
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    types_hash_max_size 2048;
    
    # Buffers - Crucial for handling POST payloads without disk I/O
    client_body_buffer_size 128k;
    client_max_body_size 10m;
    client_header_buffer_size 1k;
    large_client_header_buffers 4 4k;
    output_buffers 1 32k;
    postpone_output 1460;

    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    ##
    # SSL Settings (Modern cipher suite for 2016)
    ##
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_prefer_server_ciphers on;
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 10m;
    
    # ... logging and other includes
}

Notice worker_rlimit_nofile. This must match or exceed your system's ulimit, or Nginx will choke regardless of your kernel settings.

3. Upstream Keepalive: The Hidden Killer

This is where 90% of setups fail. By default, Nginx speaks HTTP/1.0 to upstream servers (your Node.js or PHP-FPM app) and closes the connection after every request. This forces a new TCP handshake for every single API call. In a microservices environment, this adds milliseconds of latency that compound rapidly.

You must enable keepalive in your upstream block to reuse connections.

upstream backend_api {
    server 127.0.0.1:3000;
    # Maintain 64 idle connections to the upstream
    keepalive 64;
}

server {
    listen 443 ssl http2;
    server_name api.coolvds-demo.no;

    # SSL Certificate Config...

    location / {
        proxy_pass http://backend_api;
        
        # REQUIRED for keepalive to work
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        
        # Forwarding headers
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
    }
}

Pro Tip: Never use IP hash load balancing if your upstream servers are stateless. It creates uneven distribution. Stick to Round Robin or Least Connections for better utilization of your CoolVDS vCPUs.

4. Storage I/O and The "Noisy Neighbor" Effect

You can tune software all day, but if your underlying storage is slow, your API will hang while logging requests or reading cache. This is common in cheap shared hosting where "SSD" often means a SATA SSD shared by 500 users. If one user decides to run a backup, your I/O wait shoots up.

Check your disk latency with ioping (install it if you haven't):

If you are seeing latency above 1ms on local reads, your host is overselling storage. This is why we deploy purely on NVMe storage at CoolVDS. NVMe communicates directly with the PCIe bus, bypassing the SATA controller bottleneck. For an API Gateway logging thousands of lines per second or caching JSON responses to disk, the difference between SATA SSD and NVMe is the difference between a 200 OK and a 504 Gateway Timeout.

5. Local Context: Latency and Legality

Latency is geography. If your primary user base is in Norway, hosting in Frankfurt or London adds a physical penalty of 15-30ms round trip. Hosting in Oslo reduces this to < 5ms.

The Data Sovereignty Elephant

With the invalidation of Safe Harbor last year and the new Privacy Shield framework still feeling shaky, data residency is a hot topic. Datatilsynet (The Norwegian Data Protection Authority) is becoming increasingly strict about where user data lives. Running your API gateway on a CoolVDS instance in our Oslo data center ensures that traffic termination happens within Norwegian borders, simplifying your compliance posture significantly compared to using a US-based cloud load balancer.

Summary

Performance isn't magic; it's math. By raising the kernel's connection limits, enabling upstream keepalives in Nginx, and ensuring your hardware I/O can keep pace with your network throughput, you build a system that handles spikes gracefully.

Don't wait for the Black Friday crash logs. Deploy a test instance, run these configs, and benchmark it yourself using ab or wrk.

Ready to lower your latency? Deploy a high-performance NVMe KVM instance on CoolVDS today and keep your data in Norway.