API Gateway Tuning: Surviving The Thundering Herd on Norwegian Infrastructure

Let’s be honest: default configurations are for hobbyists. If you are running a high-traffic API gateway—whether it's Kong, NGINX, or HAProxy—and you haven't touched your sysctl.conf since the OS install, you are sitting on a ticking time bomb. I’ve seen it happen too many times. A marketing campaign goes live, traffic spikes by 400%, and suddenly your fancy microservices architecture is just a collection of 502 Bad Gateway errors.

It’s not usually the code. It’s the plumbing.

In this guide, I’m going to walk you through the exact kernel and application-level tuning we use to push thousands of requests per second (RPS) without breaking a sweat. We will focus on the Linux networking stack and NGINX, as that remains the de-facto standard for API gateways in 2022.

The "War Story": When 10k Connections vanish

Last year, I audited a setup for a logistics company dealing with heavy cross-border traffic between Oslo and Stockholm. They were running a Kubernetes cluster on a generic hyperscaler. The latency was erratic. Sometimes 20ms, sometimes 400ms. The application logic was sound, but the gateway was choking.

The culprit? Ephemeral port exhaustion and excessive context switching caused by "noisy neighbors" on shared vCPU instances. The hypervisor was stealing CPU cycles just when the handshake needed them most.

Pro Tip: Never trust "burstable" instances for an API Gateway. You need dedicated CPU time. This is why we default to KVM virtualization on CoolVDS—you get the cycles you pay for, without fighting other tenants for resources.

Step 1: The Linux Kernel is Your Bottleneck

Linux is tuned for general-purpose computing out of the box, not for handling 50,000 simultaneous TCP connections. You need to tell the kernel that it’s okay to open more files and reuse sockets faster.

First, check your current limits. If this number is 1024, you’re in trouble:

You need to bump this up. But the real magic happens in /etc/sysctl.conf. Here is a production-ready configuration I use for high-throughput gateways. This optimizes the TCP stack to handle a flood of short-lived connections (typical for REST APIs).

Production sysctl.conf Configuration

# /etc/sysctl.conf

# Maximize the number of open file descriptors
fs.file-max = 2097152

# Increase the size of the receive queue
net.core.netdev_max_backlog = 16384
net.core.somaxconn = 32768

# Reuse sockets in TIME_WAIT state for new connections
# (Crucial for API gateways connecting to backend services)
net.ipv4.tcp_tw_reuse = 1

# Increase TCP buffer sizes for modern high-speed networks
net.ipv4.tcp_rmem = 4096 87380 67108864
net.ipv4.tcp_wmem = 4096 65536 67108864

# Protect against SYN flood attacks while handling legitimate bursts
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.tcp_syncookies = 1

# Port range expansion
net.ipv4.ip_local_port_range = 1024 65535

Apply these changes with sysctl -p. This ensures that when a burst of traffic hits your VPS Norway instance, the kernel doesn't panic and drop packets.

Step 2: NGINX Optimization

Now that the OS can breathe, let's look at the gateway software. NGINX defaults are conservative. The most common mistake I see is not enabling keepalives to upstream servers. Without this, NGINX opens and closes a new TCP connection to your backend service for every single request. This adds unnecessary latency and burns through CPU.

Here is how you configure an optimized upstream block:

upstream backend_api {
    server 10.0.0.5:8080;
    server 10.0.0.6:8080;

    # Keep at least 64 idle connections open to the backend
    keepalive 64;
}

server {
    location /api/ {
        proxy_pass http://backend_api;
        
        # Required for keepalive to work
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        
        # Buffer tuning
        proxy_buffers 16 16k;
        proxy_buffer_size 32k;
    }
}

We also need to look at the global worker settings. The rule of thumb in 2022 is one worker per CPU core, but the worker_connections limit needs to be high.

With multi_accept on, a worker will accept all new connections at once, rather than one by one. This is aggressive but necessary for high-performance gateways.

Step 3: The Hardware Reality (Latency & NVMe)

You can tune software all day, but you cannot tune the speed of light. If your customers are in Oslo, Bergen, or Trondheim, and your server is in a massive datacenter in Frankfurt or Amsterdam, you are eating a 20-30ms latency penalty on every round trip. For an API that makes multiple sequential calls, this adds up to perceptible lag.

Furthermore, disk I/O matters. API gateways log heavily. Access logs, error logs, audit trails. If you are on standard SATA SSDs (or worse, spinning rust), your disk write queue will block your request processing.

Benchmark: Standard VPS vs. CoolVDS NVMe

I ran a quick wrk benchmark comparing a standard cloud instance against a CoolVDS instance with local NVMe storage. Both had 4 vCPUs and 8GB RAM.

The difference isn't just speed; it's consistency. The CoolVDS instance uses high-frequency cores and direct-attached NVMe, meaning logging doesn't block the network stack.

Data Sovereignty and Compliance

Since the Schrems II ruling, transferring personal data outside the EEA has become a legal minefield. Using US-owned cloud providers adds a layer of complexity regarding the CLOUD Act. Hosting your API gateway on Norwegian soil isn't just a performance decision; it's a risk management strategy.

By keeping the data processing within Norway (as we do at CoolVDS), you simplify your GDPR compliance posture significantly. You don't have to worry about data transiting through third-party countries before it even hits your database.

Conclusion

Performance is a stack. It starts with the physics of location, moves to the hardware (NVMe/CPU), goes through the kernel (sysctl), and ends at the application config (NGINX). You can't ignore any layer.

If you are tired of unexplained latency spikes or worrying about data sovereignty, it might be time to move your gateway closer to your users.

Don't let slow I/O kill your API performance. Deploy a test instance on CoolVDS in 55 seconds and see the difference raw power makes.