API Gateway Performance Tuning: Shaving Milliseconds in a High-Latency World

If I see one more default NGINX configuration in front of a high-traffic microservices cluster, I might just flip a table. We talk endless game about microservices and decoupling, but we often ignore the hefty tax we pay for it: network latency. Every hop is a cost. If your API Gateway is sluggish, your entire architecture feels broken, no matter how fast your Go backend is.

In 2022, "it works" isn't good enough. Users expect instant interactions. When you are serving the Nordic market, routing traffic through Frankfurt or Amsterdam is lazy engineering. You are adding 20-30ms of Round Trip Time (RTT) before the request even touches your application logic.

I’m going to show you how to tune your gateway for raw throughput. We aren't talking about basic caching headers here. We are talking about kernel-level tuning, connection pooling, and the hardware reality that most cloud providers hide from you.

The Hidden Bottleneck: Ephemeral Ports and File Descriptors

The most common failure mode I see during load testing isn't CPU exhaustion; it's running out of file descriptors or ephemeral ports. Linux handles every connection as a file. Default limits are usually set to 1024. For a high-performance gateway, that is a joke.

You need to tell the kernel to allow a massive number of open files and to recycle TIME_WAIT sockets faster. If you don't, your gateway will choke under load, dropping packets while your CPU sits idle at 10%.

Step 1: Kernel Tuning (sysctl.conf)

Open /etc/sysctl.conf. These settings optimize the TCP stack for high concurrency. This is standard practice for any serious edge node.

# Increase system-wide file descriptor limit
fs.file-max = 2097152

# Increase the read/write buffers for TCP connections
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

# Increase the backlog of incoming connections
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535

# Allow reusing sockets in TIME_WAIT state for new connections
net.ipv4.tcp_tw_reuse = 1

# Increase local port range to avoid exhaustion
net.ipv4.ip_local_port_range = 1024 65535

Apply these changes with sysctl -p. If you are running this inside a Docker container, remember that some of these are namespaced, but many (like somaxconn) depend on the host configurations. This is where the "noisy neighbor" problem bites hard on shared hosting.

Pro Tip: On cheap VPS providers, you often cannot modify certain kernel parameters because you are in a restricted container (OpenVZ/LXC). At CoolVDS, we use KVM virtualization. You get a full, dedicated kernel. You want to tune `tcp_tw_reuse`? Go ahead. It's your kernel.

The "Keepalive" Mistake Everyone Makes

I recently audited a setup for a client in Oslo. They were using NGINX as a reverse proxy for a Node.js API. Their complaint: "Latency is double what we expected."

The culprit? They weren't using upstream keepalives. By default, NGINX opens a new connection to the backend service for every single request, does the handshake, sends data, and closes it. That is expensive. You need to keep those connections open.

Step 2: NGINX Upstream Configuration

Here is the correct way to configure an upstream block to reuse connections. Pay attention to the keepalive directive and the proxy version.

upstream backend_api {
    server 10.0.0.5:8080;
    server 10.0.0.6:8080;

    # Keep 64 idle connections open to the upstream
    keepalive 64;
}

server {
    listen 443 ssl http2;
    server_name api.coolvds-example.no;

    # SSL optimizations for 2022 standards
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256;
    ssl_prefer_server_ciphers on;

    location / {
        proxy_pass http://backend_api;

        # REQUIRED for keepalive to work
        proxy_http_version 1.1;
        proxy_set_header Connection "";

        # Pass real IP (critical for logging/security)
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

Without proxy_set_header Connection "";, NGINX forwards the client's "close" header to the backend, killing the connection you tried to save.

The Hardware Factor: Why NVMe and Location Matter

You can optimize software until 4 AM, but physics always wins. In 2022, the difference between a SATA SSD and an NVMe drive is not just bandwidth; it's IOPS and queue depth.

When your API Gateway logs requests, writes to access logs, or buffers large payloads to disk (client body buffering), slow storage creates I/O Wait. This blocks the CPU. Your top command shows high wa (wait) time, and your API requests hang.

Benchmark: SATA vs NVMe Latency

For Norwegian businesses, data sovereignty is also no joke. With the Schrems II ruling still shaking up the industry, hosting your API gateway on US-controlled infrastructure (even if the datacenter is in Europe) carries legal risk. Hosting directly in Norway, on Norwegian infrastructure, simplifies your GDPR compliance stance significantly.

Load Testing: Prove It

Don't guess. Measure. Use wrk to hammer your gateway and see how it behaves under stress. If you are on Linux, you can install it easily. This tool generates significant load with minimal thread overhead.

# Run a benchmark: 12 threads, 400 connections, for 30 seconds
wrk -t12 -c400 -d30s https://api.yourdomain.no/v1/status

Watch the Latency Distribution in the output. A low average latency is meaningless if your 99th percentile (tail latency) is 2 seconds. That means 1% of your users are hating you. Tuning the sysctl parameters we discussed above is specifically aimed at smoothing out that tail latency.

Worker Processes and NGINX limits

One final check. Ensure your NGINX worker limits match your kernel limits. In your nginx.conf main context:

worker_processes auto;

# This must be higher than worker_connections
worker_rlimit_nofile 65535;

events {
    # Max connections per worker
    worker_connections 16384;
    use epoll;
    multi_accept on;
}

The multi_accept on; directive allows a worker to accept all new connections at once, rather than one at a time. In a high-throughput environment, this is essential.

The Logical Infrastructure Choice

Performance tuning is a full-stack discipline. You need the kernel, the application config, and the hardware aligned. While container orchestration tools like Kubernetes (now at v1.23) are powerful, they add complexity and network overhead (CNI plugins, iptables rules).

Sometimes, the most performant solution is the simplest: A dedicated, tuned Linux instance sitting right on the backbone.

At CoolVDS, we provide the raw materials for performance obsessives. Pure KVM isolation means no stolen CPU cycles. Local NVMe storage eliminates I/O bottlenecks. And our location in the heart of Norway ensures your packets don't travel halfway around the world to reach a user in Trondheim.

Stop fighting against noisy neighbors and network lag.

Deploy a high-performance NVMe instance on CoolVDS today and get your P99 latency down to where it belongs.