Latency is the Only Metric That Matters

If your API gateway adds more than 5ms to a request, you are doing it wrong. In the Nordic infrastructure landscape, where fiber penetration is high and the Norwegian Internet Exchange (NIX) in Oslo offers incredible peering, there is no excuse for sluggish handshakes. Yet, I see it constantly: bloated Java wrappers running on over-provisioned public clouds, choking on simple JSON payloads.

It usually starts with a support ticket: "Our microservices are fast, but the client sees 200ms lag." The culprit is almost always the gateway layer—misconfigured and starved for I/O.

Today, we aren't discussing architectural theory. We are fixing your sysctl.conf, stripping down your Nginx config, and addressing the hardware reality that software cannot fix. This is how you tune for 2022 standards.

1. The Hardware bottleneck: Why "Cloud" Often Fails

Before we touch a single config file, we must address the platform. You cannot tune away CPU Steal. In massive public clouds, your "vCPU" is a noisy timeshare. When a neighbor spins up a crypto miner or a heavy batch job, your API gateway's SSL termination stalls. This creates the dreaded "spiky" p99 latency.

Pro Tip: Run iostat -x 1 and watch the %steal column. If it's consistently above 0.5%, move workloads. This is why we engineered CoolVDS with KVM isolation and strict resource guarantees. When you buy a core here, it's yours.

2. Kernel Tuning: Escaping the Defaults

Most Linux distributions (Ubuntu 20.04 LTS, Debian 11) ship with conservative defaults designed for desktop stability, not high-throughput packet switching. For an API gateway handling 10k+ RPS, these defaults are fatal.

Open /etc/sysctl.conf. We need to widen the TCP ephemeral port range and enable BBR congestion control to handle packet loss gracefully.

# /etc/sysctl.conf

# Max open files (requires ulimit changes too)
fs.file-max = 2097152

# Widen the port range for outgoing connections to upstream services
net.ipv4.ip_local_port_range = 10000 65000

# Reuse sockets in TIME_WAIT state for new connections
net.ipv4.tcp_tw_reuse = 1

# Increase backlog for incoming connections
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 8192

# Enable BBR (Bottleneck Bandwidth and RTT)
# Note: Requires kernel 4.9+. Check with 'uname -r'
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr

Apply these with sysctl -p. BBR is particularly critical for users connecting from mobile networks across Norway's rugged terrain, where signal handovers can cause packet jitter.

3. Nginx / OpenResty: The Gateway Engine

Whether you use Kong, basic Nginx, or Ingress Nginx on Kubernetes, the underlying mechanics are identical. The most common mistake I see in 2022? Lack of Keepalives to Upstreams.

By default, Nginx opens a new connection to your backend service (Node.js, Go, Python) for every single request. This burns file descriptors and CPU cycles performing TCP handshakes.

The Fix: Upstream Keepalive

upstream backend_microservice {
    server 10.0.0.5:8080;
    
    # The magic number. Keeps 64 idle connections open per worker.
    keepalive 64;
}

server {
    location /api/ {
        proxy_pass http://backend_microservice;
        
        # Required for keepalive to work
        proxy_http_version 1.1;
        proxy_set_header Connection "";
    }
}

This single change can drop internal latency from 20ms to under 2ms.

Worker Configuration

Don't rely on auto-detection if you are pinning processes. Explicitly define your worker limits to match your hardware capabilities.

worker_processes auto;

# Each worker needs enough file descriptors
worker_rlimit_nofile 65535;

events {
    # epoll is mandatory for Linux performance
    use epoll;
    
    # Allow worker to accept multiple new connections at once
    multi_accept on;
    
    worker_connections 16384;
}

4. The Impact of Storage I/O on Logging

API Gateways generate massive logs. Every access log write is a disk I/O operation. If your hosting provider uses spinning rust (HDD) or network-attached storage (NAS) with low IOPS, your gateway will block while waiting for the disk.

At CoolVDS, we use local NVMe storage. The difference is stark. When writing 5,000 log lines per second, standard SSDs choke. NVMe doesn't blink.

Optimization Strategy: Buffer Your Logs

If you cannot use NVMe (why aren't you?), buffer the logs in memory to reduce write frequency.

# Write to buffer (64k), flush every 5 minutes
access_log /var/log/nginx/access.log combined buffer=64k flush=5m;

Warning: You risk losing 5 minutes of logs if the server crashes. On a stable KVM platform like ours, this risk is minimal compared to the performance gain.

5. Compliance & Security: The "Schrems II" Reality

In Norway, performance isn't just speed; it's legal compliance. Since the 2020 Schrems II ruling, sending personal data to US-controlled clouds is legally risky. Datatilsynet (The Norwegian Data Protection Authority) is watching.

Hosting your API Gateway on CoolVDS guarantees data residency. Your traffic enters in Oslo and stays in Europe. We don't route your SSL termination through Virginia.

6. Benchmarking the Result

Don't take my word for it. Use wrk to stress test your setup. Do not use Apache Bench (ab), it is single-threaded and obsolete for 2022 standards.

# Install wrk (requires build-essential)
git clone https://github.com/wg/wrk.git
cd wrk && make

# Run test: 12 threads, 400 connections, 30 seconds
./wrk -t12 -c400 -d30s http://your-coolvds-ip/api/health

Final Thoughts

Performance tuning is a game of inches. You fight for every millisecond in the kernel, in the Nginx config, and in the network stack. But software tuning hits a hard wall if the underlying infrastructure is slow.

If you are building for the Norwegian market, you need low latency to Oslo, strict GDPR compliance, and hardware that doesn't steal your CPU cycles.

Stop fighting your provider's noisy neighbors. Deploy a dedicated KVM instance on CoolVDS today and see what BBR + NVMe can actually do.