API Gateway Tuning: Squeezing Microseconds Out of Nginx & Kong in High-Load Environments

If you are looking at your P99 latency charts and seeing spikes that don't make sense, stop blaming the application code. In a distributed architecture, the API Gateway is often the silent killer of performance. I recently audited a fintech setup in Oslo where the development team spent weeks optimizing Go routines, only to find their requests were queuing at the ingress layer because of default Linux kernel settings.

Performance isn't just about code; it's about the entire stack, from the silicon to the socket. In Norway, where latency to the Norwegian Internet Exchange (NIX) is measured in single-digit milliseconds, adding overhead at your gateway is inexcusable. Here is how we tune the stack for raw throughput, using standard tools available in 2023.

1. The OS Layer: Tuning the TCP Stack

Most Linux distributions ship with generic settings designed for desktop usage or light web serving, not high-concurrency API routing. Before you touch Nginx or Kong, you must fix the foundation. If your somaxconn is too low, incoming connections get dropped silently under load.

Edit your /etc/sysctl.conf. These settings optimize the handling of thousands of ephemeral ports and open files, which is critical when your gateway is proxying requests to backend microservices.

Key Kernel Directives

# /etc/sysctl.conf

# Increase system-wide file descriptors
fs.file-max = 2097152

# Increase the size of the receive queue.
net.core.somaxconn = 65535
net.core.netdev_max_backlog = 65535

# Reuse connections in TIME_WAIT state for new connections
net.ipv4.tcp_tw_reuse = 1

# Increase available ephemeral ports
net.ipv4.ip_local_port_range = 1024 65535

# Optimize TCP window size for high-bandwidth low-latency links (typical in Nordic datacenters)
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_rmem = 4096 87380 33554432
net.ipv4.tcp_wmem = 4096 65536 33554432

Apply these with sysctl -p. If you are running on shared hosting, you probably can't touch these. This is why we insist on KVM virtualization at CoolVDS; you need full kernel control to tune the network stack properly.

2. Nginx / Kong Configuration: The "Keepalive" Trap

Whether you use raw Nginx or Kong (which is built on OpenResty/Nginx), the biggest mistake I see is the lack of upstream keepalives. By default, Nginx acts as a polite HTTP/1.0 client to your upstream services: it opens a connection, sends the request, receives the response, and closes the connection.

Doing a full TCP handshake + TLS handshake for every internal microservice call destroys performance. You need to reuse connections.

Optimized Upstream Block

upstream backend_service {
    server 10.0.0.5:8080;
    
    # The Critical Setting: keep connection open
    keepalive 64;
}

server {
    location /api/v1/ {
        proxy_pass http://backend_service;
        
        # Required for keepalive to work
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        
        # Buffer tuning
        proxy_buffers 16 16k;
        proxy_buffer_size 32k;
    }
}

Setting proxy_http_version 1.1 and clearing the Connection header allows the underlying TCP connection to persist between the gateway and your backend API. In our benchmarks, this single change reduced internal latency by 40% on high-traffic nodes.

3. SSL/TLS Offloading: Hardware Matters

Decryption is expensive. In 2023, relying on software-only termination without AES-NI instruction set support is a bottleneck. You want to prioritize cipher suites that are hardware-accelerated. Additionally, enabling OCSP Stapling prevents the client from having to contact the Certificate Authority, shaving off critical milliseconds.

ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256;
ssl_prefer_server_ciphers on;

# OCSP Stapling
ssl_stapling on;
ssl_stapling_verify on;
resolver 1.1.1.1 8.8.8.8 valid=300s;
resolver_timeout 5s;

4. The Hardware Reality Check

You can optimize configs until you are blue in the face, but software cannot overcome physics. An API Gateway is I/O intensive—it logs requests, reads configurations, and swaps memory under load. If your "Cloud VPS" is running on standard SATA SSDs (or worse, spinning rust) with "noisy neighbor" stealing CPU cycles, your P99 latency will fluctuate wildly.

Pro Tip: Always check your CPU Steal time. Run top and look at the %st value. If it's above 0.5% consistently, your provider is overselling their cores. Move your workload immediately.

This is where infrastructure choice becomes a strategic decision. For gateways handling traffic in Northern Europe, we deploy CoolVDS instances on pure NVMe storage with high-frequency CPUs. The I/O throughput of NVMe is essential when you have access logs writing thousands of lines per second.

Comparison: Gateway Latency on Storage Types

5. Compliance and Data Sovereignty

Technical tuning doesn't exist in a vacuum. Since the Schrems II ruling, transferring personal data outside the EEA has become a legal minefield. When you host your API Gateway, you are defining the entry point for your user data.

Hosting on a US-controlled cloud provider (even in a generic "Europe" region) can introduce complex legal challenges regarding the US CLOUD Act. Utilizing a Norwegian-based provider like CoolVDS ensures that your data remains under strict Norwegian and EEA jurisdiction, satisfying Datatilsynet requirements. It also guarantees that traffic from Oslo users doesn't hairpin through Frankfurt or Amsterdam before returning home.

6. Verification: Stress Testing Your Tuning

Don't assume your changes worked. Verify them. We use wrk for generating significant HTTP load to test the gateway's limits.

# Install wrk (Ubuntu/Debian)
sudo apt-get update && sudo apt-get install wrk

# Run a test: 12 threads, 400 connections, for 30 seconds
wrk -t12 -c400 -d30s https://your-api-gateway.com/api/v1/ping

Look for the Socket errors count. If it is non-zero, go back to step 1 and increase your file descriptors. If your latency high is acceptable but the standard deviation is huge, check your hypervisor quality.

Low latency isn't magic; it's a combination of a tuned Linux kernel, correct Nginx configuration, and uncompromising hardware. Don't let slow I/O kill your SEO or user experience. Deploy a test instance on CoolVDS in 55 seconds and see what a proper NVMe stack does for your API response times.