Crushing API Latency: Advanced Gateway Tuning for High-Throughput Microservices

If your API response time starts with a '1'—and we are talking seconds—you have already failed. In the high-frequency trading floors of Oslo or the rapid-fire e-commerce environments of Northern Europe, milliseconds translate directly into Krone. I recently audited a payment gateway for a client based in Trondheim. Their code was clean Go, compiled and efficient. Yet, under load tests mimicking Black Friday traffic, their 99th percentile latency spiked to 4 seconds. The culprit wasn't their application logic; it was a default Linux network stack and an untuned reverse proxy choking on TCP handshakes.

Most developers treat the API Gateway—usually Nginx, HAProxy, or Kong—as a black box. You `apt-get install`, slap on a config, and hope for the best. That works for a blog. It does not work for a distributed system handling 10,000 requests per second. To get true performance, we need to peel back the layers of abstraction and tune the engine while it's running.

Step 1: The Kernel is Your Limit

Before touching the web server, look at the OS. By default, most Linux distributions (Ubuntu 18.04, CentOS 7) are configured for general-purpose desktop or light server usage. They are conservative. They protect resources rather than spend them. When acting as an API Gateway, your server needs to be a firehose.

The first wall you hit is the file descriptor limit. In Linux, everything is a file, including a TCP connection. The default limit is often 1,024. If you have 2,000 concurrent users, half of them are getting dropped.

Check your current limits:

ulimit -n

To fix this permanently, edit /etc/security/limits.conf:

* soft nofile 65535
* hard nofile 65535
root soft nofile 65535
root hard nofile 65535

TCP Stack Tuning

Next, we need to address how the kernel handles TCP connections. A common issue in high-load gateways is running out of ephemeral ports because connections sit in the TIME_WAIT state for too long. We need to tell the kernel to recycle these faster.

Add the following to your /etc/sysctl.conf. These settings are aggressive but necessary for a dedicated API gateway:

# Allow reuse of sockets in TIME_WAIT state for new connections
net.ipv4.tcp_tw_reuse = 1

# Increase the range of local ports available
net.ipv4.ip_local_port_range = 1024 65535

# Max number of packets in the receive queue
net.core.netdev_max_backlog = 5000

# Increase the maximum number of open files
fs.file-max = 2097152

# Increase TCP max syn backlog
net.ipv4.tcp_max_syn_backlog = 4096

# Disable slow start on idle connections
net.ipv4.tcp_slow_start_after_idle = 0

Run sysctl -p to apply. This specific configuration minimizes the time a socket spends doing nothing, which is critical when you are paying for throughput.

Step 2: Nginx Configuration for Raw Speed

Whether you are using raw Nginx or OpenResty, the default nginx.conf is not your friend. We need to optimize the worker processes and connection handling. The goal is to keep connections alive just long enough to be useful, but not so long they hog RAM.

Pro Tip: Do not just set worker_processes to "auto". On a virtualized environment, ensure you are pinning this to the number of vCPUs you actually have access to. Context switching is expensive.

Here is a reference configuration block for the events and http context geared towards API traffic:

worker_processes auto;
worker_rlimit_nofile 65535;

events {
    # Essential for Linux performance
    use epoll;
    
    # Allow a worker to accept all new connections at once
    multi_accept on;
    
    # Match this to your ulimit
    worker_connections 65535;
}

http {
    # ... logs and mime types ...

    # Optimization for file serving (less relevant for API, but good hygiene)
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;

    # Keepalive settings - CRITICAL for API performance
    # Allow the client to keep the connection open for 15s
    keepalive_timeout 15;
    
    # Allow up to 1000 requests per keepalive connection
    keepalive_requests 1000;

    # Buffer sizes - tune these based on your payload size
    client_body_buffer_size 128k;
    client_max_body_size 10m;
    client_header_buffer_size 1k;
    large_client_header_buffers 4 4k;
    
    # Upstream Keepalive
    upstream backend_api {
        server 10.0.0.5:8080;
        
        # Keep idle connections open to the backend application
        keepalive 64;
    }
}

The Importance of Upstream Keepalive

The keepalive 64; directive inside the upstream block is often missed. Without it, Nginx opens a new connection to your backend service (Node.js, Go, Python) for every single request. That involves a TCP handshake every time. By enabling keepalives, Nginx reuses existing connections, drastically dropping latency.

Step 3: TLS 1.3 and Encryption Overhead

It is late 2019. If you are not supporting TLS 1.3, you are living in the past. TLS 1.3 reduces the handshake latency by requiring one less round-trip compared to TLS 1.2. On mobile networks or connections traversing the length of Norway, this reduction is noticeable.

ssl_protocols TLSv1.2 TLSv1.3;
ssl_prefer_server_ciphers on;
ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384';

# Enable OCSP Stapling to speed up verification
ssl_stapling on;
ssl_stapling_verify on;

The Hardware Factor: Why "Cloud" Isn't Always Enough

You can tune sysctl until you are blue in the face, but if the underlying storage subsystem is garbage, your database calls will block your API responses. In the virtualization world, the "Noisy Neighbor" effect is real. If the VM next to you is crunching big data, your I/O wait times skyrocket.

This is where infrastructure choice becomes an architectural decision, not just a procurement one. We see many clients migrating from generic hyperscalers to CoolVDS for one specific reason: Performance consistency.

CoolVDS uses KVM (Kernel-based Virtual Machine) technology. Unlike container-based VPS solutions where you share the host's kernel, KVM provides true isolation. This means your sysctl tuning actually works the way you expect it to. Furthermore, the shift to NVMe storage means that disk I/O—often the bottleneck for database-heavy APIs—is virtually non-existent.

For Norwegian businesses, there is also the compliance angle. With the increasing scrutiny from Datatilsynet regarding data processing, hosting your API gateway and database on servers physically located in Norway simplifies your GDPR compliance posture significantly compared to routing traffic through US-owned data centers.

Conclusion

Performance isn't magic. It's the sum of a thousand small optimizations. By removing the shackles from the Linux kernel, configuring Nginx to reuse connections, and deploying on hardware that respects your need for I/O throughput, you can turn a sluggish API into a real-time asset.

Don't let high latency kill your user experience. Deploy a test instance on CoolVDS today, apply these configurations, and watch your 99th percentile response times drop through the floor.