The 10ms Mandate: Optimizing Your API Gateway Logic

If your API Gateway adds more than 10 milliseconds of overhead to a request, you are doing it wrong. In the microservices era, where a single user action might spawn twenty internal HTTP calls, latency compounds terrifyingly fast. A 50ms delay at the gateway becomes a full second of lag for the end-user. That is unacceptable.

Most developers spin up a standard Nginx instance or a default Kong deployment and wonder why their throughput plateaus. The answer usually lies deep in the Linux kernel or the virtualization layer. I recently audited a payment processing cluster here in Oslo that was timing out during peak loads. The code was fine. The infrastructure configuration was stuck in 2015.

Here is how we fixed it, using tools and techniques available right now in early 2019.

1. The Hardware Reality Check: Steal Time and I/O

Before touching a single config file, look at your infrastructure. In a virtualized environment, your neighbors are your enemies. If you are on a budget VPS using OpenVZ or heavily oversold Xen, you are suffering from "CPU Steal Time." This happens when the hypervisor queues your processes because another tenant is hogging the physical CPU.

For an API Gateway—which handles thousands of ephemeral connections—Disk I/O is the other silent killer. Access logs, error logs, and local caching require rapid writes. Spinning rust (HDD) or shared SATA SSDs simply cannot keep up with the IOPS required for high-concurrency API traffic.

Pro Tip: Check your steal time immediately. Run top and look for the st value. If it's above 0.5 consistently, migrate immediately. We built CoolVDS on KVM (Kernel-based Virtual Machine) with pure NVMe storage specifically to eliminate this noisy neighbor effect. Dedicated resources aren't a luxury; they are a requirement for consistent latency.

2. Kernel Tuning: The sysctl.conf Layer

Linux defaults are designed for general-purpose computing, not high-performance packet switching. We need to tell the kernel to handle a massive number of open files and recycle TCP connections aggressively.

Open /etc/sysctl.conf. We are going to enable TCP BBR (Bottleneck Bandwidth and RTT), a congestion control algorithm from Google that significantly improves throughput on networks with some packet loss.

First, verify you are running a kernel version 4.9 or higher (Ubuntu 18.04 LTS ships with 4.15, so you should be good):

Now, apply these settings to optimize the network stack:

# /etc/sysctl.conf

# Maximize the backlog of incoming connections
net.core.somaxconn = 65535
net.core.netdev_max_backlog = 65535

# Increase file descriptors for high concurrency
fs.file-max = 2097152

# TCP BBR Congestion Control (The 2019 Standard)
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr

# Aggressive TCP recycling (Note: tcp_tw_recycle is deprecated/dangerous in 4.x kernels, use reuse)
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 15

# Ephemeral port range
net.ipv4.ip_local_port_range = 1024 65535

# TCP Fast Open (Reduces handshake latency)
net.ipv4.tcp_fastopen = 3

Apply these changes with sysctl -p. The tcp_fastopen flag is particularly potent for recurring visitors, as it allows data exchange during the initial SYN packet.

3. Nginx Configuration: The Gateway bottlenecks

Whether you use raw Nginx or a Lua-wrapper like Kong, the underlying Nginx configuration determines your fate. The most common mistake I see in Norway's dev teams is failing to configure upstream keepalives.

By default, Nginx acts as a polite HTTP client: it opens a connection to your backend service (e.g., a Node.js app on port 3000), sends the request, receives the reply, and closes the connection. This means every single API call involves a full TCP handshake between the Gateway and the Microservice.

You must keep these connections open.

Here is a battle-tested nginx.conf snippet for an API Gateway setup:

worker_processes auto;
worker_rlimit_nofile 65535;

events {
    multi_accept on;
    use epoll;
    worker_connections 10240;
}

http {
    # ... logs and mime types ...

    # OPTIMIZATION: Buffer Sizes
    client_body_buffer_size 10K;
    client_header_buffer_size 1k;
    client_max_body_size 8m;
    large_client_header_buffers 2 1k;

    # OPTIMIZATION: Keepalive to the client
    keepalive_timeout 30;
    keepalive_requests 100000;

    upstream backend_service {
        server 127.0.0.1:3000;
        
        # CRITICAL: Keep connections to backend open
        keepalive 64;
    }

    server {
        listen 443 ssl http2;
        server_name api.yourdomain.no;

        # SSL optimizations for 2019
        ssl_protocols TLSv1.2 TLSv1.3;
        ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256;
        ssl_prefer_server_ciphers on;
        
        # OCSP Stapling (Speeds up SSL handshake)
        ssl_stapling on;
        ssl_stapling_verify on;
        resolver 1.1.1.1 8.8.8.8 valid=300s;
        resolver_timeout 5s;

        location / {
            proxy_pass http://backend_service;
            
            # Required for upstream keepalive to work
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }
    }
}

Notice the proxy_set_header Connection ""; inside the location block. Without this, Nginx forwards the client's "close" header to the backend, defeating the purpose of the upstream keepalive directive.

Validating the Config

Don't just restart and hope. Test the configuration syntax:

If that passes, reload without dropping connections:

4. Local Data Residency and Latency

Technical tuning only goes so far. Physics is the final boss. If your users are in Oslo or Bergen, and your API Gateway is hosted in a generic cloud region in Ireland or Frankfurt, you are adding 20-40ms of round-trip time (RTT) purely due to distance.

Furthermore, with the recent intensification of GDPR enforcement and the uncertainty regarding data transfers (looking at the privacy shield debates), keeping data within Norwegian borders is becoming a compliance necessity, not just a performance tweak.

CoolVDS peers directly at NIX (Norwegian Internet Exchange). This means traffic from Norwegian ISPs often hits our servers in under 5ms. When you combine that low network latency with the kernel tuning above, the results are instantaneous.

5. The TLS 1.3 Advantage

As of late 2018, OpenSSL 1.1.1 brought us TLS 1.3 support. If you are running Ubuntu 18.04, ensure you have updated your OpenSSL packages. TLS 1.3 reduces the SSL handshake from two round trips to one. For mobile networks in Europe, this reduction is noticeable.

Check your OpenSSL version:

If it returns 1.1.1 or higher, enable TLSv1.3 in your Nginx config as shown in the block above. It's free performance.

Summary: The Checklist

Performance isn't magic; it's engineering. By controlling the stack from the KVM layer up to the TLS cipher suites, you can handle thousands of requests per second on a modest budget.

Don't let slow I/O kill your application's responsiveness. Deploy a test instance on CoolVDS today, benchmark it against your current provider, and watch the latency drop.