Surviving the Request Flood: Advanced API Gateway Tuning for Nordic Traffic

Your API isn't slow because of your business logic. In 90% of the cases I audit, the bottleneck is the gatekeeper itself. The API Gateway.

I recently inherited a mess of a Kubernetes cluster for a Fintech startup based here in Oslo. They were routing traffic through a standard ingress controller hosted on a generic hyperscaler in Frankfurt. Every time a burst of transaction requests hit—common during Norwegian tax season—their p99 latency spiked from 40ms to 2.5 seconds. The database was bored. The application pods were idling. The gateway was choking on TCP handshakes.

Most DevOps engineers slap a default NGINX or Kong instance in front of their services and call it a day. That is negligence. If you are handling sensitive traffic in the Nordics, you are fighting two wars: the physics of latency and the legal minefield of Schrems II. Here is how you win both.

1. The Kernel is the Limit

Before touching the gateway config, look at the OS. Default Linux distributions are tuned for general-purpose usage, not for handling 50,000 concurrent connections. If you stick with defaults, you will see connection reset by peer errors in your logs regardless of how powerful your CPU is.

On a high-performance CoolVDS instance running Ubuntu 20.04 or AlmaLinux 8 (my preferred choices in 2022), you need to edit /etc/sysctl.conf. We need to widen the ephemeral port range and allow the reuse of sockets in TIME_WAIT state.

# /etc/sysctl.conf

# Maximize the backlog of incoming connections
net.core.somaxconn = 65535
net.core.netdev_max_backlog = 65535

# Allow reusing sockets in TIME_WAIT state for new connections
# Critical for high-throughput API gateways
net.ipv4.tcp_tw_reuse = 1

# Increase ephemeral port range to avoid port exhaustion
net.ipv4.ip_local_port_range = 1024 65535

# Minimize the time a connection stays in FIN-WAIT-2
net.ipv4.tcp_fin_timeout = 15

Apply these with sysctl -p. If you are running this on a shared container platform, you might not have permission to change these namespaces. That is a dealbreaker. This is why we use KVM virtualization at CoolVDS; you get your own kernel context. You can tune the stack deep down to the interrupt request level.

2. NGINX: The Directives That Matter

Whether you use raw NGINX, OpenResty, or Kong, the underlying engine is the same. The biggest killer of performance is the lack of keepalive connections to the upstream backend. Without this, NGINX opens a new TCP connection to your backend service for every single request. That involves a full three-way handshake.

Here is the configuration pattern that fixed the latency issues for that Oslo fintech client:

http {
    # Worker connections determine concurrency
    worker_rlimit_nofile 65535;

    upstream backend_api {
        server 10.0.0.5:8080;
        # The number of idle keepalive connections to the upstream
        keepalive 64;
    }

    server {
        listen 443 ssl http2;
        server_name api.coolvds-client.no;

        # SSL optimizations are mandatory for latency
        ssl_session_cache shared:SSL:10m;
        ssl_session_timeout 10m;
        ssl_buffer_size 4k; # Lower buffer size reduces Time To First Byte (TTFB)

        location / {
            proxy_pass http://backend_api;
            
            # REQUIRED for keepalive to work
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            
            # Buffering adjustments
            proxy_buffers 16 16k;
            proxy_buffer_size 16k;
        }
    }
}

Pro Tip: Setting ssl_buffer_size to 4k instead of the default 16k improves TTFB significantly for API payloads, which are usually small JSON objects. It allows the first part of the data to be sent without waiting for the full 16k buffer to fill up.

3. The "Noisy Neighbor" Problem & Infrastructure

You can have the most optimized nginx.conf in the world, but if your disk I/O is fighting for scraps, your API gateway will stutter during logging operations. Access logs are synchronous by default (unless buffered). If the disk is slow, the request hangs.

In 2022, there is no excuse for spinning rust or shared SATA SSDs for gateway nodes. You need NVMe. Period.

Benchmark: Shared Cloud vs. CoolVDS Dedicated NVMe

High I/O wait (iowait) steals CPU cycles that should be encrypting packets. By using CoolVDS instances with dedicated NVMe storage, we eliminate the I/O bottleneck entirely. This ensures that when you write to /var/log/nginx/access.log, it happens instantly.

4. Data Sovereignty and Network Topology

Let's talk geography. If your users are in Norway, but your gateway is in Frankfurt or Ireland, you are adding 20-30ms of round-trip time (RTT) purely due to physics. That is the baseline floor you cannot optimize away with code.

Furthermore, Datatilsynet (The Norwegian Data Protection Authority) has been increasingly vocal following the Schrems II ruling. Routing personal data through US-owned cloud providers—even their European datacenters—is a legal grey area that is turning black.

Hosting your API Gateway on CoolVDS in our Oslo datacenter solves two problems:

1. Latency: You are peering directly at NIX (Norwegian Internet Exchange). The path from a Telenor mobile user to your API is practically instant.
2. Compliance: The data never leaves Norwegian jurisdiction.

5. Testing the Improvements

Do not guess. Verify. Use wrk to load test your setup. Run this from a separate machine (preferably another CoolVDS instance to ensure the network pipe isn't the bottleneck).

# Run a 30-second benchmark with 12 threads and 400 connections
wrk -t12 -c400 -d30s --latency https://api.yourdomain.no/v1/status

If you see a "Socket errors: connect" count greater than zero, go back to step 1 and increase your file descriptors. If your p99 latency is high but CPU usage is low, check your upstream keepalive settings.

The Verdict

Performance tuning is an iterative process. It requires a harmony between the kernel, the application configuration, and the metal it runs on. You cannot execute low-level tuning on restricted container platforms, and you cannot get sub-millisecond disk latency on shared storage.

If you are building an API that needs to scale without selling your data sovereignty to the giants, you need the right foundation. Check your sysctl.conf, fix your upstream keepalives, and get closer to your users.

Ready to drop your latency? Deploy a high-performance NVMe KVM instance on CoolVDS today and keep your data in Norway.