Architecting for Speed: API Gateway Tuning in the Norwegian Cloud

If you are running your API Gateway on default settings, you are effectively throwing money into a furnace. I recently audited a fintech setup in Oslo where a perfectly decent microservices architecture was being strangled by a default Nginx configuration, adding 200ms of latency to every transaction. In the world of high-frequency trading or real-time bidding, that is an eternity. We are seeing a massive shift in 2019 toward microservices using Kong or custom OpenResty builds, but nobody seems to be talking about the underlying OS tuning required to support thousands of concurrent connections. You cannot simply deploy a Docker container and expect it to handle the traffic of a Black Friday sale without tweaking the kernel. This guide is not for the faint of heart; we are going deep into the Linux network stack and Nginx internals to squeeze every millisecond out of your infrastructure.

The Hardware Foundation: Why "Cloud" Often Means "Slow"

Before we touch a single config file, we need to address the elephant in the server room: Hypervisor Steal Time. You can tune your TCP stack until you are blue in the face, but if your neighbor on a noisy public cloud is compiling the Linux kernel 24/7, your API performance will fluctuate wildly. This is where the choice of virtualization matters immensely. In 2019, running on legacy Xen setups often results in unpredictable I/O wait times. We utilize KVM (Kernel-based Virtual Machine) at CoolVDS because it treats the hypervisor as an OS process, ensuring fairer scheduling and strict isolation. When your API gateway logs are writing gigabytes of access data, standard SSDs choke. You need NVMe storage. If you verify your disk I/O and see high latency, stop reading and migrate your workload. No amount of software tuning fixes bad hardware.

To check if your current host is stealing CPU cycles from you, run this simple command and look at the %st (steal) column:

Layer 1: Linux Kernel Tuning for High Concurrency

The Linux kernel defaults are designed for general-purpose computing, not for handling 50,000 concurrent API connections. When an API gateway sits between your users and your backend services, it opens two connections for every request: one to the client and one to the upstream service. This exhausts ephemeral ports rapidly. We need to widen the port range and enable aggressive TCP reuse. If you don't do this, you will see TIME_WAIT sockets piling up in netstat until your server stops accepting new connections. Below is the sysctl.conf configuration I deploy on every production gateway. This adjusts the backlog queue and enables BBR congestion control, which has been available since kernel 4.9 and is a lifesaver for users connecting from mobile networks with packet loss.

/etc/sysctl.conf Optimization

# Maximize the number of open file descriptors
fs.file-max = 2097152

# Increase the size of the receive and send queues
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

# Increase the maximum number of connections in the backlog
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535

# Reuse sockets in TIME_WAIT state for new connections
net.ipv4.tcp_tw_reuse = 1

# Expand the local port range to allow more connections
net.ipv4.ip_local_port_range = 1024 65535

# Enable BBR Congestion Control (Requires Kernel 4.9+)
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr

Apply these changes immediately without rebooting by running:

Layer 2: Nginx & OpenResty Configuration

The single most common mistake I see in 2019 is the lack of keepalive connections to the upstream backend. By default, Nginx acts as a polite HTTP/1.0 client to your backend services: it opens a connection, sends the request, gets the response, and closes the connection. This requires a full TCP handshake (and potentially SSL handshake) for every single internal request. If your microservices are in the same datacenter (or even the same rack at CoolVDS), this overhead is unnecessary and expensive. You must configure the upstream block to keep connections open. Furthermore, we need to increase the worker_rlimit_nofile to match our system limits, or Nginx will hit a wall regardless of our kernel tuning.

Pro Tip: Turn off access logging for your static asset routes or high-frequency health checks. Writing to disk is the most expensive operation Nginx does per request. If you must log, use the buffer parameter.

Optimized nginx.conf Structure

worker_processes auto;
worker_rlimit_nofile 100000;

events {
    worker_connections 4096;
    use epoll;
    multi_accept on;
}

http {
    # ... mime types and other defaults ...

    # OPTIMIZATION: Disable emitting nginx version on error pages
    server_tokens off;

    # OPTIMIZATION: Zero copy data transfer
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;

    # OPTIMIZATION: Keepalive settings for clients
    keepalive_timeout 30;
    keepalive_requests 100000;

    # DEFINING THE UPSTREAM WITH KEEPALIVE
    upstream backend_service {
        server 10.0.0.5:8080;
        server 10.0.0.6:8080;
        
        # CRITICAL: Keep 64 idle connections open to the backend
        keepalive 64;
    }

    server {
        listen 443 ssl http2;
        server_name api.coolvds-client.no;

        # SSL config would go here (see next section)

        location / {
            proxy_pass http://backend_service;
            
            # REQUIRED for HTTP/1.1 Keepalive to backends
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
        }
    }
}

Always verify your syntax before reloading, as a typo in a production gateway brings down the whole platform:

Layer 3: TLS 1.3 and The Cost of Encryption

We are now in 2019, and with the release of OpenSSL 1.1.1 last year, TLS 1.3 is finally stable and ready for production. If you are serving traffic to modern clients, enabling TLS 1.3 is the easiest performance win you will ever get. It reduces the SSL handshake from two round-trips to just one (1-RTT). For users on mobile networks in rural Norway, this reduction in latency is perceptible. However, you must ensure your underlying OS has a new enough OpenSSL version. On older CentOS 7 systems, this can be a pain, but our CoolVDS templates for Ubuntu 18.04 LTS come with this support out of the box. Do not neglect ssl_session_cache; caching SSL parameters allows returning clients to resume sessions without a full handshake, saving significant CPU resources.

Modern SSL Configuration Block

ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384';
ssl_prefer_server_ciphers on;

# Session Caching
ssl_session_cache shared:SSL:50m;
ssl_session_timeout 1d;
ssl_session_tickets off;

# HSTS (Enable with caution, ensures browsers only use HTTPS)
add_header Strict-Transport-Security "max-age=63072000" always;

# OCSP Stapling
ssl_stapling on;
ssl_stapling_verify on;
resolver 8.8.8.8 8.8.4.4 valid=300s;
resolver_timeout 5s;

Local Latency and Data Sovereignty

There is a legal and physical aspect to performance. With the implementation of GDPR last year (May 2018), and the watchful eye of Datatilsynet here in Norway, keeping data close to home is not just about speed—it is about compliance. Hosting your API Gateway in Frankfurt or London adds 20-40ms of round-trip time (RTT) for users in Oslo. By hosting locally on CoolVDS, you leverage the Norwegian Internet Exchange (NIX), dropping that latency to sub-5ms for local traffic. Low latency improves the perception of your application's speed more than any amount of JavaScript optimization ever could.

Performance tuning is an iterative process. You cannot manage what you do not measure. Use tools like wrk or ab to benchmark your changes.

If you are tired of fighting with noisy neighbors and sluggish I/O on commodity VPS providers, it is time to upgrade. A properly tuned Nginx instance on a CoolVDS NVMe server doesn't just run; it flies. Deploy your optimized gateway today and watch your wait times plummet.