Scaling Past the C10k Problem: Nginx Reverse Proxy Architecture for High-Traffic Sites

The Apache Bottleneck: Why Your Server is Swapping

I watched a client's server melt yesterday. It wasn't a DDoS attack; it was a marketing email. They sent out a newsletter to 50,000 Norwegian customers, and within three minutes, their 4GB RAM server was unresponsive. The culprit? The Apache prefork MPM (Multi-Processing Module).

In 2010, the standard LAMP stack is reliable, but it is heavy. Apache spawns a new process for every single connection. If a user on a slow 3G connection takes 5 seconds to download an image, that Apache process (and its 20MB-50MB of RAM) is locked up for 5 seconds. Multiply that by 500 users, and you are swapping to disk. Game over.

There is a better way. By placing Nginx in front of Apache as a reverse proxy, we can handle static files efficiently and only bother Apache for dynamic PHP processing. This is the architecture we use to keep high-traffic sites alive.

The Architecture: Nginx + Apache

The concept is simple. Nginx listens on port 80 (public). Apache listens on port 8080 (local). Nginx handles the heavy lifting of connection management—something it excels at thanks to its event-driven, asynchronous architecture. It doesn't use threads; it uses the kernel's epoll (on Linux) to handle thousands of connections with a tiny memory footprint.

Pro Tip: Do not just install packages blindly. On CentOS 5, you'll want the EPEL repository to get a recent version of Nginx (0.7.x or 0.8.x). The default repositories are often ancient.

Step 1: Reconfiguring Apache

First, we need to move Apache off port 80. Edit your /etc/apache2/ports.conf (Ubuntu) or /etc/httpd/conf/httpd.conf (CentOS):

NameVirtualHost *:8080
Listen 8080

You also need to update your virtual hosts to match. Once done, restart Apache. Your site is now down—don't panic. We are about to fix it.

Step 2: The Nginx Proxy Configuration

Now, install Nginx. Inside your nginx.conf or a specific site config in /etc/nginx/sites-available/, we define the proxy parameters. We want to pass the request to Apache, but we must ensure headers are passed correctly so Apache knows the request isn't actually coming from localhost.

server {
    listen 80;
    server_name example.no www.example.no;

    # Serve static files directly - Massive RAM saver
    location ~* \.(jpg|jpeg|gif|png|css|js|ico|xml)$ {
        access_log        off;
        expires           30d;
        root /var/www/example.no/public_html;
    }

    # Pass everything else to Apache
    location / {
        proxy_pass         http://127.0.0.1:8080/;
        proxy_redirect     off;
        proxy_set_header   Host             $host;
        proxy_set_header   X-Real-IP        $remote_addr;
        proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
        
        client_max_body_size       10m;
        client_body_buffer_size    128k;
        
        proxy_connect_timeout      90;
        proxy_send_timeout         90;
        proxy_read_timeout         90;
        proxy_buffer_size          4k;
        proxy_buffers              4 32k;
        proxy_busy_buffers_size    64k;
        proxy_temp_file_write_size 64k;
    }
}

The Critical Missing Piece: Real IP Addresses

Here is where most tutorials fail you. After setting this up, your Apache logs will show 127.0.0.1 for every single visitor. This breaks your analytics and, more importantly, your security tools like Fail2Ban. If you block 127.0.0.1, you block everyone.

To fix this, you must install libapache2-mod-rpaf on Debian/Ubuntu or mod_rpaf on CentOS. This module reads the X-Real-IP header sent by Nginx and overwrites the remote IP in Apache. It makes the proxy invisible to the backend application.

Hardware Matters: Latency and I/O

Configuration can only save you so much. If your underlying storage is slow, Nginx buffering to disk (proxy_temp_path) will bottleneck the CPU. This is especially true for database-heavy sites like Magento or heavy Drupal installations.

When hosting for the Norwegian market, physical location and disk speed are the two factors you cannot optimize via code.

At CoolVDS, we don't oversell. We use KVM virtualization which guarantees your RAM is yours. Furthermore, our nodes are connected to the NIX (Norwegian Internet Exchange) infrastructure. Under strict Norwegian data laws (Personopplysningsloven), keeping data local is not just a performance perk; it's becoming a compliance necessity for many businesses.

Conclusion: Speed is Trust

In 2010, users expect pages to load in under 2 seconds. A slow site loses money. By decoupling the connection handling (Nginx) from the logic processing (Apache), you can serve 10x the traffic on the same hardware hardware.

However, if your host's disk I/O is thrashing because of noisy neighbors, no amount of Nginx tuning will save you. You need dedicated resources and fast I/O.

Ready to stop the swapping? Deploy a high-performance, SSD-backed KVM instance on CoolVDS today. We offer native IPv6 support and direct peering in Oslo.