Scaling Past the C10K Barrier: A Battle-Hardened Guide to HAProxy on CentOS 6

Let’s be honest: if you are running a business-critical application on a single server in 2012, you are not a systems administrator. You are a gambler. I recently watched a client's Magento store implode during a modest marketing campaign. The database locked up, Apache ran out of workers, and the kernel started killing processes to save memory. It was a bloodbath.

The solution isn't just "throwing more RAM at it." Vertical scaling has a ceiling, and it's expensive. The real answer is horizontal scaling—splitting the load across multiple commodity nodes. Today, we are going to set up HAProxy 1.4 (High Availability Proxy). It is the de facto standard for open-source load balancing, used by the likes of GitHub and Stack Overflow. It is fast, it is stable, and it fits perfectly on a lean CoolVDS instance.

The Architecture: Why Latency to NIX Matters

Before we touch the config files, let's talk topology. We are going to build a classic setup:

• 1x Load Balancer: HAProxy (Public IP)
• 2x Web Servers: Nginx/Apache (Private Network)

In Norway, physics is your enemy. If your load balancer is in Frankfurt but your users are in Oslo, you are adding 20-30ms of round-trip time just for the handshake. For high-frequency trading or snappy e-commerce, that lag is perceptible. This is why we deploy on CoolVDS in Oslo. We peer directly at NIX (Norwegian Internet Exchange). Your packets stay local. Your users stay happy.

Pro Tip: Don't route internal traffic over the public interface. It's insecure and eats your bandwidth quota. CoolVDS offers private VLANs. Use them. Your backend web servers should listen on private IPs (e.g., 10.0.0.x) and only accept traffic from the Load Balancer.

Step 1: Installing HAProxy on CentOS 6.3

We assume you are using a clean install of CentOS 6.3. The base repositories are often outdated, so we will grab the EPEL (Extra Packages for Enterprise Linux) repository to ensure we aren't installing ancient history.

# Import the EPEL Repo
rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

# Install HAProxy
yum install haproxy

Once installed, do not start it yet. The default config is useless for our needs.

Step 2: The Configuration (The Meat)

Back up the original config. Always.

cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.bak

Open /etc/haproxy/haproxy.cfg using vi or nano. We are going to wipe it and create a configuration that handles HTTP traffic. We will use the leastconn algorithm. Round-robin is fine for static content, but for dynamic PHP apps, you don't want to send a new request to a server that is already choking on a heavy SQL query. `leastconn` sends traffic to the server with the fewest active connections.

global
    log         127.0.0.1 local2
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon
    # Turning on stats socket for monitoring
    stats socket /var/lib/haproxy/stats

defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000

# The Frontend: This is what the public sees
frontend  main_http_incoming
    bind *:80
    default_backend             app_servers

# The Backend: This is where the work gets done
backend app_servers
    balance     leastconn
    # Cookie insertion ensures session stickiness (vital for PHP sessions)
    cookie      SERVERID insert indirect nocache
    
    # The 'check' flag enables health monitoring
    server  web1 10.0.0.2:80 cookie A check
    server  web2 10.0.0.3:80 cookie B check

# Bonus: A stats page to watch your traffic in real-time
listen stats *:1936
    stats enable
    stats uri /
    stats hide-version
    stats auth admin:supersecretpassword

Understanding the Config

We added cookie SERVERID insert. Why? HTTP is stateless. If a user logs into your application on Server A, and the next request goes to Server B, their session is lost. This setting injects a cookie into the browser so HAProxy knows to send them back to the same server for the duration of their session.

Step 3: Managing the Service

CentOS 6 uses SysVinit. We need to enable the service to start at boot and fire it up.

# specific to CentOS 6 / RedHat 6
chkconfig haproxy on
service haproxy start

# Check if it is running
netstat -tulpn | grep haproxy

You should see it listening on port 80 and port 1936.

Storage Performance: The Hidden Bottleneck

A load balancer is only as fast as the servers behind it. If your backend web servers are waiting on disk I/O, HAProxy will simply queue requests until the timeout is reached. This is where hardware selection becomes critical.

Most budget VPS providers in Europe still run on spinning HDDs (SAS or SATA) with shared spindles. The "I/O Wait" on those nodes can spike to 40% under load. At CoolVDS, we have standardized on Pure SSD storage arrays. The difference in random read/write speeds is not marginal; it is exponential. When your MySQL database needs to write session data or read product indexes, SSDs eliminate the bottleneck.

Data Privacy in 2012: The Patriot Act Factor

Hosting outside of Norway brings legal complexities. Under the current Norwegian Personal Data Act (Personopplysningsloven), you are responsible for where your user data lives. Many sysadmins are rightfully worried about the US Patriot Act, which theoretically allows US authorities access to data stored on US-owned servers, even if they are located in Europe.

By keeping your stack on CoolVDS—a Norwegian-owned entity subject to Datatilsynet and Norwegian courts—you mitigate this risk. It’s not just about ping times; it’s about compliance and sovereignty.

Troubleshooting: When Things Go Wrong

If your health checks fail (the servers turn red in the stats page), check your local firewalls on the backend nodes. IPTables is likely dropping the connection from the load balancer.

# On the Backend Web Servers (Allow traffic from LB)
iptables -I INPUT -s 10.0.0.1 -p tcp --dport 80 -j ACCEPT
service iptables save

Don't let downtime define your reputation. Implementing HAProxy is the first step toward a resilient infrastructure. The second step is ensuring the underlying hardware can keep up with your ambition. Don't let slow I/O kill your SEO or your user experience. Deploy a test instance on CoolVDS today and feel the difference of local peering and pure SSD power.