Beyond Bridges: Architecting High-Performance LXC Networking with Open vSwitch

I spent last Tuesday debugging an ARP storm that took down a production cluster. The culprit? A standard Linux bridge (br0) overloaded by a developer spawning fifty LXC containers for a load test. If you are still relying on bridge-utils for anything more complex than a home server, you are building on sand.

We are seeing a shift. With the recent hype around dotCloud's "Docker" release back in March, everyone wants to run containers. But nobody is talking about how to network them securely. If you are running high-traffic workloads here in Norway—whether it is serving heavy Magento assets or handling sensitive data subject to the Personal Data Act—latency and isolation aren't optional.

This is a deep dive into Open vSwitch (OVS). We are going to rip out the standard Linux bridge and replace it with something that understands VLANs, traffic shaping, and doesn't choke when you push 50,000 packets per second (PPS).

The Limitation of Standard Linux Bridges

The standard brctl approach is dumb. It broadcasts traffic. When you have ten containers, it's fine. When you have a hundred, you generate noise. In a shared environment, this is a security nightmare. If one container is compromised, promiscuous mode tcpdump can see traffic from neighbors on the same bridge unless you have complex ebtables rules.

We need isolation. We need Software Defined Networking (SDN).

Prerequisites

I am assuming you are running Ubuntu 12.04 LTS (Precise Pangolin). If you are on 13.04, God help you with stability. We will use LXC and Open vSwitch.

# Install the necessary packages
apt-get update
apt-get install lxc openvswitch-switch openvswitch-datapath-dkms

Step 1: The OVS Switch Architecture

Instead of a dumb bridge, we create a virtual switch. This allows us to tag traffic with VLAN IDs inside the host, before it ever hits the physical network interface (NIC). This is how we at CoolVDS guarantee that Customer A's database traffic never bleeds into Customer B's web traffic, even if they reside on the same physical hypervisor.

# Create the switch
ovs-vsctl add-br ovs-br0

# Add your physical interface (eth0) to the switch
# WARNING: This will drop your SSH connection if you don't script it properly.
ovs-vsctl add-port ovs-br0 eth0

# Move the IP from eth0 to ovs-br0 (console access recommended)
ifconfig eth0 0
dhclient ovs-br0

Pro Tip: Always have IPMI or KVM-over-IP access ready when messing with OVS on a remote server. If you lock yourself out of the network stack, no amount of prayer will bring eth0 back up.

Step 2: Tagging Container Traffic

Now, let's configure an LXC container to use this switch. We aren't just plugging it in; we are assigning it a VLAN tag. This ensures the container's traffic is isolated at the kernel level.

Edit your container config (usually /var/lib/lxc/my-web-node/config):

# Network configuration
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = ovs-br0
lxc.network.name = eth0
lxc.network.veth.pair = veth-web01

# Here is the magic script to handle OVS integration
lxc.network.script.up = /etc/lxc/ovs-up.sh

You need a custom up-script because LXC (as of version 0.7.5 in the repos) doesn't natively speak OVS yet. Create /etc/lxc/ovs-up.sh:

#!/bin/bash
# Add the veth interface to the OVS bridge and tag it with VLAN 10
ovs-vsctl add-port ovs-br0 $5 tag=10

This simple tag tag=10 is the difference between a secure infrastructure and a hacking playground. All traffic leaving this container is now tagged VLAN 10.

Step 3: High Availability with Keepalived

Networking isolation is useless if the service goes down. We need a Virtual IP (VIP) that floats between our load balancers. We use VRRP via keepalived. This is industry standard.

If you are hosting critical services for Norwegian clients, you know that downtime translates directly to angry emails referencing SLAs. Here is a robust keepalived.conf for a master node:

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 101
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass s3cr3t
    }
    virtual_ipaddress {
        192.168.10.200
    }
    track_script {
        chk_haproxy
    }
}

Combine this with HAProxy 1.4 for load balancing. This setup allows you to take down one LXC node for patching without dropping a single user connection.

The Hardware Bottleneck: Why I/O Matters

You can have the most elegant OVS configuration in the world, but if your underlying storage is spinning rust (HDD), your iowait will skyrocket during log rotation or database dumps. I have seen entire OVS configurations time out because the kernel was blocked waiting on disk I/O.

This is where infrastructure choice dictates architecture. At CoolVDS, we have started deploying PCIe-based SSD storage (often referred to as NVMe in emerging specs). The random read/write speeds on these units are terrifyingly fast compared to SATA SSDs.

Compliance and Latency in Norway

Local context matters. If your servers are in Frankfurt or Amsterdam, you are adding 15-30ms of latency to your Norwegian users. That feels sluggish. Furthermore, with the Data Inspectorate (Datatilsynet) tightening scrutiny on data export, keeping data within Norwegian borders is the safest legal play.

We optimize our routing through NIX (Norwegian Internet Exchange) in Oslo. When you run mtr (My Traceroute) to a CoolVDS IP, you will see fewer hops and lower latency compared to the giants hosted across the ocean.

Final Thoughts

Containers are the future—LXC proves that. But the default networking stack in Linux wasn't built for high-density tenancy. By adopting Open vSwitch today, you are future-proofing your stack for whatever comes next (maybe this Docker thing will actually take off?).

Don't let legacy networking limit your architecture. If you need a sandbox to test OVS without risking your own hardware, spin up a high-performance instance with us.

Deploy your CoolVDS instance in Oslo today and experience raw PCIe performance.