Scaling Container Networking: A Deep Dive into LXC & Open vSwitch

Let’s be honest: standard server virtualization is getting heavy. If you are running a hypervisor just to spawn a dozen instances of Apache, you are wasting RAM on kernel overhead. This is why everyone in the DevOps community is looking at Linux Containers (LXC) right now. It provides near-native speeds without the bloat. But there is a catch, and it's a big one.

Networking.

Spinning up a container on localhost is trivial. Getting a container on Server A to talk to a container on Server B—without drowning in a mess of NAT rules and port forwarding—is where the real engineers are separated from the script kiddies. In a production environment, specifically here in the Nordics where we rely on the stability of NIX (Norwegian Internet Exchange), you cannot afford packet loss due to shoddy bridging.

In this deep dive, we are going to bypass the standard Linux bridge and implement a multi-host container network using Open vSwitch (OVS) and GRE tunnels. This is the architecture you need for a truly scalable infrastructure in 2013.

The Architecture: Why Bridging Fails at Scale

The traditional brctl method works fine for a single box. But once you scale out to a second node, you hit a wall. You either have to route public IPs to every container (expensive and wasteful) or build complex iptables masquerading rules that become a nightmare to debug.

The solution is an overlay network. We create a virtual switch that spans across physical servers using GRE (Generic Routing Encapsulation) tunnels. To the containers, it looks like they are all plugged into the same physical switch. To the outside world, it's just encrypted traffic flowing between your CoolVDS nodes.

Prerequisites

We assume you are running a clean install of Ubuntu 12.04 LTS. You need a VPS provider that supports custom kernel modules, specifically openvswitch-switch and gre. Many budget providers lock this down. At CoolVDS, we give you full control over the kernel space because we know you need these tools to build serious clusters.

Step 1: Installing Open vSwitch

First, forget standard bridging. We are installing OVS. On both Node A and Node B:

sudo apt-get update
sudo apt-get install openvswitch-switch openvswitch-datapath-dkms

# Verify the module is loaded
lsmod | grep openvswitch

If you see no output, your hosting provider has restricted your kernel. This is common with oversold OpenVZ legacy platforms. Switch to a KVM-based instance on CoolVDS where this works out of the box.

Step 2: Configuring the Bridge and Tunnel

We need to create a bridge named br0 and add a GRE interface that points to the remote server's IP.

On Node A (IP: 192.168.1.10):

# Create the bridge
sudo ovs-vsctl add-br br0

# Add the GRE tunnel pointing to Node B
sudo ovs-vsctl add-port br0 gre0 -- set interface gre0 type=gre options:remote_ip=192.168.1.20

On Node B (IP: 192.168.1.20):

# Create the bridge
sudo ovs-vsctl add-br br0

# Add the GRE tunnel pointing to Node A
sudo ovs-vsctl add-port br0 gre0 -- set interface gre0 type=gre options:remote_ip=192.168.1.10

Now, both servers believe they share a physical link.

Step 3: Connecting LXC to the OVS Bridge

When you create your LXC container, you need to tell it to attach to our new OVS bridge rather than the default virbr0. Edit your container config file (usually in /var/lib/lxc/container_name/config):

lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = br0
lxc.network.hwaddr = 00:16:3e:xx:xx:xx
lxc.network.ipv4 = 10.0.0.2/24

Pro Tip: When defining internal IPs for your containers (like 10.0.0.x), ensure your MTU settings account for the GRE overhead. A standard MTU of 1500 will cause packet fragmentation and kill your performance. Set the inner interface MTU to 1450.

# Inside the container network config or start script
ifconfig eth0 mtu 1450

Performance: The I/O Bottleneck

Networking is only half the battle. When you have ten containers writing logs simultaneously, disk I/O becomes your enemy. In 2013, we still see too many providers offering spinning rust (HDD) as "enterprise storage." It’s unacceptable.

For database-heavy containers (MySQL, PostgreSQL), input/output wait times (iowait) can bring a cluster to its knees. We benchmarked this setup on standard SATA drives versus the Enterprise SSD arrays used in CoolVDS instances.

If you are aggregating logs or running high-transaction databases in your LXC cluster, SSDs are not a luxury—they are a requirement for stability.

Security and Compliance in Norway

Running a tunnel over the public internet brings security implications. While GRE encapsulates traffic, it does not encrypt it by default. If you are transmitting sensitive customer data, you are likely violating the Personal Data Act (Personopplysningsloven) monitored by Datatilsynet.

For production, you must layer IPsec over your GRE tunnel. This adds CPU overhead. This is why we recommend CoolVDS instances with dedicated CPU cores rather than shared threads. You need that raw compute power to handle encryption without slowing down the application layer.

Final Thoughts

Orchestrating containers across hosts is the next evolution of system administration. Tools like Open vSwitch are making it possible to build data-center-grade networks on virtual infrastructure. But software is only as good as the hardware underneath it.

Don't let legacy kernels or slow spinning disks bottleneck your architecture. Build your cluster on a platform designed for the modern stack.

Ready to scale? Deploy a KVM instance on CoolVDS today and get full root access to build your OVS network in minutes.