Escaping Localhost: Advanced Multi-Host Container Networking with Open vSwitch

It is April 2014. Docker 0.10 has just landed. Everyone is talking about containers replacing VMs, but nobody wants to talk about the elephant in the server room: Networking.

If you are running a single Docker host on your laptop, the default docker0 bridge is fine. But I have spent the last week migrating a high-traffic e-commerce cluster from bare metal to containers, and let me tell you: the default networking stack does not cut it for production. When you rely on standard Docker NAT, you are introducing latency. In a high-frequency trading environment or a heavy read/write Magento backend, that NAT overhead adds up. Every packet rewritten is CPU cycles stolen from your application.

Furthermore, managing port mappings (`-p 8080:80`) across a cluster of ten servers is a chaotic mess. You need a flat network where containers on Host A can talk to containers on Host B without Network Address Translation (NAT) getting in the way.

This is not a tutorial for beginners. This is how we architect real container networks using Open vSwitch (OVS) and KVM.

The Limitation: Why standard bridges fail

By default, Docker allocates IPs from a private range (usually `172.17.0.0/16`) and NATs traffic out via the host's IP. This works, but it obscures the container's identity. Your firewall logs show the host IP, not the container IP. If you are hosting in Norway and need to comply with strict logging requirements for the Datatilsynet (Data Inspectorate), losing that granularity is a compliance risk.

We need true L2 bridging. We want our containers to have routable IPs on the LAN, or at least a dedicated VLAN spanning multiple hosts.

The Solution: Open vSwitch & Pipework

To solve this, we replace the Linux bridge with Open vSwitch. OVS is a production-quality, multilayer virtual switch. It supports VLAN tagging, VXLAN, and GRE tunneling—essential for multi-host setups.

Infrastructure Note: This setup requires kernel-level access to network modules. You cannot do this on shared hosting or legacy OpenVZ containers. You need a true hypervisor. I am running these tests on CoolVDS KVM instances in Oslo because they pass the full CPU flags and allow custom kernel modules without support tickets. Stability on the Norwegian grid is just a bonus.

Step 1: Preparing the Host (Ubuntu 14.04 LTS)

Trusty Tahr (14.04) was released just days ago. If you're on it, here is how we prep the switch.

# Update and install OVS
sudo apt-get update
sudo apt-get install openvswitch-switch

# Verify the module is loaded
lsmod | grep openvswitch

If that returns nothing, your VPS provider has locked down your kernel. Move to a provider that respects your root access.

Step 2: Creating the OVS Bridge

We are going to create a bridge named ovs-br0 and attach it to our physical interface (e.g., eth0). Warning: Doing this remotely can sever your SSH connection if you aren't careful. Ideally, use the CoolVDS out-of-band VNC console for this part.

# Create the bridge
sudo ovs-vsctl add-br ovs-br0

# Map the physical interface to the bridge (CAREFUL HERE)
sudo ovs-vsctl add-port ovs-br0 eth0

# Zero out the eth0 IP and move it to the bridge
sudo ifconfig eth0 0.0.0.0
sudo dhclient ovs-br0

Step 3: Connecting Containers with Pipework

Docker doesn't natively support OVS yet. To bridge the gap, we use pipework, a shell script by Jérôme Petazzoni that is currently the industry standard for advanced container networking.

First, launch a container without network (-net=none isn't fully stable in all versions, so we usually just let it launch and then override):

# Launch a container in detached mode
CONTAINER_ID=$(docker run -d -i -t ubuntu:14.04 /bin/bash)

# Download pipework
wget https://raw.githubusercontent.com/jpetazzo/pipework/master/pipework
chmod +x pipework

# Assign a static IP (192.168.1.50) attached to our OVS bridge
sudo ./pipework ovs-br0 $CONTAINER_ID 192.168.1.50/24

Now, your container bypasses the Docker NAT entirely. It sits directly on the network segment. The latency drops significantly because packets aren't traversing the iptables masquerade rules.

Performance: The Packet Cost

Why go through this trouble? I ran `iperf` between two containers on separate hosts connected via a gigabit switch.

In a database cluster, that latency difference is the difference between a snappy checkout and a timeout. On CoolVDS's infrastructure, which leverages high-performance SSD storage (rare in this market), removing the network bottleneck allows the disk I/O to actually shine.

Troubleshooting the "Black Hole"

When you start messing with bridges, you will eventually break something. If you can't ping your container, check the OVS flow table:

sudo ovs-ofctl show ovs-br0

Also, check your iptables. Docker loves to add forward chains that might conflict with your manual bridging.

sudo iptables -L -n -v | grep FORWARD

Often, you need to set the default policy to ACCEPT for the bridge traffic:

sudo iptables -P FORWARD ACCEPT

The Future is Clustered

Right now, manual orchestration with tools like Chef or Puppet controlling these Docker hosts is the best we have. There are rumors of Google open-sourcing their internal container manager soon, but until then, OVS plus Docker is the most robust architecture for serious setups.

Do not let your infrastructure be the bottleneck. Use KVM. Use OVS. And for the love of uptime, host it somewhere with decent peering to the rest of Europe.

Ready to build your cluster? Deploy a KVM instance on CoolVDS today. We give you the root access and kernel modules you need to actually do your job.