Scaling Beyond Localhost: A Deep Dive into Multi-Host Docker Networking and Linux Bridges

Let’s be honest: Docker is brilliant for development, but deploying it into production across multiple nodes is currently a networking minefield. If you are running version 0.9 or the brand new 0.10, you know the pain. The default docker0 bridge is isolated to a single host. You are stuck managing a fragile web of NAT rules, random high-number ports, and manual --link flags that break the moment a container restarts.

I recently spent three sleepless nights trying to get a Redis cluster talking to a Python Flask frontend across two different physical servers in Oslo. The latency was killing us, and `iptables` became a tangled mess. This post is for those who are done playing with `localhost` and are ready to engineer actual infrastructure.

The Problem: The "Ambassador" Pattern is Not Enough

Right now, the community is pushing the "Ambassador Pattern" heavily. While clever, chaining socat pipes between containers just to route traffic across hosts adds unnecessary overhead. In a high-performance environment—like the financial systems we host here in Norway—every millisecond of context switching counts.

When you rely on standard VPS hosting, you often hit a wall because the provider restricts kernel modules. You try to set up a GRE tunnel or install Open vSwitch (OVS), and the hypervisor denies you. This is why we built CoolVDS on pure KVM. We give you the kernel access you need to manipulate the network stack, which is mandatory for what I am about to show you.

Solution: flattening the Network with Open vSwitch

To get real performance, we need to bypass Docker's default NAT and treat containers as first-class citizens on the network. We are going to use Open vSwitch (OVS) to create a GRE tunnel between two CoolVDS instances. This simulates a local LAN over the public internet, allowing containers on Host A to ping containers on Host B directly.

Step 1: Preparing the Hosts

Assume we have two servers running the freshly released Ubuntu 14.04 LTS:

• Node A (Oslo): 192.168.1.10 (Public IP)
• Node B (Bergen): 192.168.1.11 (Public IP)

First, install OVS. Don't rely on the default repositories if they are outdated; grab the latest stable source if you can, but apt-get is usually fine for 2.0.1.

sudo apt-get update
sudo apt-get install openvswitch-switch

Step 2: Configuring the Bridge

On both nodes, we create a new bridge named obr0. This will replace the default docker bridge eventually.

sudo ovs-vsctl add-br obr0
sudo ifconfig obr0 172.16.42.1 netmask 255.255.255.0 up

Step 3: Creating the GRE Tunnel

This is where the magic happens. We encap traffic from Node A to Node B. Note that GRE protocol 47 must be allowed through your firewall. On CoolVDS instances, we provide a clean pipe, but check your `iptables`.

On Node A:

sudo ovs-vsctl add-port obr0 gre0 -- set interface gre0 type=gre options:remote_ip=192.168.1.11

On Node B:

sudo ovs-vsctl add-port obr0 gre0 -- set interface gre0 type=gre options:remote_ip=192.168.1.10

Connecting Docker to the OVS Bridge

Now we need to tell the Docker daemon to use our new OVS bridge instead of creating its own. This requires editing your Docker init configuration (usually in /etc/default/docker on Ubuntu).

# /etc/default/docker
DOCKER_OPTS="-b obr0 -H tcp://0.0.0.0:4243 -H unix:///var/run/docker.sock"

Restart the Docker service. Now, when you launch a container, it gets an IP on the 172.16.42.x subnet. However, we have a routing issue. Both hosts will try to assign the same IPs. Until Docker releases better orchestration tools, we have to manually segment the network or use a tool like pipework.

Pro Tip: Jérôme Petazzoni’s pipework script is currently the most robust way to assign static IPs to containers. Download it immediately.

Instead of relying on the daemon's IP assignment, let's launch a container with no networking and attach it manually:

# Launch container without net
docker run -i -t --net=none --name=web_worker ubuntu /bin/bash

# Assign specific IP on Node A using pipework
sudo pipework obr0 web_worker 172.16.42.10/24

Do the equivalent on Node B with IP 172.16.42.11. Now, ping from web_worker on Node A to the container on Node B. You should see sub-millisecond latency if you are within the same datacenter zone.

Performance Tuning for Production

Getting the packets to flow is only half the battle. If you are running high-traffic applications, default Linux sysctl settings will throttle you. I've seen `nf_conntrack` tables fill up instantly during load tests.

Update your /etc/sysctl.conf with these values to handle higher connection concurrency:

net.ipv4.ip_forward = 1
net.ipv4.conf.all.rp_filter = 0
net.ipv4.neigh.default.gc_thresh1 = 4096
net.ipv4.neigh.default.gc_thresh2 = 8192
net.ipv4.neigh.default.gc_thresh3 = 16384
net.netfilter.nf_conntrack_max = 262144

Make sure to apply these with sysctl -p.

The CoolVDS Advantage: Why Hardware Matters

Virtualization overhead is the enemy of network throughput. When encapsulating packets (GRE/VXLAN), CPU usage spikes because the kernel has to process headers for every packet. On "budget" VPS providers that oversell their CPUs, this results in jitter. You might see 2ms latency one second and 200ms the next when a neighbor spins up a compile job.

At CoolVDS, we don't play the "cpu steal" game. Our KVM instances are pinned to high-frequency cores, ensuring that your software-defined networking (SDN) stack doesn't choke. Furthermore, for our Norwegian clients, keeping traffic local to NIX (Norwegian Internet Exchange) ensures that your data stays within the jurisdiction of Norwegian privacy laws—a concern that is becoming increasingly relevant given the recent discussions around Safe Harbor.

Conclusion

We are in the early days of container orchestration. Tools like Mesos are promising, and Google is rumored to be working on something internal, but for now, we have to build these bridges ourselves. By using Open vSwitch and solid KVM foundations, you can achieve a "cluster-like" experience today.

Don't let legacy infrastructure hold back your modern deployment workflow. If you want to test this OVS setup without worrying about packet loss or noisy neighbors, spin up a CoolVDS instance. We have the latency and the raw CPU power to make your packet encapsulation invisible.

Ready to build your cluster? Deploy a high-performance CoolVDS KVM instance in Oslo today.