Kubernetes 1.1 Networking: Surviving the Packet Jungle in Production

Let’s be honest. Getting Kubernetes up and running is the easy part. You spin up a master, join a few minions, and the API server responds. Great. But then you try to get a frontend pod on Node A to talk to a backend pod on Node B, and suddenly your packets vanish into the void.

I’ve spent the last three weeks debugging a distributed cluster that kept dropping connections under load. The culprit wasn't code; it was the assumption that network overlays are magic. They aren't. They are encapsulation headers, and they cost CPU cycles.

If you are deploying Kubernetes 1.1 in production today—especially here in Norway where we actually care about latency to the NIX (Norwegian Internet Exchange)—you need to understand what happens to a packet when it leaves a container. This isn't theoretical. This is about keeping your TTY open when the load balancer gets hit.

The "IP-Per-Pod" Mandate

Docker’s default networking model (the bridge mode we all know) is a nightmare for multi-host setups. You end up mapping ports and praying for no collisions. Kubernetes rejects this.

The Golden Rule: Every Pod gets its own IP address. Pod A can talk to Pod B directly, without NAT, even if they are on different physical nodes.

This flat address space is beautiful for developers but a headache for ops. Since our underlying physical network (the VDS provider) doesn't know about these Pod IPs, we need an overlay network. Right now, in early 2016, your best bet is Flannel (by CoreOS). Weave is an option, but Flannel’s simplicity with VXLAN is winning for raw throughput.

Configuring Flannel for Performance

Flannel runs a small binary, flanneld, on every node. It allocates a subnet to each host. But here is where people mess up: they stick with the default UDP backend. Do not use UDP. It moves packets into userspace, wraps them, and sends them back. It is slow.

Use the VXLAN backend. It leverages the kernel’s native support for VXLAN, offloading packet encapsulation. This is critical on virtualized hardware.

First, configuring your etcd key (assuming you have etcd running):

etcdctl set /coreos.com/network/config '{
    "Network": "10.244.0.0/16",
    "Backend": {
        "Type": "vxlan"
    }
}'

Once flanneld starts, it creates a flannel.1 interface. Check it:

$ ip -d link show flannel.1
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT group default 
    link/ether 6e:f5:12:03:1a:2b brd ff:ff:ff:ff:ff:ff promiscuity 0 
    vxlan id 1 local 192.168.10.5 dev eth0 srcport 0 0 dstport 8472 nolearning ageing 300

Notice the MTU 1450? This is the "Packet Tax." The VXLAN header eats 50 bytes. If your application attempts to push a full 1500-byte packet, it fragments. Fragmentation kills performance. Ensure your Docker daemon is started with the correct MTU to match the overlay:

# /etc/default/docker
DOCKER_OPTS="--mtu=1450 --bip=10.244.15.1/24"

The Kube-Proxy Bottleneck

Services in Kubernetes are just iptables rules. When you create a Service, `kube-proxy` watches the API and updates the node's firewall rules.

In K8s 1.0, `kube-proxy` ran in userspace mode. It was a literal proxy process. Traffic went from Kernel -> Userspace (kube-proxy) -> Kernel. It was stable but slow. Latency spiked under load.

In Kubernetes 1.1, we have the new iptables mode. It is technically experimental, but if you care about performance, you should use it. It handles routing purely in kernel space via netfilter. No context switching.

You can verify which mode you are running by checking the logs or inspecting the nat table:

sudo iptables -t nat -L KUBE-SERVICES

Chain KUBE-SERVICES (2 references)
target     prot opt source               destination         
KUBE-SVC-TCOU7JCQXEZGVUNU  tcp  --  anywhere             10.0.0.123           /* default/kubernetes:https cluster IP */ tcp dpt:443
KUBE-SVC-BECE2KVLZ4H7QW2E  tcp  --  anywhere             10.0.0.215           /* default/my-nginx: cluster IP */ tcp dpt:80

If you see these chains, you are good. If you don't, you are proxying through userspace. Fix it.

The Hardware Reality: Why Your VDS Matters

Here is the uncomfortable truth: You can tune sysctl and iptables all day, but if your underlying host is garbage, your cluster will crawl. Overlay networks like VXLAN require the processor to do extra work for every single packet.

Many "cheap" VPS providers in Europe oversell their CPU. When your neighbor spins up a Bitcoin miner, your UDP packet encapsulation stalls. This creates jitter. In a microservices architecture, one stalled service creates a cascade of timeouts.

This is why we built CoolVDS on KVM (Kernel-based Virtual Machine) with strict isolation. Unlike OpenVZ containers (which share a kernel and often block TUN/TAP devices needed for Flannel), KVM gives you a dedicated kernel. You can load your own modules. You can tune your own TCP stack.

Privacy & The "Schrems" Fallout

We cannot talk about networking in 2016 without mentioning the legal elephant in the room. The European Court of Justice invalidated Safe Harbor last October. If you are piping traffic through US-owned cloud load balancers, you are currently in a legal grey zone regarding Norwegian data privacy laws (Personopplysningsloven).

Running your Kubernetes cluster on Norwegian soil, with a provider like CoolVDS, isn't just a latency play—it's a compliance play. Keep the packets inside the border.

Troubleshooting Checklist

If your pods can't talk, run this gauntlet:

1. Check UDP Port 8472: Ensure your security groups allow UDP traffic on port 8472 between all nodes. This is the VXLAN highway.
2. Validate MTU: Run ping -s 1442 -M do <pod-ip>. If it fails, your MTU is mismatched.
3. Inspect Docker Bridge: Ensure docker0 is not conflicting with your physical network CIDR.
4. Check IP Forwarding: It sounds stupid, but verify it: sysctl net.ipv4.ip_forward must be 1.

Kubernetes is the future of operations, but it demands respect for the networking layer. Don't treat the network as an abstraction. It's plumbing. If the pipes clog, the house floods.

Ready to build a cluster that actually performs? Deploy a KVM instance on CoolVDS today. We give you the raw kernel access you need to run Kubernetes 1.1 the way it was meant to be run.