Stop Treating Kubernetes Networking Like Magic

It is April 2018. The container wars are effectively over. Kubernetes won. But while deploying a pod is now trivial, debugging why Service A cannot talk to Service B across nodes is still the stuff of nightmares. I recently spent 48 hours debugging a packet loss issue on a cluster that was supposedly "production-ready." It turned out to be a conflict between the overlay network and the underlying hypervisor's MTU settings.

If you are running infrastructure in Norway, you have two additional pressures right now: the looming May 25th GDPR enforcement date requiring strict data boundaries, and the need for low latency routing through NIX (Norwegian Internet Exchange). Let's tear down the abstraction layers and look at how packets actually move.

The Fundamental Lie: "Flat Networking"

Kubernetes imposes a requirement: every pod must be able to communicate with every other pod without NAT. Ideally, your cluster looks like one giant switch. In reality, unless you own the physical copper, you are likely relying on an overlay network (encapsulation).

When you run kubectl get pods -o wide, you see IP addresses. These IPs are virtual. They exist only inside the cluster's software defined network (SDN).

$ kubectl get pods -o wide --namespace=production
NAME                        READY     STATUS    RESTARTS   AGE       IP            NODE
nginx-ingress-w4p5d         1/1       Running   0          4d        10.244.1.15   node-1
redis-slave-8b6h9           1/1       Running   0          4d        10.244.2.8    node-2

If Node-1 tries to ping 10.244.2.8, the kernel routing table needs to know where to send that packet. This is where CNI (Container Network Interface) plugins come in.

CNI Showdown: Flannel vs. Calico

In 2018, you are likely choosing between these two. Your choice dictates your latency and CPU overhead.

1. Flannel (VXLAN)

Flannel is the default for many because it is simple. It uses VXLAN to encapsulate packets. It wraps your TCP packet inside a UDP packet, sends it to the other node, and unwraps it.

Pro Tip: VXLAN adds overhead. Each packet requires CPU cycles to encapsulate and decapsulate. On a budget VPS with shared CPU (