Kubernetes Networking: Stop Guessing, Start Tracing Packets

Let’s be honest. You didn't adopt Kubernetes because it was simple. You adopted it because managing Docker containers across five different servers with shell scripts and hope was destroying your weekends. But now you’ve traded explicit port mapping for a black box called the "Pod Network," and when a service times out, `docker logs` won't save you.

I recently spent 48 hours debugging a cluster where intermittent packet loss was plaguing a microservices deployment. The developers blamed the code; the network engineers blamed the firewall. The culprit? A mismatch in MTU settings on the VXLAN overlay interface. This is the reality of Kubernetes networking in late 2016. It promises a flat address space, but underneath, it's a warzone of routing tables and encapsulation.

If you are running Kubernetes in production—or just trying to get `kubeadm` to finish initializing without error—you need to understand what happens to a packet when it leaves a Pod.

The Great Illusion: IP-Per-Pod

The fundamental requirement of the Kubernetes network model is simple: every Pod gets its own IP address. Containers within a Pod share `localhost`, and Pods can talk to all other Pods without NAT. It sounds elegant.

But unless you are Google running on a custom L3 network, you are likely running an overlay network. This means your data is wrapped inside another packet, shipped across the physical network, and unwrapped on the other side. This adds overhead. CPU overhead for encapsulation, and network overhead for headers.

Pro Tip: If you are seeing strange connection resets, check your MTU. The overlay adds headers (often 50 bytes for VXLAN). If your physical interface is 1500 and your CNI plugin defaults to 1500, packets will fragment or drop. Set your overlay MTU to 1450.

Choosing Your Weapon: CNI Plugins in 2016

The Container Network Interface (CNI) is the pluggable layer that makes this happen. Right now, there are two main contenders for general use:

1. Flannel (The "It Just Works" Option)

Flannel from CoreOS is the standard for a reason. It creates a simple overlay mesh. By default, it uses VXLAN. It's easy to set up, but that encapsulation hits your CPU.

# A typical Flannel DaemonSet config snippet (v1.4 compatible)
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: kube-flannel-ds
  namespace: kube-system
spec:
  template:
    spec:
      containers:
      - name: kube-flannel
        image: quay.io/coreos/flannel:v0.6.1-amd64
        command: [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr" ]
        securityContext:
          privileged: true

2. Calico (The Performance Option)

Calico operates at Layer 3 using BGP. No encapsulation if you are within the same L2 segment. It’s faster, but debugging BGP route reflection is not for the faint of heart. However, if latency is your enemy, Calico is your friend.

The `iptables` Nightmare: Service Discovery

So, your Pods have IPs. How does traffic get to a Service VIP (Virtual IP)? Magic? No. `iptables`.

The `kube-proxy` component runs on every worker node. In Kubernetes 1.2+, the default mode switched from userspace proxying to pure `iptables`. This was a massive performance win, but it means debugging requires reading generated chains that look like spaghetti.

Run this on one of your worker nodes:

sudo iptables -t nat -L KUBE-SERVICES -n | head -n 10

You will see a chain of rules that probabilistically rewrite destination IPs. If you have 3 replicas of a backend, `iptables` uses the `statistic` module to load balance requests randomly (33% probability to the first, 50% of the remainder to the second, etc.).

Why Underlying Hardware Matters (The CoolVDS Factor)

Here is the hard truth that cloud providers obscure: Virtualization steals cycles.

When you run an overlay network (Packet A) inside a KVM/Xen hypervisor (Packet B), you are dealing with double encapsulation. If your host node is fighting for CPU time due to "noisy neighbors," your network latency spikes. This is where CoolVDS differs.

We use strict KVM isolation and, crucially, we don't oversell our cores. When `kube-proxy` needs to process thousands of packet rewrites per second for a high-traffic Nginx ingress, you need that raw CPU power available instantly.

The Hidden Killer: Etcd Latency

Networking config lives in `etcd`. Every time a Pod dies and comes back, the network state updates. If `etcd` is slow, your cluster state drifts. `etcd` is incredibly sensitive to disk write latency (fsync).

Most budget VPS providers run on shared SATA SSDs or spinning rust. If your disk write latency exceeds 10ms, `etcd` starts throwing leader election timeouts. Your network policy updates stall. Your cluster effectively freezes.

At CoolVDS, we deploy purely on NVMe storage. We’ve benchmarked our disk latency consistently below 1ms. For a Kubernetes control plane, this isn't a luxury; it's a requirement.

# Check your etcd disk latency (run on master node)
iostat -x -d 1 5

Look at the `await` column. If it's consistently over 10, move your control plane immediately.

Local Context: The Nordic Advantage

For those of us deploying here in Norway, we have unique advantages and constraints. The NIX (Norwegian Internet Exchange) in Oslo provides incredible peering options. If your target audience is Norwegian users, routing traffic through a "big cloud" region in Ireland or Frankfurt adds unnecessary milliseconds.

Hosting your Kubernetes nodes on CoolVDS in Oslo ensures your traffic stays local. Plus, with the Data Protection Directive (95/46/EC) and the upcoming privacy regulations everyone is whispering about for 2018, keeping data within national borders is becoming a boardroom conversation, not just a tech one.

Conclusion

Kubernetes is the future of deployment, but it doesn't abstract away the physics of networking. Packets still need to flow, and disks still need to write. Don't cripple your sophisticated orchestration layer by running it on substandard infrastructure.

If you are ready to stop debugging network flakes and start shipping code, you need a foundation that respects the packet. Spin up a CoolVDS NVMe instance today—because your `iptables` rules deserve to execute instantly.