Unraveling Kubernetes Networking: Packets, Pods, and Pain

Let's be honest. Everyone loves the idea of immutable infrastructure and the new Kubernetes 1.2 release. The concept of declaring a Deployment and watching the scheduler handle the chaos is intoxicating. But then reality hits. You deploy your first microservices cluster, and suddenly Service A times out trying to talk to Service B. You run docker ps, everything looks fine. You look at the logs, silence.

Welcome to the networking layer. This is where the abstraction leaks.

In the monolithic days, we just tuned Nginx and ensured our VLANs were clean. Now, with the rise of Docker and container orchestration, we are building networks on top of networks (overlays) on top of virtual machines. If you don't understand what iptables is doing under the hood, or how VXLAN encapsulation affects your MTU, you are building a house of cards.

The Kubernetes Networking Model: The "Flat" Lie

Kubernetes imposes a specific networking model that differs wildly from standard Docker networking. In standard Docker, we got used to port mapping (-p 8080:80). Kubernetes throws that out the window. The fundamental rules are:

1. All containers can communicate with all other containers without NAT.
2. All nodes can communicate with all containers (and vice-versa) without NAT.
3. The IP that a container sees itself as is the same IP that others see it as.

This sounds great until you have to implement it across multiple physical or virtual hosts. To make this work, we usually rely on CNI (Container Network Interface) plugins. The two heavy hitters right now are Flannel (CoreOS) and Calico.

Flannel: The UDP Encapsulation Trap

Flannel is usually the default choice because it's simple. It creates an overlay network using VXLAN or UDP. It grabs a subnet (e.g., 10.1.0.0/16) and assigns a slice (/24) to every worker node. When a packet leaves a pod, Flannel wraps it in a UDP packet, sends it to the other host, unwraps it, and delivers it.

Here is where I see DevOps teams fail. Overhead.

Encapsulation costs CPU cycles. I recently debugged a cluster for a client in Oslo where their application latency was spiking randomly. They were running on a budget VPS provider with "shared" CPU cycles. Because Flannel relies on the kernel to encapsulate packets, every network request was stealing CPU time from the application.

Pro Tip: If you are using Flannel, ensure you are using the vxlan backend, not udp. The udp backend acts in userland and is painfully slow. The vxlan backend uses the kernel's in-tree VXLAN support.

Here is how you verify your Flannel configuration in etcd (assuming you're using etcd2):

etcdctl get /coreos.com/network/config
{ "Network": "10.1.0.0/16", "Backend": { "Type": "vxlan" } }

The MTU Nightmare

This is the specific issue that burns weeks of engineering time. Standard Ethernet MTU is 1500 bytes. When you wrap a packet in VXLAN headers, you add 50 bytes of overhead. If your inner container thinks it can send 1500 bytes, but the outer physical interface can only handle 1500, the packet gets dropped or fragmented. Fragmentation kills performance.

In a recent deployment, we saw MySQL replication failures between nodes. The fix wasn't in MySQL; it was in the Docker daemon config.

You must instruct Docker to use a smaller MTU to accommodate the overlay. On CoolVDS instances, where we provide KVM isolation, you have full control over the interface settings.

# Inside /etc/default/docker or your systemd unit file
DOCKER_OPTS="--mtu=1450 ..."

If you don't set this, the TLS handshake packets (which are often large) get dropped silently, and your application just hangs. It’s maddening to debug unless you are running tcpdump on the host interface.

iptables: The Engine of kube-proxy

With Kubernetes 1.2, kube-proxy now defaults to iptables mode instead of the old userspace mode. This is a massive performance win, increasing throughput by an order of magnitude. However, it makes debugging complex.

Instead of a simple proxy process passing bytes, Kubernetes writes thousands of NAT rules to direct traffic from a Service IP (ClusterIP) to a random Pod IP.

You can see this chaos by running:

iptables -t nat -L KUBE-SERVICES -n | head -n 20

You'll see chains of probability logic used for load balancing:

target     prot opt source               destination
KUBE-SVC-X  tcp  --  0.0.0.0/0            10.0.0.123           /* default/my-service:cluster-tcp */

# Inside KUBE-SVC-X chain:
KUBE-SEP-Y  all  --  0.0.0.0/0            0.0.0.0/0            /* 33% probability */ statistic mode random probability 0.33332999982
KUBE-SEP-Z  all  --  0.0.0.0/0            0.0.0.0/0            /* 50% probability */ statistic mode random probability 0.50000000000
KUBE-SEP-Q  all  --  0.0.0.0/0            0.0.0.0/0

If your underlying VPS has high "steal time" (noisy neighbors), the kernel takes longer to process these massive iptables chains. This adds latency to every single connection establishment.

Why Hardware Choice Dictates Network Stability

This brings us to the infrastructure. You cannot build a reliable Kubernetes cluster on oversold OpenVZ containers or budget shared hosting. The kernel switching costs and the network I/O requirements for overlay networks are too high.

When we architected the CoolVDS platform, we specifically chose KVM (Kernel-based Virtual Machine) to ensure that your kernel is yours. This allows you to load specific kernel modules needed for advanced CNI plugins like Calico or Weave without begging support to enable them on the host.

Latency to the Exchange

For our Norwegian clients, physics is the final boss. If your cluster is distributed, or if you are connecting to external APIs, latency to the NIX (Norwegian Internet Exchange) in Oslo is critical. A delay of 30ms might not seem like much, but in a microservices architecture where one user request triggers 50 internal RPC calls, that 30ms compounds into a 1.5-second delay for the user.

We optimize our routing specifically for the Nordic region. Running a mtr (My Traceroute) from a CoolVDS instance to major Norwegian ISPs usually shows single-digit latency.

Conclusion: Architect for the Worst Case

Kubernetes is powerful, but it is not magic. It is just Linux primitives—namespaces, cgroups, and iptables—wrapped in YAML. To succeed with K8s in production in 2016, you need to understand the network path of your packets.

Don't let packet fragmentation or noisy neighbors kill your cluster's performance. Start with a solid foundation.

Ready to build a cluster that actually performs? Deploy a KVM-based instance on CoolVDS today. We give you the raw root access and NVMe performance you need to tame the Kubernetes networking beast.