Kubernetes Networking: Taming the Packet Flow on Bare Metal

If you have ever tried to debug a connection timeout inside a Kubernetes cluster, you know the specific kind of headache it induces. It’s not just a network issue; it’s a layer-cake of virtual interfaces, iptables rules, and encapsulation headers. With the recent release of Kubernetes 1.3, the landscape is stabilizing, but the complexity remains high. We are moving away from the slow userspace proxy, but iptables mode brings its own chaos to the kernel.

In this deep dive, we aren't talking about theoretical clouds. We are talking about running K8s on real VPS instances, specifically here in Norway where latency to the NIX (Norwegian Internet Exchange) matters. We will look at how packets actually move, why your overlay network might be strangling your I/O, and how to fix it.

The Fundamental Lie: "Flat Network"

Kubernetes promises a flat network where every pod can talk to every other pod without NAT. This is a beautiful abstraction. Under the hood, however, it is a war zone of routing.

On a typical setup using Flannel (the most common CNI choice right now), your traffic is encapsulated in VXLAN packets. This adds overhead. Every packet is wrapped, shipped, unwrapped, and delivered. If you are hosting on a budget VPS with "shared" CPU cycles, that encapsulation process is stealing resources from your application.

Configuring the CNI Correctly

Many sysadmins just run kubectl apply -f kube-flannel.yml and hope for the best. Don't do that. You need to match the backend to your kernel capabilities. Here is a production-ready Flannel configuration stored in etcd that prioritizes the vxlan backend but ensures the MTU is handled correctly:

{
  "Network": "10.244.0.0/16",
  "SubnetLen": 24,
  "Backend": {
    "Type": "vxlan",
    "VNI": 1
  }
}

Pro Tip: If you are seeing dropped packets, check your MTU. The physical interface on your VPS usually has an MTU of 1500. The Flannel interface (flannel.1) defaults to 1450 to account for the VXLAN header. If your application attempts to push a full 1500-byte payload, fragmentation occurs, and performance nosedives. Always clamp MSS on your ingress points.

Service Discovery: The iptables Maze

Before Kubernetes 1.2, kube-proxy ran in userspace. It was stable but slow. It required context switching for every packet. Now, in 2016, the default is iptables mode. This is faster, as traffic is handled entirely in the kernel, but it turns your firewall rules into a massive list of probabilities.

When a Service is created, kube-proxy writes a series of rules. If you have 3 replicas of a backend pod, iptables uses the statistic module to load balance traffic randomly.

Let's inspect what this actually looks like on a live node. Run this command:

You will see chains that look like this:

Chain KUBE-SERVICES (2 references)
target          prot opt source               destination         
KUBE-SVC-X7Q3   tcp  --  0.0.0.0/0            10.100.23.15         /* default/my-nginx:http cluster IP */ tcp dpt:80

Chain KUBE-SVC-X7Q3 (1 references)
target          prot opt source               destination         
KUBE-SEP-4Y82   all  --  0.0.0.0/0            0.0.0.0/0            /* default/my-nginx:http */ statistic mode random probability 0.33332999982
KUBE-SEP-2Z91   all  --  0.0.0.0/0            0.0.0.0/0            /* default/my-nginx:http */ statistic mode random probability 0.50000000000
KUBE-SEP-1X55   all  --  0.0.0.0/0            0.0.0.0/0            /* default/my-nginx:http */

See that probability? That is your load balancer. It’s elegant, but it adds CPU load to the kernel if you have thousands of services. This is why underlying hardware matters.

The Ingress Controller: Exposing to the World

We are still in the early days of the Ingress resource (currently beta in 1.3), but it is the future. Instead of wasting public IPs on LoadBalancer services (which are expensive or unavailable on bare metal), we use an Nginx Ingress Controller to route traffic based on Host headers.

Here is a robust Ingress definition for a typical Norwegian e-commerce site ensuring TLS termination:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: butikk-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    ingress.kubernetes.io/ssl-redirect: "true"
spec:
  tls:
  - hosts:
    - butikk.no
    secretName: butikk-tls
  rules:
  - host: butikk.no
    http:
      paths:
      - path: /
        backend:
          serviceName: magento-frontend
          servicePort: 80

For this to work flawlessly, your Nginx controller needs high-speed disk I/O for logging and buffering, and low latency for the SSL handshake. This is where the infrastructure choice becomes critical.

Why Infrastructure Dictates Network Stability

You can tune sysctl parameters until you are blue in the face. You can tweak net.ipv4.ip_forward and maximize your file descriptors. But if your underlying VPS suffers from "noisy neighbors" or high I/O wait, your network throughput will collapse.

In a containerized environment, network packets are essentially interrupts. If the hypervisor is busy handling a storage request for another tenant, your packet waits. In the world of microservices, a 50ms delay on the network layer cascades into a 500ms delay for the end user.

This is why at CoolVDS, we enforce strict isolation. We utilize KVM (Kernel-based Virtual Machine) which provides a higher degree of separation than OpenVZ alternatives. When you run a Kubernetes minion on our NVMe storage tiers, the I/O bottleneck is virtually eliminated. This is particularly vital for the etcd cluster, which requires extremely low latency to maintain quorum.

Local Compliance and Latency

For those of us operating out of Norway, the recent invalidation of the Safe Harbor agreement (October 2015) and the upcoming GDPR regulations (expected 2018) mean data residency is no longer optional. Datatilsynet is watching. Hosting your Kubernetes cluster on CoolVDS ensures your data stays within Norwegian borders, utilizing local power and complying with local jurisdiction, all while benefiting from sub-millisecond latency to the Oslo exchange.

Summary

Kubernetes 1.3 is powerful, but it assumes your network is robust. Don't let packet encapsulation or iptables processing power starve your actual application logic.

• Use Flannel with VXLAN if you need simplicity, but watch the MTU.
• Understand the iptables rules that kube-proxy generates.
• Host on infrastructure that guarantees dedicated resources.

Your cluster is only as fast as the pipes connecting it. Don't let slow I/O kill your SEO or your uptime. Deploy a test node on CoolVDS today and see the difference raw performance makes.