Surviving the Packet Storm: A Deep Dive into Kubernetes 1.11 Networking

Let’s be honest. The first time you looked at `iptables-save` on a Kubernetes node, you probably wanted to close the terminal and go back to managing monolithic VMs. It’s a mess of NAT tables, masquerading rules, and virtual interfaces that seems designed to break at 3 AM on a Sunday.

I've been there. In a recent deployment for a Norwegian fintech startup, we hit a wall. Random 502 errors between microservices. The logs were clean. The application code was fine. The culprit? A mismatch in MTU settings between the overlay network (Flannel) and the underlying physical interface on the host, causing packet fragmentation that silently dropped requests under load. That’s the reality of Kubernetes networking: it works like magic until it doesn't.

With the release of Kubernetes 1.11 this summer, things are getting better (hello, IPVS General Availability), but the complexity remains. In this post, we are going to rip open the hood of the cluster network, look at CNI choices available in 2018, and discuss how to secure your traffic to keep the Datatilsynet (Norwegian Data Protection Authority) happy.

The Flat Network Lie

Kubernetes imposes a fundamental requirement: all pods must be able to communicate with all other pods without NAT. This creates a "flat" network view. But unless you are running on bare metal with a very specific switch configuration, this is an illusion created by your Container Network Interface (CNI) plugin.

Your choice of CNI determines your latency, security features, and debugging misery level. Let's look at the two contenders dominating the landscape right now.

Flannel vs. Calico: The 2018 Showdown

If you are building a simple cluster for a dev environment, Flannel is fine. But if you are pushing serious traffic or need strict segmentation for GDPR compliance, Calico is the professional choice. The lack of encapsulation means lower CPU usage on the host, which translates to better packet throughput.

Pro Tip: When using overlay networks like VXLAN, CPU steal time is your enemy. Packet encapsulation requires raw compute. This is why we configure CoolVDS instances with KVM and dedicated CPU slices. If your provider over-provisions CPUs (common in budget VPS), your network latency will spike unpredictably as the hypervisor context switches.

Switching to IPVS: The Performance Booster

Prior to Kubernetes 1.11, `kube-proxy` relied heavily on iptables. When you have 5,000 services, iptables becomes a sequential list of rules that the kernel has to traverse linearly. It’s slow. O(n) complexity slow.

Kubernetes 1.11 marked IPVS (IP Virtual Server) as generally available. IPVS uses hash tables, making rule lookups O(1). It is significantly faster and supports better load balancing algorithms like least-connection.

To enable this in your cluster, you need to ensure the modules are loaded on your underlying Linux node (likely CentOS 7 or Ubuntu 16.04/18.04).

# Load kernel modules required for IPVS
modprobe ip_vs
modprobe ip_vs_rr
modprobe ip_vs_wrr
modprobe ip_vs_sh
modprobe nf_conntrack_ipv4

# Check if they are loaded
lsmod | grep -e ip_vs -e nf_conntrack_ipv4

Then, update your `kube-proxy` config map to switch modes:

apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: "ipvs"
ipvs:
  scheduler: "rr"
  strictARP: false

Restart your kube-proxy daemonset, and you've just future-proofed your cluster's networking throughput.

Locking Down Traffic (GDPR & Security)

By default, Kubernetes allows all traffic. Any pod can talk to any pod. In the context of GDPR and modern security practices, this is a nightmare. If a hacker breaches your frontend, they have a direct line to your database.

We need NetworkPolicies. These are firewall rules for pods. Note that you need a CNI that supports this (like Calico). Flannel users are out of luck here unless they layer Calico on top (Canal).

Here is a real-world example. We want to ensure that only the `frontend` pods can talk to the `backend` api, and nothing else.

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: access-backend
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - protocol: TCP
      port: 8080

Applying this creates a default deny for everything else targeting the backend. This is crucial for compliance. It proves you have taken technical measures to separate data concerns.

The Latency Factor: Why Hosting Location Matters

You can optimize your CNI and switch to IPVS, but you cannot defeat the speed of light. If your customers are in Oslo or Bergen, and your cluster is hosted in a massive datacenter in Virginia or even Frankfurt, you are adding latency on every request.

Round trip time (RTT) from Oslo to Frankfurt is roughly 15-20ms. RTT from Oslo to a local datacenter is <2ms. For a microservices architecture where a single user request might spawn 10 internal service calls (some of which might reach out to external managed services), that latency compounds.

We built CoolVDS specifically for this Nordic context. Our infrastructure peers directly at NIX (Norwegian Internet Exchange). When you deploy a Kubernetes node on our VPS Norway platform, you aren't just getting a VM; you are getting a low-latency pipe to your local user base.

Storage I/O and Etcd

Kubernetes stores all cluster state in etcd. Etcd is incredibly sensitive to disk write latency. If your disk is slow, etcd heartbeats fail, and your cluster creates a split-brain scenario or thinks nodes are dead.

Spinning rust (HDD) doesn't cut it for etcd in 2018. You need NVMe storage. We’ve seen clusters implode because the hosting provider used shared SATA SSDs that got choked by a neighbor doing heavy database imports. This is why CoolVDS enforces strict I/O isolation and uses enterprise-grade NVMe. It keeps `fsync` times low so your cluster state stays consistent.

Debugging 101: When Packets Drop

When things break, don't guess. Measure. Here is my go-to toolkit for debugging network issues inside a pod. Since most container images are stripped down (Alpine/Distroless), you often need to launch a sidecar debugger.

# Launch a netshoot pod for debugging (contains tcpdump, curl, nslookup)
kubectl run tmp-shell --rm -i --tty --image nicolaka/netshoot -- /bin/bash

# Once inside, check DNS resolution (often the actual problem)
nslookup backend-service.production.svc.cluster.local

# Check connection timing
curl -w "@curl-format.txt" -o /dev/null -s http://backend-service:8080/health

If DNS works but TCP fails, check your NetworkPolicies. If TCP connects but times out, check your MTU settings on the host interface.

Conclusion

Kubernetes 1.11 provides the tools to build robust, scalable platforms, but the default settings are rarely production-ready for high-performance needs. By choosing the right CNI (Calico), leveraging IPVS, and implementing strict NetworkPolicies, you build a fortress, not just a cluster.

However, software configuration is only half the battle. The hardware beneath your feet determines your stability. Don't let slow I/O kill your etcd cluster or poor routing add latency to your Norwegian users.

Ready to stabilize your infrastructure? Deploy a CoolVDS instance with pure NVMe storage and direct NIX peering today. Your packets deserve the express lane.