Kubernetes Networking Deep Dive: Surviving the Packet Jungle in 2022
If you have ever spent a Friday night debugging why a Pod in Node A cannot talk to a Service backed by endpoints in Node B, you know the truth: Kubernetes networking is black magic. It works beautifully until it doesn't. Then it becomes a nightmare of iptables rules, VXLAN encapsulation, and cryptic DNS timeouts.
With the release of Kubernetes 1.24 earlier this year, and the final removal of Dockershim, the landscape has shifted. We are dealing strictly with CRI-compatible runtimes like containerd now. The abstraction layer is thinner, which is good for performance, but it means you need to understand exactly how packets flow through your cluster. Especially if you are deploying here in Norway, where data sovereignty and latency to NIX (Norwegian Internet Exchange) are not just technical metrics—they are business requirements.
The CNI Dilemma: Flannel, Calico, or Cilium?
The Container Network Interface (CNI) you choose dictates your cluster's performance ceiling. Too many teams default to Flannel because it's "easy." Flannel creates a simple overlay network (VXLAN), but it lacks advanced network policies. If you are running a serious production workload, you need control. You need to block traffic between namespaces. You need BGP peering.
My go-to recommendation for 2022 is Calico. It strikes the balance between performance and feature set. It can run in pure Layer 3 mode (no encapsulation) if your underlying network supports it, or use IPIP/VXLAN if necessary. More importantly, it supports NetworkPolicies out of the box, which is mandatory for GDPR compliance when you need to prove strict isolation between workloads.
Here is how we typically bootstrap a Calico installation on a fresh cluster using the Tigera operator, which has become the standard method over raw manifests:
kubectl create -f https://projectcalico.docs.tigera.io/manifests/tigera-operator.yaml
# Configure the installation
cat <Notice encapsulation: VXLANCrossSubnet. This is a pragmatic optimization. It uses high-performance native routing within the subnet and only encapsulates packets when they cross subnet boundaries. It lowers CPU overhead significantly compared to full VXLAN.
Kube-Proxy: Stop Using Iptables
By default, kube-proxy uses iptables to handle Service VIPs. This is fine for small clusters. But iptables rules are processed sequentially. If you have 5,000 services, every packet has to traverse a linear list of rules. Performance degrades to O(n).
Switch to IPVS (IP Virtual Server). It is built on top of the netfilter framework but uses hash tables. The complexity is O(1). Whether you have 10 services or 10,000, the routing lookup time is constant. In high-traffic scenarios, this reduces the latency added by the kernel networking stack significantly.
To enable this, ensure the ip_vs kernel modules are loaded on your worker nodes, then update your kube-proxy ConfigMap:
# Check for modules
lsmod | grep ip_vs
# Edit the config map
kubectl edit configmap kube-proxy -n kube-system
# Look for 'mode' and change it:
# ...
# mode: "ipvs"
# ...
Pro Tip: If you are migrating an existing cluster to IPVS, you must flush the old iptables rules manually, or you will end up with ghost rules causing routing loops. Use iptables -t nat -F carefully during a maintenance window.
The DNS Latency Trap (ndots:5)
This is the silent killer of application performance. By default, Kubernetes sets the ndots option in /etc/resolv.conf to 5. This means if your code tries to resolve google.com, the system first tries to resolve it as an internal cluster domain: google.com.default.svc.cluster.local, then google.com.svc.cluster.local, and so on. It triggers multiple DNS queries for every single external call.
We saw a Magento backend hosted on a generic VPS provider adding 200ms of latency to every payment gateway request just because of this DNS churn. The fix is to explicitly configure the dnsConfig in your deployment manifests if you know you are hitting external FQDNs:
apiVersion: apps/v1
kind: Deployment
metadata:
name: payment-service
spec:
template:
spec:
dnsPolicy: "None"
dnsConfig:
nameservers:
- 1.1.1.1
searches:
- default.svc.cluster.local
- svc.cluster.local
- cluster.local
options:
- name: ndots
value: "2"The Infrastructure Layer: Why CoolVDS Wins
You can tune sysctl parameters and CNI configs all day, but if the underlying hypervisor steals your CPU cycles, your network latency will jitter. Kubernetes networking is CPU-intensive. Encapsulation, decryption (if using WireGuard or mTLS), and packet switching all burn cycles.
Many