Kubernetes Networking in Production: Debugging the Invisible Packet Drops

"It works on my laptop" is the most expensive lie in our industry. In a local Kind cluster or Minikube, networking is a polite fiction. You don't deal with packet fragmentation, neighbor table overflows, or the random latency spikes that occur when a noisy neighbor on a generic cloud provider decides to mine crypto on the same physical host. When you move to production, networking stops being an abstraction and starts being a crime scene.

I have spent the last six months migrating a fintech workload from a monolithic bare-metal setup to Kubernetes 1.27. The compute part was easy. The storage was manageable. The networking? That was where the bodies were buried. If you are running mission-critical workloads in Norway or the broader EU, you cannot treat the network as a black box. This guide cuts through the vendor marketing fluff and looks at what actually happens to a packet when it leaves your pod.

The CNI Decision: Stop Using Flannel by Default

In 2023, the default CNI (Container Network Interface) choice determines your debugging capability for the next three years. Too many teams start with Flannel because it is easy, then panic when they need network policies. For high-throughput environments, you are essentially choosing between Calico (standard iptables/IPVS) and Cilium (eBPF).

Last month, we debugged a cluster where API latency fluctuated wildly between 10ms and 400ms. The culprit wasn't the application code; it was the overhead of thousands of iptables rules being updated every time a pod scaled up or down. We switched to an eBPF-based data path, and the latency flatlined immediately.

Pro Tip: If you are running on kernels 5.10+ (which you should be), eBPF is not experimental anymore. It is the standard for performance.

Implementing eBPF for Performance

If you are deploying on a KVM-based platform like CoolVDS, you have full control to load the necessary kernel modules. Here is how we configure Cilium to replace kube-proxy entirely, bypassing the legacy netfilter path for service resolution:

helm install cilium cilium/cilium --version 1.14.0 \n  --namespace kube-system \n  --set kubeProxyReplacement=strict \n  --set k8sServiceHost=${API_SERVER_IP} \n  --set k8sServicePort=${API_SERVER_PORT} \n  --set loadBalancer.mode=dsr

Setting kubeProxyReplacement=strict is the critical flag here. It removes the need for kube-proxy to touch iptables, pushing service load balancing directly into the kernel via eBPF maps. On high-traffic nodes, this reduces CPU context switching significantly.

DNS: The Root of All Evil

It is never the network; it is always DNS. In Kubernetes, ndots:5 is the default configuration that silently kills performance. If your application tries to resolve api.external.com, the glibc resolver will walk through the search domains first:

• api.external.com.namespace.svc.cluster.local
• api.external.com.svc.cluster.local
• api.external.com.cluster.local
• ...and so on.

This multiplies DNS traffic by 5x or more. If your underlying infrastructure has high latency to the upstream nameserver, your application hangs. We mitigated this recently for a client in Oslo by tuning the `Corefile` directly to force TCP for upstream connections, which stabilizes resolution over slightly jittery public networks.

apiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: coredns\n  namespace: kube-system\ndata:\n  Corefile: |\n    .:53 {\n        errors\n        health {\n           lameduck 5s\n        }\n        ready\n        kubernetes cluster.local in-addr.arpa ip6.arpa {\n           pods insecure\n           fallthrough in-addr.arpa ip6.arpa\n           ttl 30\n        }\n        prometheus :9153\n        forward . /etc/resolv.conf {\n           force_tcp\n        }\n        cache 30\n        loop\n        reload\n        loadbalance\n    }

The Hardware Reality: Why Virtualization Matters

You can tune sysctls until you are blue in the face, but you cannot software-engineer your way out of bad hardware. This is where the "cloud is a commodity" narrative breaks down. In a containerized environment, packets per second (PPS) is often the bottleneck before bandwidth.

We ran a benchmark comparing a standard shared-core instance from a "Big 3" provider against a CoolVDS NVMe KVM instance. The test used iperf3 between two pods on different nodes.

The difference? Noisy neighbors. On CoolVDS, the KVM virtualization ensures that your CPU cycles for packet processing are actually yours. When your neighbor spins up a video rendering job, your network stack doesn't get starved of CPU time. For applications serving users in Norway, having that predictable low latency to NIX (Norwegian Internet Exchange) is vital.

External Traffic Policy: Preserving Source IP

Another common headache is losing the client's source IP address. By default, Kubernetes performs SNAT (Source Network Address Translation) when traffic enters a node that doesn't host the target pod. This makes your access logs useless for security auditing or geo-blocking.

To fix this, you must set the externalTrafficPolicy to Local. This forces the load balancer to send traffic only to nodes that actually run the pod, preserving the source IP.

apiVersion: v1\nkind: Service\nmetadata:\n  name: nginx-ingress\n  namespace: ingress-nginx\nspec:\n  type: LoadBalancer\n  externalTrafficPolicy: Local\n  ports:\n  - port: 80\n    targetPort: 80\n    protocol: TCP\n  selector:\n    app: nginx-ingress

Warning: This can lead to imbalanced traffic if your pods are not evenly distributed across nodes. However, for a compliant setup in Europe (GDPR requirements often necessitate accurate logging), accurate IP tracking is non-negotiable.

Data Sovereignty and Local Compliance

In 2023, the legal landscape is as complex as the technical one. With the Datatilsynet (Norwegian Data Protection Authority) keeping a close watch on data transfers, hosting your Kubernetes cluster on US-controlled infrastructure is becoming a compliance risk under Schrems II interpretations.

This is where infrastructure choice becomes a legal strategy. By running your control plane and worker nodes on CoolVDS, you ensure data residency remains strictly within the intended jurisdiction. You aren't just getting better I/O performance; you are simplifying your compliance audit.

Final Thoughts: Don't Skimp on the Foundation

Kubernetes is complex enough without fighting the underlying infrastructure. I have seen too many deployments fail because the team assumed a VPS is just a VPS. It isn't. Networking performance relies heavily on the hypervisor's ability to process interrupts quickly.

If you are tired of debugging mysterious timeouts and want a foundation that respects your `sysctl` tuning, it is time to evaluate where your bits actually live. Don't let slow I/O or stolen CPU cycles kill your SEO or your uptime.

Ready to see what raw performance feels like? Deploy a test instance on CoolVDS in 55 seconds and run your own benchmarks.