Stop Blaming the Code: Your Kubernetes Network is the Bottleneck

If I had a nickel for every time a developer told me their microservice is optimized, yet the request latency graph looks like a heartbeat during a cardiac arrest, I'd own a private island. In 2018, deploying Kubernetes (K8s) is easy. kubeadm init and you're done, right? Wrong. The default networking stack in Kubernetes is not designed for scale. It is designed for compatibility.

We are seeing this constantly with teams migrating to container orchestration here in Europe. They deploy a cluster, scale up to 500 services, and suddenly CPU usage on the nodes spikes even with zero traffic. Why? Because iptables was never meant to be a load balancer. If you are serious about running high-load workloads, you need to stop treating the network like a black box.

This is a deep dive into the guts of K8s networking as of late 2018. We are looking at the transition to IPVS, choosing the right CNI, and why the physical location of your VPS in Norway actually impacts your packet drops.

The Linear Nightmare: iptables vs. IPVS

For years, Kubernetes relied on iptables to handle Service discovery and load balancing. When you create a Service, K8s creates iptables rules to NAT the traffic to the pods. This works fine for small clusters.

Here is the problem: iptables rule processing is sequential. It is an O(n) algorithm. Every packet has to traverse the list of rules until it finds a match.

Pro Tip: Run iptables-save | wc -l on one of your worker nodes. If that number is over 20,000, your latency issues are caused by the kernel filtering packets, not your application code.

In a recent project for a media streaming client here in Oslo, we hit a wall. We had roughly 3,000 services. The iptables utility became the bottleneck. Updates to the service list took minutes to propagate because the kernel lock (XT locks) stopped everything else while the tables were rewritten.

Enter IPVS (IP Virtual Server)

As of Kubernetes 1.11 (released mid-2018), IPVS is finally a stable option for kube-proxy. Unlike iptables, IPVS uses hash tables. The lookup complexity is O(1). It doesn't matter if you have 10 services or 10,000; the performance remains constant.

Here is how you enable it. You need to modify your kube-proxy ConfigMap or the flags passed to the daemon.

By switching to IPVS, we saw a 30% drop in CPU overhead on the nodes and a stabilization of p99 latency. If you are running on CoolVDS, our KVM images come with the necessary kernel modules (ip_vs, ip_vs_rr, ip_vs_wrr, ip_vs_sh) pre-enabled, so you don't have to recompile the kernel yourself.

CNI Wars: Flannel vs. Calico

The Container Network Interface (CNI) is where the real religion starts. In 2018, the two main contenders for general use are Flannel and Calico.

The Case Against Encapsulation (VXLAN)

Flannel is the default for many. It usually uses VXLAN to create an overlay network. This encapsulates Layer 2 frames inside Layer 4 UDP packets. It is simple. It works everywhere.

It is also slow.

Encapsulation adds CPU overhead for wrapping and unwrapping packets. It also reduces the Maximum Transmission Unit (MTU), leading to fragmentation if not configured perfectly.

For performance-critical environments, we prefer Calico in BGP mode. Calico routes packets natively without encapsulation (if your network supports it). It turns your nodes into virtual routers.

When running Calico on a VPS, you must ensure the underlying network allows the BGP traffic. At CoolVDS, we don't block the internal traffic between your private IP instances, allowing Calico to establish peering sessions seamlessly.

The Physical Layer: Why "VPS Norway" Matters

You can optimize your Kubernetes networking config all day, but you cannot code your way out of physics. Latency is defined by distance and hops.

If your target audience is in Scandinavia, hosting your cluster in Frankfurt or London adds unnecessary milliseconds. But there is a legal angle too. With GDPR in full effect since May, and the strict oversight of Datatilsynet, data sovereignty is critical.

When you use a US-centric provider, your data often traverses network paths you don't control. By utilizing a local provider, you reduce the hops to the Norwegian Internet Exchange (NIX). Low latency isn't just about speed; it's about the consistency of TCP handshakes.

Etcd Needs NVMe

Kubernetes state is stored in etcd. Etcd is extremely sensitive to disk write latency. If fsync takes too long, the cluster becomes unstable and leader election fails.

Do not run a production Kubernetes cluster on standard spinning HDD or cheap SATA SSDs. You need NVMe storage. We have seen clusters implode simply because a "noisy neighbor" on a shared host saturated the disk I/O.

Here is a quick check you can run to test if your current VPS storage is fast enough for etcd:

If you aren't seeing low latency on these writes, your API server will be sluggish.

Ingress Tuning: The Front Door

Finally, let's talk about getting traffic into the cluster. The Nginx Ingress Controller is the standard workhorse. However, the default config is conservative.

If you are handling high concurrency, you need to tune the worker processes and open file limits. Here is a snippet from a ConfigMap we use for high-traffic sites:

Combining IPVS for internal routing, Calico for flat networking, and NVMe storage for etcd stability creates a Kubernetes cluster that is actually ready for production. Don't let default configurations kill your performance.

Ready to build a cluster that doesn't melt? Deploy a high-performance NVMe VPS in Norway with CoolVDS today. We handle the infrastructure; you handle the code.