The Packet Drop is Coming from Inside the Cluster

Most developers treat Kubernetes networking as magic. You define a Service, the packet arrives, everyone goes home happy. Until they don't. Until your latency spikes to 400ms on a simple REST call between microservices, or your database connections start timing out during peak load. If you are reading this in February 2020, you know exactly what I am talking about. The abstraction leakage in K8s networking is real, and it is usually painful.

I recently audited a setup for a client in Oslo. They were running a decent sized cluster on a generic European cloud provider. Their complaint? "Random slowness." The culprit wasn't their code; it was the network overlay. They were double-encapsulating packets inside a noisy neighbor environment.

If you care about performance, you need to understand what is happening at the CNI (Container Network Interface) layer. Let's rip open the hood.

The Overlay Tax: VXLAN vs. BGP

By default, many K8s installers (like `kubeadm` or Rancher) might push you toward Flannel using VXLAN. It is simple. It works everywhere. But it adds overhead. Every packet leaving a pod is wrapped in a UDP packet (encapsulated), sent to the destination node, and unwrapped (decapsulated). This burns CPU cycles and reduces the Maximum Transmission Unit (MTU) available to your application.

On a dedicated KVM slice—like what we provide at CoolVDS—you have access to the raw networking performance of the underlying hardware. Why waste it on encapsulation if you don't have to?

Enter Calico and BGP

For production workloads in 2020, I almost exclusively recommend Calico in BGP mode. Instead of wrapping packets, Calico turns your nodes into virtual routers. They share routing tables. The packet goes from Pod A to Pod B without encapsulation logic eating your CPU.

Here is a standard configuration snippet to apply Calico v3.11 (current stable) manifest:

However, simply applying the YAML isn't enough. You need to ensure your `CALICO_IPV4POOL_IPIP` is set to `"Never"` if your infrastructure supports it (Layer 2 adjacency), or `"CrossSubnet"` if you are spanning availability zones.

- name: CALICO_IPV4POOL_IPIP
  value: "Never" # Disable IPIP encapsulation for raw performance

The Silent Killer: MTU Mismatches

This is where 90% of networking issues originate. The standard Ethernet MTU is 1500 bytes. If you use an overlay like VXLAN, the wrapping header takes 50 bytes. Your inner MTU becomes 1450. If your application tries to force a 1500-byte packet through, it gets fragmented or dropped.

We see this constantly with clients migrating from bare metal to virtualized K8s. The symptoms are subtle: TLS handshakes hang, or SSH freezes, but ICMP (Ping) works fine.

Pro Tip: Always clamp your MSS (Maximum Segment Size). If you can't control the underlay network, configure Calico to auto-detect the MTU.

To verify your current MTU settings across nodes:

kubectl get nodes -o jsonpath='{range .items[*]}Node: {.metadata.name} | Interface: {.status.addresses[?(@.type=="InternalIP")].address}{"\n"}{end}'

# Then ssh into a node and check the actual interface
ip link show eth0 | grep mtu

At CoolVDS, our network core in Oslo is configured to support standard MTU sizing seamlessly, but we still see users misconfigure their CNI. If you are seeing drops, perform a sweep ping to find the breaking point:

ping -M do -s 1472 8.8.8.8 
# 1472 + 28 bytes header = 1500. If this fails, lower the size until it passes.

Kube-Proxy: IPVS vs. Iptables

Until recently, the default mode for `kube-proxy` was `iptables`. In a small cluster, this is fine. But `iptables` rules are processed sequentially. If you have 5,000 services, the kernel has to traverse a massive list of rules for every packet.

In 2020, you should be looking at IPVS (IP Virtual Server) mode. It uses hash tables instead of linear lists. The lookup time is constant, O(1), regardless of how many services you run.

To enable IPVS in a `kubeadm` config:

apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: "ipvs"

Before doing this, ensure the kernel modules are loaded on your underlying VPS:

modprobe ip_vs
modprobe ip_vs_rr
modprobe ip_vs_wrr
modprobe ip_vs_sh
lsmod | grep ip_vs

We include these kernel modules by default in our standard Ubuntu 18.04 and CentOS 7 images at CoolVDS because we know our clients run high-load clusters.

Latency and Geography: The Norway Advantage

Let's talk about physics. You can optimize your CNI configuration all day, but if your server is in Frankfurt and your users are in Oslo, you are fighting a losing battle against the speed of light (and network hops).

Latency to the NIX (Norwegian Internet Exchange) is critical for local businesses. A round trip from Oslo to a generic cloud provider in Central Europe can take 25-40ms. Within Oslo? It's sub-2ms.

For databases relying on synchronous replication (like Galera Cluster or etcd for K8s itself), high latency increases commit times and lowers write throughput. By keeping your control plane and data nodes in Norway, you reduce `etcd` peer latency, which stabilizes the entire cluster.

Data Sovereignty (GDPR)

While we await further clarity on international data transfers (the privacy landscape is volatile right now), keeping data within Norwegian borders satisfies the strictest interpretations of the Datatilsynet guidelines. Hosting on US-owned infrastructure, even if located in Europe, is becoming a gray area for legal teams. Local infrastructure is the safest bet.

Architecture Recommendation

If you are deploying Kubernetes in 2020, here is your reference architecture for high performance:

1. Infrastructure: KVM-based VPS (avoid shared containers). NVMe storage is mandatory for etcd performance.
2. OS: Ubuntu 18.04 LTS (Kernel 4.15+).
3. CNI: Calico in BGP mode (disable IPIP).
4. Proxy: IPVS mode.
5. Ingress: Nginx Ingress Controller (host network mode for maximum throughput).

We built the CoolVDS platform to serve exactly this stack. We provide the pure KVM isolation needed to run BGP routing without the "noisy neighbor" packet loss common in oversold clouds.

Don't let your network be the bottleneck. Check your MTU, switch to IPVS, and put your bits closer to your users.

Ready to lower your latency? Deploy a high-performance NVMe KVM instance in Oslo today with CoolVDS.