Kubernetes Networking Deep Dive: Surviving the Packet Jungle

Let’s be honest. Kubernetes networking is where the abstraction leaks. You can define all the Deployment yamls you want, but the moment a packet drops between a frontend pod and your Redis slave, you aren't a Platform Engineer anymore—you are a network plumber digging through iptables rules generated by a machine. I've seen entire clusters in production environments grind to a halt not because of CPU starvation, but because of TCP retransmissions caused by a misconfigured MTU on the overlay network.

If you are deploying Kubernetes in 2021, specifically targeting the Norwegian market or the broader European sector, you are dealing with two distinct beasts: the technical complexity of CNI (Container Network Interface) plugins and the legal/physical reality of data sovereignty and latency. With the Schrems II ruling from last year still causing headaches for legal teams, hosting your cluster on US-controlled infrastructure is a risk many CTOs are trying to offload. But moving to local VPS providers requires you to understand what happens under the Kubernetes hood.

The Overlay Tax: VXLAN vs. BGP

By default, many manage Kubernetes installers slap Flannel with VXLAN on your cluster. It works. It's simple. But it encapsulates your packets inside UDP packets. This adds headers, reduces the effective MTU (Maximum Transmission Unit), and burns CPU cycles for encapsulation/decapsulation on every single hop.

If you are running on high-performance infrastructure like CoolVDS NVMe instances, you want raw speed. You don't want your CPU stealing cycles to wrap packets. This is why I almost exclusively recommend Calico with BGP (Border Gateway Protocol) peering if your provider supports layer 2 adjacency, or at least IPIP mode if they don't.

Here is a snippet from a standard calico.yaml configuration I use to ensure we are detecting the correct interface on a VPS where the public IP might be bound differently:

- name: IP_AUTODETECTION_METHOD
  value: "interface=eth0"
- name: CALICO_IPV4POOL_IPIP
  value: "Always"
- name: FELIX_IPINIPMTU
  value: "1480"

Pro Tip: Always manually set your MTU. If the physical interface is 1500, set your overlay to 1480 (for IPIP) or 1450 (for VXLAN). If you don't, large packets will fragment, and your performance will plummet silently. We see this constantly on unmanaged VPS providers. CoolVDS standardizes underlying MTU, but your CNI config must match it.

Ingress Controllers: NGINX is Still King

While Traefik v2 is fancy with its CRDs and middleware, the ingress-nginx controller (the official Kubernetes one, not the F5 one) remains the battle-standard for 2021. It is essentially a wrapper around NGINX, which means we can tune it using the sysctl knowledge we've had for a decade.

In a recent project serving a high-traffic media site in Oslo, we noticed sporadic 502 errors during traffic spikes. The cause? Ephemeral port exhaustion on the ingress controller. The solution wasn't more pods; it was kernel tuning via sysctl and NGINX configuration.

Here is how you inject the necessary optimizations into your Ingress Controller ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: ingress-nginx-controller
  namespace: ingress-nginx
data:
  worker-processes: "4"
  max-worker-connections: "65536"
  keep-alive: "60"
  upstream-keepalive-connections: "100"
  compute-full-forwarded-for: "true"
  use-forwarded-headers: "true"

And specifically, to handle the high throughput allowed by NVMe-backed storage when serving static assets through the ingress, you need to adjust the kernel parameters on the node itself. On CoolVDS KVM slices, you have full kernel control, so you should apply this via a DaemonSet or your provisioning script:

# /etc/sysctl.d/k8s-tuning.conf
net.core.somaxconn = 32768
net.core.netdev_max_backlog = 16384
net.ipv4.tcp_max_syn_backlog = 8096
net.ipv4.ip_local_port_range = 1024 65535

The Latency Factor: Oslo and the NIX

Why does geography matter in networking? Speed of light. If your users are in Norway, but your K8s cluster is in Frankfurt, you are adding 20-30ms of round-trip time (RTT) to every packet. For a modern SPA (Single Page Application) that makes 50 API calls to render a dashboard, that latency stacks up. It kills the user experience (UX).

Hosting in Norway, connected to NIX (Norwegian Internet Exchange), keeps that latency under 5ms. However, local hosting often scares people who fear "unmanaged" hardware. This is where the choice of VPS matters. You need a provider that guarantees isolation.

We use CoolVDS for our Kubernetes nodes because they provide KVM (Kernel-based Virtual Machine) virtualization. Unlike OpenVZ or LXC containers used by budget hosts, KVM prevents "noisy neighbors" from stealing your network I/O. When you are running a distributed system like K8s, inconsistent network latency between nodes (jitter) can cause leader election failures in Etcd.

Etcd Sensitivity

Speaking of Etcd, it is the brain of your cluster. It requires low latency storage to persist state. If your disk fsync takes too long, Etcd assumes the node is down. This is why NVMe storage is non-negotiable for control plane nodes in 2021.

Security: Network Policies

By default, Kubernetes is a flat network. Any pod can talk to any pod. If an attacker breaches your frontend, they can scan your database directly. This is negligence.

You must implement NetworkPolicies. If you are using Calico, this is built-in. Here is a "Default Deny" policy that every production namespace should have. It forces you to explicitly allow traffic.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

Once this is applied, silence reigns. You then poke holes for specific traffic:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-to-backend
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - protocol: TCP
      port: 8080

Debugging: When it breaks

It will break. When a service is unreachable, don't guess. Use tcpdump. Since most container images are stripped of tools (distroless is the trend in 2021), you often need to debug from the node or use a sidecar.

If you have node access (which you do on a VPS), find the virtual interface for the pod and dump the traffic:

# On the node
# 1. Find the cali/veth interface
ip route | grep 

# 2. Dump traffic
tcpdump -i cali12345678 -nn -vv port 80

If you see packets leaving the interface but never arriving at the destination, check your security groups or iptables trace.

Conclusion

Kubernetes networking is powerful, but it relies heavily on the quality of the underlying network. You can configure Calico perfectly, but if the physical switch gets congested or the hypervisor is overcommitted, your cluster suffers.

For critical workloads in the Nordic region, control your stack. Move away from opaque cloud load balancers and run your ingress on dedicated resources. We've found that combining the raw I/O performance of CoolVDS NVMe instances with a tuned BGP networking stack offers the best price-to-performance ratio currently available.

Don't let latency kill your application. Deploy a 3-node K8s cluster on CoolVDS today and see the difference single-digit latency makes.