Kubernetes Networking Deep Dive: Optimizing CNI, BGP, and Latency for Nordic Workloads

Let’s be honest: Kubernetes networking is where the abstraction leaks. You can wrap your application in as many YAML manifests as you want, but eventually, a packet has to traverse a physical wire, a virtual switch, an overlay network, and a NAT table. If any link in that chain is weak, your microservices architecture becomes a distributed latency generator.

I recently audited a cluster for a fintech client in Oslo. They were complaining about random timeouts in their payment gateway. The developers blamed the code; the sysadmins blamed the database. The real culprit? A saturated conntrack table on the worker nodes caused by an inefficient CNI configuration combined with noisy-neighbor interference from a budget VPS provider. The packets weren't lost; they were just waiting in line.

In this deep dive, we are ignoring the "hello world" tutorials. We are looking at the networking stack for high-performance production clusters running on June 2022 standards: Kubernetes 1.24, eBPF data planes, and bare-metal optimized VPS.

The Overlay Tax: eBPF vs. Iptables

For years, kube-proxy using iptables was the standard. It works, but it scales linearly with the number of services. If you have 5,000 services, iptables becomes a bottleneck because the kernel has to evaluate rules sequentially. In 2022, if you are building a high-load cluster, relying on iptables is a mistake.

The industry is shifting rapidly toward eBPF (Extended Berkeley Packet Filter). Tools like Cilium allow us to bypass iptables entirely, handling logic inside the kernel. This reduces latency and CPU overhead significantly.

Pro Tip: If you are deploying on CoolVDS NVMe instances, you have the kernel support required for eBPF. Many budget providers run outdated kernels that force you back into legacy iptables mode. Always check uname -r—you want 5.10+ for optimal eBPF features.

Installing Cilium with Helm (The Modern Way)

Forget editing massive YAML files manually. Here is how we deploy Cilium with full eBPF replacement on a fresh K8s 1.24 cluster:

helm repo add cilium https://helm.cilium.io/
helm install cilium cilium/cilium --version 1.11.5 \
  --namespace kube-system \
  --set kubeProxyReplacement=strict \
  --set k8sServiceHost=API_SERVER_IP \
  --set k8sServicePort=6443

The kubeProxyReplacement=strict flag is crucial. It tells Cilium to completely take over service load balancing, removing the iptables bottleneck.

Ingress on Bare Metal: Solving the "Pending External IP"

One of the biggest shocks for engineers moving from AWS or GKE to a VPS provider like CoolVDS is that `LoadBalancer` services stay in `Pending` state. There is no magic cloud controller unless you configure it.

For a robust setup, we use MetalLB. It allows your Kubernetes cluster to answer ARP requests for external IPs, effectively turning your VPS instances into edge routers. This is critical for Nordic businesses needing data to stay within Norway (GDPR compliance) without relying on US-owned hyperscalers.

Configuring MetalLB in Layer 2 Mode

First, install MetalLB:

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.12.1/manifests/namespace.yaml
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.12.1/manifests/metallb.yaml

Next, we map a reserved IP block. On CoolVDS, you can request a secondary IP block routed to your instance. Here is the ConfigMap implementation:

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 185.xxx.xxx.10-185.xxx.xxx.20

This configuration allows your services to claim public IPs instantly. Because CoolVDS provides unmetered internal traffic, communication between nodes (if you are running a cluster across multiple instances) is extremely fast.

Network Latency: The Silent Killer

Latency is physics. If your users are in Oslo, serving them from a datacenter in Frankfurt adds 20-30ms round trip time (RTT) before application processing even begins. Serving them from US East adds 90ms+.

However, latency isn't just about geography; it's about I/O wait. Kubernetes relies heavily on etcd for state. If your disk I/O is slow, etcd writes stall. If etcd stalls, the API server hangs. If the API server hangs, network updates (CNI) don't propagate.

We see this constantly with "shared CPU" hosting. A neighbor spikes their usage, your disk IOPS drop, and suddenly your Pods can't resolve DNS. This is why we emphasize NVMe storage. Let's look at how to verify your disk latency relative to etcd requirements:

# Use fio to test etcd-like write patterns
fio --rw=write --ioengine=sync --fdatasync=1 \
    --directory=test-data --size=22m --bs=2300 \
    --name=mytest

On a standard CoolVDS NVMe instance, you should see fdatasync latencies in the microseconds, not milliseconds. This speed ensures your cluster state remains consistent even during heavy scaling events.

Troubleshooting: When Packets Die

When a pod can't reach a service, `kubectl logs` rarely tells you why. You need to get into the network namespace. In 2022, with the deprecation of Dockershim in Kubernetes 1.24, you might be using `containerd` or `CRI-O`, which changes how you debug.

Instead of relying on legacy tools, use nsenter to jump into a node's networking stack to debug from the host's perspective:

# Find the Process ID of a container
crictl inspect --output go-template --template '{{.info.pid}}' 

# Enter the network namespace
nsenter -t  -n ip route

Debugging DNS Resolution

DNS is the root of 90% of "network" incidents. If CoreDNS is throttling, your application looks offline. Check your resolv.conf inside the pod:

kubectl exec -it  -- cat /etc/resolv.conf

If you see `options ndots:5`, know that your application is making 5 DNS queries for every external lookup before succeeding. This multiplies latency. Optimizing this in your `PodSpec` is an easy win:

dnsConfig:
  options:
    - name: ndots
      value: "2"

Local Sovereignty and NIX

For Norwegian entities, the Schrems II ruling has made data transfer to non-EU providers legally complex. Hosting on a VPS provider physically located in Norway, peered at NIX (Norwegian Internet Exchange), solves two problems:

1. Compliance: Data stays within legal jurisdiction.
2. Speed: Traffic routes locally rather than hairpinning through Sweden or Germany.

CoolVDS infrastructure is designed with this specific topology in mind. We don't just resell capacity; we optimize the routing tables to ensure the shortest path to Norwegian ISPs.

Summary

Building a Kubernetes cluster on bare VPS requires you to understand the stack from the kernel up. You cannot hide behind a managed load balancer. By choosing eBPF over iptables, configuring MetalLB correctly, and ensuring your underlying storage infrastructure (NVMe) can keep up with etcd, you build a system that is resilient, compliant, and incredibly fast.

Don't let poor I/O wait times kill your cluster's performance. Deploy a test environment where the hardware respects your engineering.

Ready to lower your latency? Deploy a high-performance CoolVDS instance in Oslo today and feel the difference NVMe makes for Kubernetes.