The Packet Drop Is Coming From Inside the Cluster

I still wake up in a cold sweat thinking about a debugging session from 2021. We had a microservices architecture that worked perfectly in staging but fell apart under load. Random 502s. Latency spikes. The culprit wasn't the application code; it was conntrack table exhaustion on the nodes. We were hitting the limits of iptables.

Fast forward to April 2024. If you are still relying on legacy iptables for routing traffic inside Kubernetes, you are choosing to be slow. The landscape has shifted. The "CNI Wars" are largely over, and eBPF is the victor for high-performance setups. But software is only half the battle. You can tune your CNI until you're blue in the face, but if your underlying infrastructure—the actual VPS or metal—has "noisy neighbor" CPU steal, your network stack will suffer.

This is a deep dive into the reality of K8s networking today, focusing on the Nordic infrastructure context.

1. The CNI Choice: Why eBPF is Non-Negotiable in 2024

For years, Flannel and Calico (in iptables mode) were the defaults. They work. But they rely on long chains of rules that the kernel has to traverse for every packet. At scale, this adds measurable latency.

By 2024, Cilium (using eBPF) has become the gold standard for production clusters where performance matters. Instead of context-switching to user space or traversing iptables, eBPF allows us to run sandboxed programs directly in the kernel. It’s faster, safer, and provides observability that iptables can never match.

Deploying Cilium for Performance

Don't just install the defaults. If you are running on CoolVDS KVM instances, you have the kernel privileges to run strict eBPF modes. Here is a production-ready Helm config for April 2024 that enables Hubble (observability) and replaces kube-proxy entirely:

helm install cilium cilium/cilium --version 1.15.1 \
  --namespace kube-system \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost=API_SERVER_IP \
  --set k8sServicePort=6443 \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true \
  --set bpf.masquerade=true \
  --set ipam.mode=kubernetes

Pro Tip: Setting kubeProxyReplacement=true is the critical step. It removes the need for iptables to manage Services, drastically reducing CPU overhead on the node during high connection churn. I've seen this drop softIRQ CPU usage by 40% on high-traffic ingress nodes.

2. The "Flat Network" Lie and Latency

Kubernetes promises a flat network where every pod can talk to every other pod. Under the hood, this is encapsulation (VXLAN, Geneve) or direct routing. Encapsulation adds overhead. It requires the CPU to wrap and unwrap packets.

If you are hosting in Norway, you are likely sensitive to latency. You want your user in Oslo to hit NIX (Norwegian Internet Exchange) and reach your server instantly. If your hosting provider oversubscribes CPU, the time spent encapsulating packets increases.

This is where infrastructure choice becomes architectural. We use CoolVDS for our heavy K8s workloads specifically because the NVMe storage and dedicated CPU cycles prevent the "stutter" often seen in budget VPS providers during packet processing. When the kernel is busy waiting for I/O, it's not forwarding packets.

Benchmarking Network Latency

Don't guess. Measure. Use iperf3 between pods on different nodes to see the real throughput and jitter.

# On Node A (Server)
kubectl run iperf-server --image=networkstatic/iperf3 -- -s

# On Node B (Client)
kubectl run iperf-client --image=networkstatic/iperf3 -- -c <SERVER_POD_IP>

If you see high jitter on a standard VPS, your provider is likely stealing CPU cycles.

3. Security: The default is "Open", which is "Wrong"

In 2024, with the geopolitical situation in Europe and strict GDPR enforcement by Datatilsynet, running a cluster without NetworkPolicy is negligence. By default, a compromised frontend pod can scan your entire database backend.

Here is a "Deny-All" policy you should apply to every namespace immediately after creation, then whitelist traffic as needed. This is the Zero Trust model implemented at the CNI level.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: backend-services
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

Once you apply this, everything stops. Then, you explicitly allow your frontend to talk to your backend. This precision is why we prefer KVM-based virtualization like CoolVDS over shared containers; you want the isolation to be absolute, from the hardware up to the CNI.

4. Gateway API vs. Ingress: The 2024 Standard

The Ingress API has been frozen for a while. As of April 2024, the Gateway API (v1.0 was released late last year) is where the development is happening. It separates the role of the Cluster Operator (who manages the load balancer) from the Developer (who defines routes).

Why switch? Advanced traffic splitting (canary deployments) and header matching are native. You don't need a mess of NGINX annotations anymore.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: web-route
  namespace: default
spec:
  parentRefs:
  - name: external-gateway
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /api/v2
    backendRefs:
    - name: backend-v2
      port: 8080
      weight: 90
    - name: backend-canary
      port: 8080
      weight: 10

This native weighting is cleaner and more portable than provider-specific annotations.

5. Local Compliance and Data Sovereignty

For Norwegian businesses, the Schrems II ruling is still a massive headache in 2024. If your Kubernetes cluster relies on ingress controllers or load balancers that route traffic through US-owned infrastructure, you are creating a compliance risk.

Hosting on CoolVDS in Norway solves a layer of this problem physically. Your data resides on NVMe arrays in local datacenters. But you must ensure your networking configuration doesn't accidentally route traffic externally. configure your CNI to masquerade traffic correctly and ensure your Egress policies block connections to non-EU subnets unless explicitly whitelisted.

Summary: The Stack that Wins

Kubernetes networking is unforgiving. It magnifies the weaknesses of the underlying hardware. If your storage is slow, your etcd cluster degrades, and your network topology updates fail. If your CPU is stolen, packet encapsulation creates latency spikes.

Don't let your infrastructure be the bottleneck for your architecture. If you are building for the Nordic market, you need low latency to Oslo and rock-solid I/O. Spin up a CoolVDS instance today, install Cilium, and see what a network is supposed to feel like.