Kubernetes Networking is Broken. Here’s How We Fix It.

It’s 3:00 AM on a Tuesday. Your monitoring dashboard—likely Prometheus backed by Thanos—is screaming. The frontend pods in your Oslo cluster can’t reach the payment gateway service. You exec into the pod, run nslookup, and it resolves instantly. You run curl, and it hangs.

It’s not DNS. It’s almost never DNS when the resolution works but the packets die. It’s the abstraction tax.

In 2025, Kubernetes networking has evolved, but the complexity has compounded. We’ve moved from simple iptables rules to complex eBPF maps, yet many DevOps engineers are still treating K8s networking like a black box. If you are running high-traffic workloads in Norway, routing traffic through the NIX (Norwegian Internet Exchange) with suboptimal CNI configurations is burning your budget and your latency budgets.

This is a deep dive into the packet flow. No fluff. Just raw networking.

The Hidden Cost of Encapsulation (VXLAN vs. Direct Routing)

Most managed Kubernetes offerings default to an overlay network using VXLAN or Geneve. This encapsulates your Layer 2 frame inside a Layer 3 UDP packet. It’s easy to set up. It works everywhere.

It’s also a performance vampire.

Every packet leaving a pod needs to be encapsulated, routed, decapsulated, and delivered. This consumes CPU cycles. On a noisy public cloud instance where CPU steal is high, this processing delay causes jitter. In a latency-sensitive environment—say, a fintech app serving users in Bergen—adding 2ms of overhead per request is unacceptable.

Pro Tip: If your underlying network supports it (Layer 2 adjacency), always choose Direct Routing or BGP mode over VXLAN. You eliminate the encapsulation header and the CPU cost of processing it.

The Hardware Variable

We ran a benchmark comparing VXLAN throughput on standard cloud instances versus CoolVDS NVMe instances (which use KVM with optimized virtio-net drivers). The difference wasn't in the throughput; it was in the CPU usage.

When your hypervisor isn't oversubscribing resources, your CNI has the breathing room to process packets without queuing. This is why we insist on KVM and dedicated resources at CoolVDS.

Killing iptables: The Shift to eBPF

By 2025, if you are still using `kube-proxy` in iptables mode for a cluster with more than 500 services, you are doing it wrong. Iptables is a linear list. Updating it is O(N). Matching rules is O(N). At scale, rule updates can take minutes, leaving your cluster in an inconsistent state.

The standard now is Cilium (or Calico with eBPF enabled). eBPF allows us to run sandboxed programs in the Linux kernel without changing kernel source code. It replaces the iptables spaghetti with hash maps (O(1) lookups).

Here is a production-grade Helm configuration for Cilium that we use for high-performance clusters in Northern Europe:

helm install cilium cilium/cilium --version 1.16.0 \
  --namespace kube-system \
  --set kubeProxyReplacement=strict \
  --set k8sServiceHost=${API_SERVER_IP} \
  --set k8sServicePort=${API_SERVER_PORT} \
  --set bpf.masquerade=true \
  --set routingMode=native \
  --set autoDirectNodeRoutes=true \
  --set ipv4.nativeRoutingCIDR=10.0.0.0/8 \
  --set loadBalancer.mode=dsr  # Direct Server Return for massive throughput

Note the loadBalancer.mode=dsr. Direct Server Return means the return traffic from the pod goes directly to the client, bypassing the load balancer node entirely. This cuts your bandwidth usage on the ingress node by half.

Debugging When Packets Die

When things break, `kubectl logs` won't save you. You need to see the wire. However, running `tcpdump` inside a container is often restricted.

We use `pwru` (Packet, Where Are You?) or Cilium's `hubble`. But sometimes, you need to go old school. Here is how to verify if MTU fragmentation is killing your connections (common in Norway when traversing different ISPs to reach data centers in Oslo):

# Don't just ping. Ping with the "Do Not Fragment" bit set.
# Start with 1472 bytes (1500 - 28 bytes IP/ICMP header)
ping -M do -s 1472 10.244.0.5

# If that fails, lower the size until it passes.
# If you find it passes at 1422, your MTU is 1450.
# You must configure your CNI MTU to match underlying infrastructure.

If you are hosting on CoolVDS, our infrastructure supports Jumbo Frames (MTU 9000) on the private network, which significantly increases throughput for database replication (e.g., PostgreSQL streaming replication) between pods.

The Gateway API Revolution

We finally stopped using the fragmented `Ingress` annotations in late 2024. The Gateway API is the mature standard in 2025. It separates the role of the infrastructure provider (CoolVDS/Platform Ops) from the role of the developer.

Here is how you expose a service properly today, avoiding the Nginx config hell:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: my-app-route
  namespace: default
spec:
  parentRefs:
  - name: external-gateway
    namespace: infra
  hostnames:
  - "api.coolvds-demo.no"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /v2
    backendRefs:
    - name: my-service-v2
      port: 8080
      weight: 90
    - name: my-service-canary
      port: 8080
      weight: 10

This declarative traffic splitting is native. No more lua scripts injected into ingress controllers.

Kernel Tuning for Heavy Loads

Your Kubernetes nodes are still Linux servers. The defaults are set for general usage, not for handling 50k concurrent connections. If you deploy a cluster on CoolVDS without tuning `sysctl`, you are driving a Ferrari in first gear.

Apply this via a DaemonSet or Cloud-Init to every node:

# /etc/sysctl.d/99-k8s-network.conf

# Increase the range of ephemeral ports
net.ipv4.ip_local_port_range = 1024 65535

# Allow reuse of sockets in TIME_WAIT state for new connections
net.ipv4.tcp_tw_reuse = 1

# Max backlog of connection requests (crucial for burst traffic)
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 8192

# Increase BPF JIT limit for Cilium
net.core.bpf_jit_limit = 1000000000

Why Infrastructure Matters

You can have the most optimized eBPF code in the world, but if your underlying hypervisor pauses your VM for 200ms to service a neighbor, your P99 latency is ruined. Network processing is CPU-bound.

At CoolVDS, we don't play the "burst CPU" game with production workloads. Our NVMe storage arrays ensure that when your Kafka pods flush to disk, the I/O wait doesn't block the network stack. In the Norwegian market, where data sovereignty and speed are paramount (damn, I said I wouldn't use that word—where speed is critical), owning your packet path is the only way to guarantee stability.

Final Thoughts

Kubernetes networking in 2025 is about visibility and efficiency. Move to eBPF. Use Gateway API. And ensure your underlying metal respects your packets.

Don't let legacy virtualization kill your cluster's performance. Spin up a CoolVDS instance in Oslo today, apply the sysctl tunings above, and watch your latency drop.