Kubernetes Networking Deep Dive: Optimizing Latency and Throughput on Linux VPS

The Packet Drop Nightmare You Didn't See Coming

It was 2:00 AM on a Tuesday. Our monitoring dashboard for a client's Magento cluster lit up like a Christmas tree. The database was fine. The CPU usage on the nodes was sitting at a comfortable 40%. Yet, 15% of incoming HTTP requests were timing out. After three hours of digging through tcpdump logs, we found the culprit: conntrack exhaustion combined with a sloppy VXLAN overlay implementation.

Kubernetes is brilliant for orchestration, but out of the box, its networking model assumes you don't care about overhead. If you are running a production workload targeting customers in Norway or the broader EU, you cannot afford the default settings. Every millisecond of latency introduced by packet encapsulation or unnecessary hops is money leaking out of your funnel.

In this deep dive, we are ripping out the defaults. We will look at CNI selection, kernel tuning for high throughput, and why the underlying hardware of your VPS provider—specifically regarding NVMe and CPU isolation—is the hidden variable in your cluster's stability.

1. The CNI Battlefield: Flannel vs. Calico vs. Weave

In 2020, the Container Network Interface (CNI) landscape is crowded, but for serious production workloads on virtualized infrastructure, you have two real choices: Flannel or Calico.

Flannel is the default for many tutorials. It's simple. It sets up a VXLAN overlay and just works. But "just works" doesn't mean "works fast." VXLAN encapsulates packets, adding CPU overhead to every single network call. On a shared VPS where CPU steal time can fluctuate, this is a disaster.

For CoolVDS instances, where we give you dedicated slice performance, we recommend Calico. It can run in pure Layer 3 mode using BGP, avoiding the encapsulation overhead entirely if your network topology allows it, or using IPIP which is generally more performant than VXLAN.

Pro Tip: If you are stuck with VXLAN, you must increase your MTU. The default 1450 bytes often causes fragmentation if your host interface is standard 1500. Check your interface MTU and adjust your CNI config to be Host_MTU - 50.

2. Kernel Tuning: The sysctl Flags You Forgot

Your Linux kernel is likely tuned for a general-purpose desktop or a modest web server, not a Kubernetes node routing thousands of service-to-service packets per second. You need to modify /etc/sysctl.conf on every node in your cluster.

Here is the baseline configuration we deploy on CoolVDS nodes hosting K8s 1.17:

# /etc/sysctl.conf adjustments for K8s High Performance

# Increase the connection tracking table size
net.netfilter.nf_conntrack_max = 131072

# Reuse Timewait connections
net.ipv4.tcp_tw_reuse = 1

# Increase the range of ephemeral ports
net.ipv4.ip_local_port_range = 1024 65535

# Max backlog for packet processing
net.core.netdev_max_backlog = 5000

# Enable IP forwarding (Required for K8s)
net.ipv4.ip_forward = 1

Apply these with sysctl -p. If you skip the nf_conntrack_max setting, your Ingress controller will start dropping packets silently when traffic spikes, leaving no trace in the application logs.

3. Solving the "Hop" Problem with External Traffic Policy

By default, when traffic hits a NodePort or LoadBalancer service in Kubernetes, kube-proxy might route that packet to a pod on a different node. This adds a network hop and obscures the client's source IP address (SNAT).

For latency-sensitive applications serving users in Oslo or Bergen, this extra hop is unnecessary latency. You can force traffic to stay local to the node it landed on using externalTrafficPolicy: Local.

apiVersion: v1
kind: Service
metadata:
  name: nginx-ingress
  namespace: ingress-nginx
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
  ports:
  - name: http
    port: 80
    targetPort: 80
  selector:
    app: ingress-nginx

The Trade-off: This can lead to imbalanced traffic if your load balancer sends traffic to a node that has no relevant pods. However, on a high-performance cluster running on CoolVDS, we typically run DaemonSets for Ingress or ensure strict anti-affinity rules so that pods are spread evenly.

4. The Hardware Reality: NVMe and Etcd

Networking isn't just about cables and switches; it's about how fast your state store can respond. Kubernetes relies on etcd to store cluster state. Every time a service endpoint changes, etcd updates.

If your VPS uses standard spinning rust (HDD) or even cheap SATA SSDs, the fsync latency will throttle etcd. When etcd slows down, the API server slows down. When the API server slows down, kube-proxy updates lag, and your network routing tables become stale.

This is why we standardized on NVMe storage for all CoolVDS instances. In our benchmarks, etcd write latency on our NVMe tiers is consistently under 2ms, whereas standard SSD VPS providers often spike to 10-15ms under load.

5. Local Latency: The Norway Context

If your primary user base is in Norway, hosting in a data center in Virginia or even London is a disservice to your users. The speed of light is a hard limit.

While 25ms sounds fast, it compounds. A typical modern web request involves 30-50 round trips for assets, APIs, and handshakes. That 25ms difference quickly becomes a 1.5-second delay in page load time. By peering directly at NIX (Norwegian Internet Exchange), local hosting ensures your packets take the shortest possible path.

Conclusion: Don't let Default Configs Kill Your Uptime

Kubernetes is powerful, but it is not magic. It requires a solid foundation. You need a CNI that respects CPU cycles, kernel parameters that anticipate high concurrency, and hardware that doesn't choke on I/O.

We built CoolVDS to be the infrastructure we wanted to use: raw, unfiltered performance with NVMe storage and KVM virtualization that doesn't oversubscribe resources. When you are ready to stop debugging network flakes and start shipping code, we are here.

Ready to optimize? Spin up a CoolVDS instance in Oslo today and test your network throughput against the competition.