Kubernetes Networking Deep Dive: Optimizing Throughput on VPS Infrastructure

Most tutorials lie to you. They spin up a minikube instance, deploy an Nginx pod, curl localhost, and call it a day. That is not infrastructure. That is a toy.

If you are reading this in April 2021, you are likely wrestling with the real beast: running a self-managed Kubernetes cluster on VPS or bare metal because you need control, lower costs, or strict data residency compliance in Norway. You aren't on AWS EKS or GKE. You don't have a magical cloud LoadBalancer ready to dispense IPs. You have Linux nodes, a switch, and a headache.

I have spent the last week debugging a cluster that was dropping 3% of packets randomly between nodes. The culprit wasn't the application code; it was a default kube-proxy configuration hitting iptables limits. Let's fix your networking before you hit the same wall.

1. The CNI Decision: Stop Using Flannel for Production

In the early days, Flannel was great. It was simple. But in 2021, if you are running a serious workload, you need more than just a flat overlay network. You need network policies, BGP peering, and encryption.

We strictly recommend Calico for deployments on CoolVDS. Why? Because Calico allows you to use pure Layer 3 routing (no overlay) if your nodes share a subnet, or highly efficient IPIP/VXLAN encapsulation if they don't.

More importantly, with the recent scrutiny from Datatilsynet regarding data privacy, Calico's integration with WireGuard (introduced in v3.15) allows us to encrypt pod-to-pod traffic without the massive overhead of a Service Mesh like Istio.

Installing Calico with Custom MTU

Don't just apply the default manifest. If your underlying VPS network interface has an MTU of 1500, and you use VXLAN, you must account for the 50-byte overhead. If you don't, you get packet fragmentation and terrible performance.

# calico-custom.yaml snippet
kind: ConfigMap
apiVersion: v1
metadata:
  name: calico-config
  namespace: kube-system
data:
  # Typha is needed for clusters > 50 nodes, but good practice anyway
  typha_service_name: "none"
  # Configure the MTU. 1450 is safe for VXLAN on a 1500 MTU host network.
  veth_mtu: "1450"

2. Exposing Services: Life Without Cloud LoadBalancers

On CoolVDS, you control the metal. You don't have an Amazon Elastic Load Balancer. When you set a Service to type: LoadBalancer, it stays in Pending state forever. This is where MetalLB becomes essential.

MetalLB allows your cluster to respond to ARP requests for external IPs. In Layer 2 mode, one node attracts the traffic and spreads it to pods. It is simple, robust, and works perfectly within our Norwegian data centers.

Pro Tip: Avoid using externalTrafficPolicy: Cluster (the default) if you care about preserving client source IPs. It causes extra hops. Use externalTrafficPolicy: Local to force traffic to the node running the pod, preserving the true client IP for your access logs and geo-blocking logic.

Here is a working Layer 2 configuration for MetalLB v0.9.6:

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 192.168.10.240-192.168.10.250  # IPs reserved for your services

3. Kube-Proxy: IPTables vs. IPVS

This is the bottleneck that killed my latency last week. By default, Kubernetes uses iptables to route service traffic. This works fine for 50 services. But if you have 5,000 services, the kernel has to traverse a massive sequential list of rules for every packet.

IPVS (IP Virtual Server) is the solution. It uses hash tables instead of linear lists, offering O(1) complexity. This means your latency stays constant whether you have 10 services or 10,000.

To enable IPVS mode in your cluster (assuming kubeadm init), you must edit your KubeProxyConfiguration:

apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: "ipvs"
ipvs:
  strictARP: true
  scheduler: "rr" # Round Robin

Before applying this, ensure the kernel modules are loaded on your CoolVDS node:

modprobe ip_vs
modprobe ip_vs_rr
modprobe ip_vs_wrr
modprobe ip_vs_sh
lsmod | grep ip_vs

4. Network Policies & GDPR Compliance

Operating in Europe implies strict adherence to data minimization. A flat network where every pod can talk to the database is a security violation waiting to happen.

If you are hosting sensitive customer data in our Oslo facility, you should implement a "Default Deny" policy. This ensures that if an attacker compromises your frontend web server, they cannot port-scan your entire internal network.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: default
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

After applying this, you must explicitly allow traffic. For example, allowing Nginx to talk to your backend API, but nothing else.

5. The Hardware Reality: Why IOPS Matter for Networking

You might wonder why we talk about disk speed in a networking article. In Kubernetes, the state of the cluster is stored in etcd. Every network change, every pod spin-up, every service update is a write to etcd.

If your disk latency is high, etcd slows down. If etcd slows down, the API server hangs. If the API server hangs, network updates (like endpoint slices) don't propagate.

We often see "cheap" VPS providers overselling storage with spinning rust or shared SATA SSDs. When a neighbor runs a heavy backup, your K8s network convergence time spikes from milliseconds to seconds. This is unacceptable for production.

CoolVDS uses exclusively local NVMe storage with high queue depths. We don't throttle IOPS on our premium tiers because we know that for distributed systems like Kubernetes, storage latency is network latency.

Performance Benchmark Check

Run this simple fio test on your current host. If you aren't seeing random write speeds above 100MB/s, your etcd cluster is in danger.

fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test \
--filename=test --bs=4k --iodepth=64 --size=1G --readwrite=randwrite

Final Thoughts

Kubernetes networking is not magic; it is just Linux networking with a lot of YAML on top. By moving from Flannel to Calico, leveraging MetalLB for ingress, and switching to IPVS, you build a foundation that can handle real traffic.

But software optimization only goes so far. You need iron that keeps up. Don't let slow I/O kill your SEO or your uptime.

Ready to build a cluster that doesn't choke? Deploy a high-performance NVMe instance on CoolVDS in Oslo today.