Surviving the Container Wars: Swarm vs. Kubernetes vs. Mesos

It works on your laptop. Great. Now try keeping that container alive when your traffic spikes to 5,000 requests per second and your database node just kernel panicked. If you are still SSH-ing into servers to run docker run -d -p 80:80 nginx, you are doing it wrong. We are in 2016, and the "works on my machine" excuse doesn't fly in production.

The container ecosystem has exploded. Just a few years ago, we were happy hacking LXC. Now, we are drowning in choices for orchestration. The big three contenders right now are Docker Swarm, Kubernetes, and Apache Mesos (with Marathon). I have spent the last month deploying all three on our infrastructure to see which one actually handles the load without collapsing under its own complexity.

The Elephant in the Server Room: Why Orchestration Breaks Cheap VPS

Before we look at the software, let's talk about the hardware. Orchestration tools are chatty. They constantly health-check, schedule, and move workloads. If you run a Kubernetes cluster on a budget VPS with spinning rust (HDD) storage, your cluster state database—etcd—will hit I/O wait bottlenecks. When etcd chokes, your entire cluster loses its mind.

War Story: Last October, we tried running a Mesos cluster on a standard cloud provider in Frankfurt. The "Steal Time" (CPU cycles stolen by the hypervisor for other tenants) hit 15%. The result? Marathon thought perfectly healthy nodes were dead and started rescheduling containers in a loop. It took down the production API for 45 minutes.

This is why we standardized our internal benchmarks on CoolVDS. You need dedicated resources. Specifically, you need the low latency of NVMe storage to keep etcd or ZooKeeper happy. If the disk latency spikes above 10ms, you are going to have a bad time.

Contender 1: Docker Swarm (The Native Choice)

Docker Swarm is the path of least resistance. Since it uses the standard Docker API, any tool that talks to Docker (like Jenkins or Docker Compose) can talk to Swarm. It treats a cluster of Docker hosts as a single, virtual host.

The Good: Setup is ridiculously fast. You don't need to learn a new CLI.

The Bad: It's less mature regarding high availability compared to the others. It relies heavily on the discovery backend.

Here is how we bootstrap a quick cluster. Note that we are using the token discovery for simplicity, but for production, you should use Consul or etcd.

# On the manager node
docker run swarm create
# Returns token: 52a114...

# Join the agent nodes (run this on your CoolVDS workers)
docker run -d swarm join --addr=10.0.0.2:2375 token://52a114...

# Start the manager
docker run -d -p 2375:2375 swarm manage token://52a114...

Simple? Yes. But when you need complex rescheduling policies or persistent storage claims, Swarm currently feels a bit thin compared to K8s.

Contender 2: Kubernetes (The Google Juggernaut)

Kubernetes (K8s) is the new heavyweight champion, reaching v1.1 recently. It is opinionated. It is complex. It brings concepts like Pods, Services, and Replication Controllers that force you to rethink your architecture. But it is also incredibly robust.

The main pain point in early 2016? Installation. There is no single binary that "just works" everywhere yet. We are manually configuring systemd units for kube-apiserver, kube-controller-manager, kube-scheduler, and kubelet.

Here is a snippet of a Kubernetes ReplicationController we use to keep Nginx alive. This guarantees that 3 copies are always running, even if a node dies.

apiVersion: v1
kind: ReplicationController
metadata:
  name: nginx-controller
spec:
  replicas: 3
  selector:
    app: nginx
  template:
    metadata:
      name: nginx
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.9
        ports:
        - containerPort: 80

If you are serious about DevOps, learn Kubernetes. It is winning the mindshare war. Just make sure your underlying network is solid. K8s relies heavily on overlay networks (like Flannel or Weave). On a congested network, packet encapsulation adds overhead. We tested this between two CoolVDS instances in Oslo: the latency penalty was negligible due to the high-bandwidth backbone, but on public internet-facing nodes, it can be significant.

Contender 3: Apache Mesos + Marathon

Mesos is the old guard. It abstracts CPU, memory, storage, and other compute resources away from machines, enabling fault-tolerant and elastic distributed systems. It's what Twitter and Airbnb run.

However, for a medium-sized shop in Norway, it is likely overkill. It requires a ZooKeeper cluster, Mesos Masters, Mesos Slaves, and then a framework like Marathon on top to actually run containers. The operational overhead is massive.

The Data Residency Elephant: Safe Harbor is Dead

We cannot ignore the legal landscape. In October 2015, the ECJ invalidated the Safe Harbor agreement. If you are a Norwegian company storing customer data on US-controlled servers (even if they have a data center in Dublin), you are in a legal grey area right now. The privacy activists are not sleeping.

This is a purely pragmatic reason to host locally. Keeping your orchestration cluster on Norwegian soil (like CoolVDS's Oslo zone) simplifies compliance with the Data Protection Directive. You don't have to worry about the NSA peeking at your `secrets.yaml`.

Optimizing the Node: The Kernel Tuning

Regardless of which orchestrator you pick, the default Linux kernel settings on most distributions (Debian 8, CentOS 7) are not tuned for high-density container traffic. You will hit limits on open files and ARP caches.

Add this to your /etc/sysctl.conf on every worker node before joining it to the cluster:

# Increase the number of connections
net.core.somaxconn = 4096

# Allow more port reuse
net.ipv4.tcp_tw_reuse = 1

# Increase ARP cache size (crucial for K8s Service discovery)
net.ipv4.neigh.default.gc_thresh1 = 1024
net.ipv4.neigh.default.gc_thresh2 = 2048
net.ipv4.neigh.default.gc_thresh3 = 4096

# Docker specific: Enable IP forwarding
net.ipv4.ip_forward = 1

Apply with sysctl -p.

Verdict: What should you use in 2016?

If you are a small team in Oslo just wanting to push code, start with Docker Swarm. It is built into the tools you already use.

If you are building a platform that needs to last the next 5 years, bite the bullet and learn Kubernetes. The ecosystem is growing faster than anything I have seen since the early Linux days.

But remember: software cannot fix broken hardware. Latency kills distributed systems. Don't let slow I/O kill your SEO or your uptime. Ensure your nodes are running on pure NVMe storage with guaranteed CPU cycles.

Ready to build your cluster? Deploy a high-performance KVM instance on CoolVDS in 55 seconds and stop fighting with noisy neighbors.