The Container Orchestration Wars: Kubernetes vs. Mesos vs. Swarm (June 2015 Edition)

Let’s be honest: docker run is the honeymoon phase. We all fell in love with it last year. It works on your laptop, it works on your staging box, and it feels like magic. But then you hit production.

Suddenly, you aren't managing one container; you're managing fifty across three different nodes. You need service discovery, you need health checks, and you need to know why the database container keeps dying at 3 AM. The manual scripts we wrote in 2014 aren't cutting it anymore.

Welcome to the orchestration wars. As of mid-2015, three major contenders are fighting for the soul of your infrastructure: Apache Mesos, Google's Kubernetes, and Docker Swarm. I’ve spent the last month deploying high-load clusters on CoolVDS infrastructure to see which one actually holds up under fire.

1. Apache Mesos + Marathon: The Heavy Artillery

If you are Twitter or Airbnb, you use Mesos. It’s been around longer than the others, and it treats your entire datacenter like a single pool of resources. You don't just run containers; you run frameworks. For Docker, we use the Marathon framework.

The Good

It is rock solid. Mesos has been battle-tested at scales that make most Norwegian startups look like a rounding error. It handles high availability (HA) beautifully, provided you have your Zookeeper quorum configured correctly.

The Bad

It is a beast to configure. You aren't just installing a binary; you are architecting a distributed system. You need Zookeeper nodes, Mesos Masters, Mesos Slaves, and then Marathon on top. The learning curve is vertical.

Pro Tip: When defining your Marathon JSON, always set "upgradeStrategy": {"minimumHealthCapacity": 0.5} for rolling restarts. If you leave it at default, a bad deploy can wipe out 100% of your instances before you blink.

2. Kubernetes: The Google Prodigy (Beta)

Kubernetes (K8s) is the new kid on the block, born from Google's internal Borg system. We are currently looking at version v0.19. It is technically still beta/pre-1.0, but the hype is deafening.

The Good

The concepts are superior. Pods (groups of containers sharing an IP) make so much sense for tight coupling, like a web server and a log shipper. Services and ReplicationControllers provide a level of abstraction that feels "cloud-native" rather than just "scripted."

The Bad

It is moving too fast. The API version changed from v1beta1 to v1beta3 in the blink of an eye, breaking half our manifests. Networking is also notoriously difficult—setting up flannel or weave overlays adds significant overhead if your underlying network is slow.

# A typical v1beta3 Pod definition we are testing
apiVersion: v1beta3
kind: Pod
metadata:
  name: nginx-frontend
  labels:
    app: web
spec:
  containers:
    - name: nginx
      image: nginx:1.7
      ports:
        - containerPort: 80

3. Docker Swarm: The Native Choice

Docker Swarm (standalone) is the simplest approach. It turns a pool of Docker hosts into a single, virtual Docker host. You point your Docker CLI at the Swarm manager, and it schedules containers for you.

The Good

No new tools to learn. If you know the Docker API, you know Swarm. It integrates perfectly with Docker Compose (formerly Fig).

The Bad

It’s very immature compared to Mesos. The scheduling logic is primitive. If you need complex affinity rules (e.g., "don't run two database replicas on the same physical rack"), Swarm struggles. It feels like a tool for developers, not for battle-hardened sysadmins.

The Elephant in the Room: Underlying Infrastructure

Here is the hard truth that orchestration tutorials won't tell you: Orchestration adds overhead.

When you run an overlay network (like flannel for K8s) or a heavy agent (like Mesos slave), you are consuming CPU cycles just to manage the cluster. If you are running this on a budget VPS with "shared vCPUs," your performance will be erratic. I've seen Kubernetes kubelet processes time out simply because a "noisy neighbor" on the host machine decided to compile a kernel.

Why We Use KVM at CoolVDS

This is why we strictly use KVM (Kernel-based Virtual Machine) virtualization at CoolVDS. Unlike OpenVZ, where you share the host's kernel, KVM gives you a dedicated kernel. This is non-negotiable for Docker.

• Isolation: If a container crashes your kernel, it only crashes your VM, not the neighbor's.
• Docker Compatibility: You can tune your own kernel parameters (like net.ipv4.ip_forward) without begging support to do it for you.
• Disk I/O: We use Enterprise SSDs in RAID 10. Container images are comprised of many small files (layers); slow mechanical drives will kill your docker pull times.

Compliance: The Norwegian Context

We are seeing tighter scrutiny from Datatilsynet regarding where data lives. With the uncertainty surrounding the EU Data Protection Directive and the US Safe Harbor agreement, keeping your data inside Norway is the safest bet for legal compliance.

Latency matters too. If your users are in Oslo or Bergen, routing traffic through a cheap VPS in Frankfurt adds 30-40ms of round-trip time. Hosting locally on CoolVDS cuts that to <5ms via NIX (Norwegian Internet Exchange).

Verdict

My advice for June 2015?

If you are building a massive platform today, stick with Mesos/Marathon. It’s painful to set up, but it won’t wake you up at night once it’s running.

If you are forward-looking and willing to deal with breaking changes, start learning Kubernetes. It is the future, but don't bet the farm on it until v1.0 drops.

Whatever you choose, don't cripple it with slow I/O. Spin up a CoolVDS SSD instance in Oslo today, and give your containers the dedicated resources they actually need.