Stop Waiting for Builds: A DevOps Engineer's Guide to Pipeline Speed

There is nothing quite as soul-crushing as pushing a hotfix on a Friday afternoon and staring at a spinning yellow circle for 25 minutes. We tell ourselves it's a "compiling" break, but deep down, we know it's wasted time. In 2021, with the tools we have available, a slow CI/CD pipeline isn't just an annoyance; it's an infrastructure failure.

I've audited enough pipelines across the Nordics to see the pattern. It's rarely the code. It's almost always the infrastructure underneath the runner. You are trying to build heavy Docker images on oversubscribed, spinning-rust storage in a massive public cloud region halfway across the continent. That latency adds up.

The Hidden Bottleneck: Disk I/O Wait

Most developers focus on CPU. They throw more cores at Jenkins or GitLab Runners. But analyze your build logs. Look at the time spent on npm install, pip install, or extracting Docker layers. That is pure I/O.

If your VPS provider uses shared storage (CEPH or similar network-attached block storage) without strict QoS, your build times fluctuate based on what your neighbors are doing. To fix this, you need dedicated I/O throughput. You need local NVMe.

Benchmarking Your Current Runner

Don't take my word for it. Run fio on your current build server. If you aren't seeing random read/write speeds consistent with NVMe (usually 10k+ IOPS for decent VDS), you are bottlenecking your own deployment.

# Install fio (Debian/Ubuntu)
sudo apt-get update && sudo apt-get install -y fio

# Run a random write test mimicking a heavy build process
fio --name=random-write --ioengine=libaio --rw=randwrite --bs=4k --size=1G --numjobs=1 --iodepth=16 --runtime=60 --time_based --end_fsync=1

On a standard HDD-based VPS, you might see 300-500 IOPS. On a CoolVDS NVMe instance, we consistently benchmark significantly higher because we don't oversell our storage arrays. That difference turns a 5-minute npm install into a 45-second breeze.

Optimizing Docker for Speed

Beyond hardware, configuration matters. If you are running Docker-in-Docker (dind) or binding the socket, the storage driver is critical. Since kernel 4.x, overlay2 is the standard, but I still see legacy systems trying to use devicemapper.

Check your Storage Driver

docker info | grep 'Storage Driver'

If it doesn't say overlay2, you are losing performance. Here is how to force it in /etc/docker/daemon.json:

{
  "storage-driver": "overlay2",
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

Restart Docker: sudo systemctl restart docker.

GitLab Runner: The "Shell" vs "Docker" Debate

For maximum speed, I often recommend moving away from the docker+machine executor for heavy builds and using a static, persistent runner on a high-performance VPS. Why? Cache persistency.

Ephemeral runners are great for isolation, but they start with a cold cache every time. A persistent runner on a CoolVDS instance keeps your /var/lib/docker layers hot. The trade-off is you must manage cleanup scripts, but the speed gain is worth it.

Here is a snippet for a robust config.toml tuned for a 4 vCPU instance:

concurrent = 4
check_interval = 0

[[runners]]
  name = "coolvds-nvme-runner-01"
  url = "https://gitlab.com/"
  token = "YOUR_TOKEN_HERE"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.docker]
    tls_verify = false
    image = "docker:20.10.7"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
    shm_size = 0

Pro Tip: By mounting /var/run/docker.sock, the container uses the host's Docker daemon. This allows you to leverage the host's layer cache directly. Security warning: Only do this on trusted, private runners, not shared public ones.

The Compliance Angle: Schrems II and Data Sovereignty

We need to talk about the elephant in the room: Schrems II. Since the CJEU ruling last year (2020), transferring personal data to US-owned cloud providers has become a legal minefield. If your CI/CD pipeline includes database dumps that contain real user data (for staging tests), and that runner is hosted on a US hyperscaler, you might be non-compliant with GDPR.

Keeping your build infrastructure in Norway isn't just about latency—though 2ms ping to Oslo requires local routing—it's about risk management. The Norwegian Datatilsynet is strict. Hosting your runners on CoolVDS ensures your data stays within the EEA, on infrastructure owned by a European entity. No cloud act, no legal headaches.

Why CoolVDS is the Reference Implementation

You can optimize Dockerfiles and tune configs all day, but you cannot software-patch slow hardware. We built CoolVDS specifically to solve the I/O bottleneck.

• KVM Virtualization: No container neighbors stealing your kernel resources.
• Pure NVMe Storage: We don't tier storage. You get the fast stuff by default.
• Local Peering: If your team is in Oslo or Bergen, the latency to your build server is negligible, making interactive debugging via SSH feel like localhost.

Final Thoughts

If your pipeline takes longer than 10 minutes, you are breaking the feedback loop. Stop accepting slow builds as "normal." Audit your storage drivers, switch to persistent runners for caching, and ensure your underlying hardware isn't fighting you.

Ready to cut your build times in half? Deploy a high-performance runner on CoolVDS today. Use our "NVMe-Turbo" plan to see what your CI/CD is actually capable of.