Stop Blaming the Code for Slow Builds

It’s 2019. We are deploying microservices, we are embracing Kubernetes (v1.14 just dropped, finally stable-ish), and yet, I still see teams treating their CI/CD infrastructure like it's a shared hosting plan from 2010. I recently audited a setup for a fintech client in Oslo. Their developers were complaining that `npm install` took five minutes. They blamed the `package.json`. They blamed the registry.

I blamed the disk. I was right.

When you run concurrent Docker builds, you aren't just eating CPU. You are hammering the storage subsystem with random reads and writes. If your VPS provider is putting you on standard SSDs (or heaven forbid, spinning rust) with noisy neighbors, your build pipeline is doomed to stall in iowait hell. Here is how to fix it, specifically tailored for the Norwegian infrastructure landscape.

The Bottleneck You Can't Refactor: Disk I/O

Let’s get technical. A typical CI job involves pulling images, extracting layers, compiling binaries, and writing artifacts. This is I/O intensive. If you are using Jenkins or GitLab CI runners, the default configurations often choke on I/O operations per second (IOPS).

I ran a benchmark comparing a standard cloud instance against a dedicated NVMe slice (similar to our CoolVDS Performance tier). The difference isn't percentage points; it's orders of magnitude.

Pro Tip: If your load average is high but CPU usage is low, run top and look at the wa (wait) percentage. If it's over 10%, your disk is the bottleneck.

To verify this on your current runner, use iotop:

sudo iotop -oPa

If you see jbd2/sda1 or your Docker daemon process sticking to the top with 99% I/O, you need better hardware. This is why CoolVDS moved strictly to local NVMe storage for our CI/CD focused instances. Network-attached storage (NAS/SAN) introduces latency that kills the small-file performance required by node_modules or heavy compilation tasks.

Optimizing the Docker Daemon for Speed

By May 2019, the overlay2 storage driver is the standard, but I still see legacy `devicemapper` configs floating around. Ensure your underlying host is using overlay2 for the Docker daemon. It is faster and more inode-efficient.

Check your current driver:

docker info | grep Storage

If it doesn't say overlay2, you need to update your /etc/docker/daemon.json immediately:

{
  "storage-driver": "overlay2",
  "storage-opts": [
    "overlay2.override_kernel_check=true"
  ],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

Note: We limit log sizes here. I've seen CI servers crash because a runaway build process filled the disk with 50GB of logs. Don't be that guy.

Kernel Tuning for High-Concurrency Runners

When you spin up 20 containers for parallel testing, you hit other limits. The Linux kernel defaults are conservative. On a CoolVDS instance running Ubuntu 18.04 LTS, I always apply the following sysctl tweaks to handle the network stack thrashing that happens during heavy git clone and artifact upload operations.

Edit /etc/sysctl.conf:

# Increase the range of ephemeral ports for high-concurrency connections
net.ipv4.ip_local_port_range = 1024 65535

# Reuse specific TCP connections (careful with this one behind NAT, but okay for internal runners)
net.ipv4.tcp_tw_reuse = 1

# Increase max open files for the kernel
fs.file-max = 2097152

# Increase virtual memory areas for Elasticsearch/Java based tests
vm.max_map_count = 262144

Apply them with sudo sysctl -p. If you don't raise fs.file-max, your Jenkins job will fail with