Your 45-Minute Build Time is an Infrastructure Failure

I recently audited a Oslo-based fintech startup. Their lead developer complained that deploying a simple hotfix took nearly an hour. The team assumed their code was bloated. They were wrong. The code was fine. Their infrastructure was choking.

They were running Jenkins on a budget VPS provider hosted somewhere in Frankfurt, sharing spindle-based storage with five hundred other noisy neighbors. Every time npm install ran, the disk I/O latency spiked to 500ms. The CPU was waiting, not working.

In the DevOps world of late 2022, where microservices and containerization are the standard, Disk I/O is the new CPU. If you are running CI/CD runners on standard SSDs—or worse, HDD-backed storage—you are burning engineering hours. Here is how we fix it, keeping Norwegian data sovereignty and raw performance in mind.

1. Diagnosing the "Noisy Neighbor" Syndrome

Before you blame Webpack or Maven, check if your server is actually doing the work you paid for. On a shared environment, 'Steal Time' (st) is the silent killer. It happens when the hypervisor forces your VM to wait while another VM uses the physical CPU.

Run this during your next pipeline execution:

top -b -n 1 | grep "Cpu(s)"

Look at the st value. If it's above 0.0% consistently, move. You cannot tune your way out of a crowded hypervisor.

However, the more common bottleneck in CI/CD is I/O Wait (wa). When building Docker images, you are essentially hammering the filesystem with thousands of small read/write operations (layers). Run iostat (part of the sysstat package) to see the truth:

# Install sysstat if missing
apt-get update && apt-get install sysstat -y

# Monitor disk stats every 2 seconds
iostat -xz 2

If your %util is near 100% but your write speeds (w/s) are low, your storage medium effectively gave up. This is where the hardware matters. At CoolVDS, we use NVMe storage exclusively because the IOPS capability is roughly 6-10x higher than standard SATA SSDs. For a heavy build pipeline involving database seeding or compilation, this reduces build times by 40-60% instantly.

2. The Docker Storage Driver Trap

I still see servers in production using devicemapper. Stop it. In 2022, the only acceptable storage driver for Linux distributions is overlay2. It is faster and uses inode-efficient layering.

Verify your Docker configuration:

docker info | grep "Storage Driver"

If it doesn't say overlay2, you need to update your /etc/docker/daemon.json immediately. While you are there, if you are pulling images frequently from Docker Hub, you might be hitting rate limits. Configure a local registry mirror if you have multiple runners.

{
  "storage-driver": "overlay2",
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

3. Optimizing GitLab Runner for Concurrency

Many Norwegian dev teams use GitLab CI (often self-hosted to comply with strict Datatilsynet data residency requirements). The default configuration is often too conservative.

If you have a CoolVDS instance with 4 vCPUs and 8GB RAM, you can easily handle multiple parallel jobs if your I/O can keep up. Open your /etc/gitlab-runner/config.toml:

concurrent = 4
check_interval = 0

[[runners]]
  name = "coolvds-nvme-runner-01"
  url = "https://gitlab.com/"
  token = "PROJECT_TOKEN"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_verify = false
    image = "docker:20.10.16"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0

Pro Tip: Notice the privileged = true flag. This is required for Docker-in-Docker (dind). However, dind is slow and has security implications. If possible, map the docker socket /var/run/docker.sock:/var/run/docker.sock instead. Just be aware this allows the container to mess with the host's containers. On a dedicated CI VM, this trade-off is usually acceptable for the speed gain.

4. Network Latency: The Silent Build Killer

Norway has fantastic connectivity, but physics is physics. If your repo is hosted in US-East and your runner is in Oslo, every git fetch and docker pull adds latency. Keeping your infrastructure local reduces Round Trip Time (RTT).

We can also tune the Linux kernel network stack to handle the bursty nature of CI/CD network traffic (lots of connections opening/closing during tests). Add these to /etc/sysctl.conf:

# Increase the range of ephemeral ports
net.ipv4.ip_local_port_range = 1024 65535

# Reuse connections in TIME_WAIT state
net.ipv4.tcp_tw_reuse = 1

# Increase max open files
fs.file-max = 2097152

Apply changes with sysctl -p. This prevents your build script from failing with "Connection timed out" errors when running high-concurrency integration tests.

5. The Compliance & Data Sovereignty Angle

We are seeing tighter enforcement of GDPR and Schrems II rulings across Europe. For Norwegian companies, storing temporary build artifacts—which often contain production database dumps for testing—on US-owned cloud infrastructure is becoming a legal gray area.

Hosting your CI/CD runners on CoolVDS in our Nordic data centers solves two problems:

1. Compliance: Your data stays within the jurisdiction.
2. Latency: You are physically closer to the NIX (Norwegian Internet Exchange), meaning faster push/pull operations for local teams.

Architecture Comparison: Why KVM Wins for CI/CD

Not all virtualization is equal. Many budget providers use OpenVZ or LXC. These are containers sharing a kernel. Docker-in-Docker often fails or performs poorly on these platforms because of cgroup limitations.

Conclusion: Stop Renting Wait Time

A pipeline that runs for 45 minutes breaks the developer's flow state (context switching costs money). A pipeline that runs in 5 minutes keeps the momentum going.

You can spend weeks refactoring your codebase to shave off seconds, or you can migrate your runners to infrastructure that actually supports the I/O throughput modern DevOps demands. The hardware you run on is the foundation of your engineering velocity.

Ready to cut your build times in half? Deploy a high-performance, NVMe-backed KVM instance on CoolVDS today. Your developers—and your CFO—will thank you.