CI/CD Pipeline Optimization: Killing the I/O Bottleneck

I watched a senior backend developer wait 24 minutes for a deployment pipeline to finish yesterday. He went to get coffee, came back, checked Slack, and it was still building. The bottleneck wasn't the compilation of the Go binary. It wasn't the unit tests. It was the file system.

Most DevOps engineers treat Continuous Integration runners as throwaway compute units. They prioritize CPU cores, thinking more parallel jobs equal faster builds. This is a fundamental misunderstanding of how modern package managers and container builders work. Whether you are running npm install, pip install, or building Docker layers, you are primarily performing an I/O stress test.

In this analysis, we are going to tear down the common performance killers in CI/CD pipelines, specifically for teams operating in Norway and Northern Europe, and look at why moving from shared cloud runners to dedicated NVMe-backed KVM instances (like those we provision at CoolVDS) is the only way to get sub-3-minute builds.

The Hidden Killer: iowait and Package Managers

Let's look at the mechanics of a standard Node.js project build. When your pipeline triggers, it pulls thousands of small files. Standard HDD-backed VPS or throttled cloud instances choke on this. They might offer high sequential throughput, but their IOPS (Input/Output Operations Per Second) for random small files are abysmal.

If you are seeing slow builds, run this diagnostic on your current runner:

iostat -xz 1 10

If your %iowait exceeds 5% during the dependency installation phase, your CPU is sitting idle, waiting for the disk to catch up. You are paying for compute you can't use.

Pro Tip: Network latency amplifies this pain. If your runner is in Frankfurt but your private registry or data is in Oslo, the TCP handshake overhead on thousands of requests adds up. Localizing your infrastructure matters.

Optimization 1: Docker BuildKit and Cache Mounts

If you are still writing Dockerfiles the way we did in 2018, you are doing it wrong. The standard layer caching is fragile; if you change one line in your package.json, the entire layer invalidates. Docker BuildKit (standard in Docker 23+) allows for cache mounts that persist across builds, even if the layer is rebuilt.

Here is how a production-grade Dockerfile should look in late 2023:

# syntax=docker/dockerfile:1.4
FROM node:20-alpine AS base

WORKDIR /app

# Combine update and install to reduce layers
RUN apk add --no-cache libc6-compat

FROM base AS deps
# Use a cache mount for npm's local cache
# This persists on the host runner between builds!
COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm \
    npm ci --omit=dev

FROM base AS builder
COPY --from=deps /app/node_modules ./node_modules
COPY . .

RUN npm run build

FROM base AS runner
ENV NODE_ENV production
COPY --from=builder /app/public ./public
COPY --from=builder /app/.next/standalone ./
COPY --from=builder /app/.next/static ./.next/static

CMD ["node", "server.js"]

Note the --mount=type=cache flag. This instructs the Docker builder to mount a volume from the host into the container at /root/.npm. The next time you run this build, even if package.json changes, the underlying npm cache is already warm. This cuts install times by 40-60%.

Enabling BuildKit

Ensure your CI environment variables invoke the modern builder:

Optimization 2: The Infrastructure Layer

Software optimization has limits. Eventually, you hit the hardware wall. We often see clients migrating from hyperscalers to CoolVDS because of the "noisy neighbor" effect. In a massive public cloud, your CI runner might be sharing physical disk spindles with a database thrashing the I/O.

For consistent pipeline performance, you need NVMe storage passed through via KVM. This ensures that the virtualization layer adds minimal overhead to disk operations.

Here is a benchmark script using fio (Flexible I/O Tester) to verify if your current VPS provider is lying to you about "SSD speeds":

fio --randrepeat=1 \
    --ioengine=libaio \
    --direct=1 \
    --gtod_reduce=1 \
    --name=test \
    --filename=test \
    --bs=4k \
    --iodepth=64 \
    --size=1G \
    --readwrite=randrw \
    --rwmixread=75

Interpreting the results:

• IOPS: Below 3000? You are on spinning rust or heavily throttled storage. Move away. CoolVDS NVMe instances typically push 15k+ IOPS easily.
• Latency: If 95th percentile latency is above 2ms, your builds will feel sluggish.

Optimization 3: Self-Hosted Runners with Terraform

The shared runners provided by GitLab or GitHub are convenient, but they are slow and often located in the US. For Norwegian companies, this introduces latency and potential GDPR headaches (Schrems II compliance). Using a self-hosted runner on a server in Oslo solves both data residency and speed issues.

Below is a snippet to deploy a dedicated GitLab runner on a CoolVDS instance using Terraform. This setup assumes you are using the Docker executor.

resource "gitlab_runner" "coolvds_runner" {
  registration_token = var.gitlab_registration_token
  description        = "CoolVDS-NVMe-Runner-Oslo"
  locked             = false
  tag_list           = ["coolvds", "nvme", "docker"]
  run_untagged       = false
}

resource "remote_file" "config_toml" {
  path = "/etc/gitlab-runner/config.toml"
  content = templatefile("${path.module}/templates/config.toml.tpl", {
    concurrent = 4
    # CoolVDS instances handle high concurrency well due to dedicated CPU allocation
  })
}

# Post-provisioning script to install Docker and Runner
provisioner "remote-exec" {
  inline = [
    "curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh | sudo bash",
    "sudo apt-get install gitlab-runner",
    "sudo usermod -aG docker gitlab-runner"
  ]
}

Local Caching Strategies

Don't just cache the package manager folders. Cache the compiler artifacts. If you use Java/Maven or C++/CMake, the object files should persist.

For a GitLab CI pipeline, mapping a local volume on the host to the runner container is more performant than using the distributed cache/S3 upload/download cycle, provided you use sticky runners (runners that pick up the same jobs).

# .gitlab-ci.yml optimization example
build_job:
  stage: build
  tags:
    - coolvds
  image: maven:3.8.6-openjdk-18
  script:
    - mvn clean package -B
  cache:
    key: "$CI_COMMIT_REF_SLUG"
    paths:
      - .m2/repository
    policy: pull-push

The Norwegian Context: Latency and Law

We cannot ignore the legal landscape in 2023. The Norwegian Datatilsynet is increasingly strict about where personal data—often inadvertently included in test databases or production dumps used in CI—is processed. Hosting your CI/CD infrastructure on US-owned soil (even if the datacenter is in Europe) carries risk.

By hosting your runners on CoolVDS in Norway, you achieve two things:

1. Compliance: Data stays within Norwegian jurisdiction.
2. Latency: Round trip time (RTT) from your Oslo office to the server is <2ms, compared to ~30ms to Frankfurt or ~100ms to US East.

Summary

Stop accepting slow builds as a fact of life. The technology exists to make them instant. It requires a shift from "default settings" to deliberate architecture.

Your Checklist:

• Switch to Docker BuildKit with --mount=type=cache.
• Verify your disk I/O with fio. If it's slow, switch providers.
• Move CI execution closer to your data and developers (geographically).
• Use self-hosted runners on dedicated KVM resources.

Your developers cost too much to have them staring at a progress bar. Deploy a high-performance runner on a CoolVDS NVMe instance today and give them their time back.