Zero-Wait Builds: Optimizing CI/CD Pipelines on Norwegian Infrastructure

I once watched a senior backend engineer play ping-pong for 45 minutes because "the build is queued." That’s not a culture problem; that’s an infrastructure failure. In 2023, with the complexity of microservices and container orchestration, your CI/CD pipeline is the heartbeat of your engineering team. If it skips a beat, or worse, drags its feet, you are burning money.

Many teams in Oslo and Bergen default to massive cloud providers hosted in Frankfurt or Ireland for their build agents. They assume raw CPU power is the only metric that matters. They are wrong. Latency, I/O throughput, and neighbor noise are the silent killers of pipeline velocity.

The Hidden Bottleneck: I/O Wait

Most CI/CD jobs are disk-heavy. You are pulling Docker images, extracting layers, compiling binaries, and writing artifacts. If you are running this on a shared VPS with standard SSDs (or heaven forbid, spinning rust), your CPU is spending half its time waiting for the disk controller. This is defined as iowait.

On a recent migration for a fintech client in Stavanger, we moved their Jenkins agents from a shared cloud instance to a CoolVDS KVM instance backed by NVMe. The build time dropped from 12 minutes to 4 minutes without changing a single line of code. Why? Because NVMe protocol parallelism allows the drive to handle thousands of queues at once.

Here is how you verify if your current runner is choking on disk I/O. Run this fio benchmark on your build agent:

fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=1G --readwrite=randwrite --rwmixwrite=75

If your IOPS are under 10,000, your developers are waiting on the hardware, not the compiler.

Geographic Latency and NIX (Norwegian Internet Exchange)

Physics is stubborn. Round-trip time (RTT) from Oslo to US-East is roughly 90-100ms. From Oslo to a data center in Oslo connected via NIX? Sub-5ms. When your CI pipeline pulls 2GB of dependencies from a private registry, that latency compounds.

Furthermore, we have to talk about compliance. In the wake of Schrems II, moving data across borders is a legal headache. By hosting your CI runners and artifact storage on a VPS in Norway, you simplify your GDPR stance. You aren't shipping code to a third country just to compile it. You keep the intellectual property within the jurisdiction of Datatilsynet.

Optimizing Docker for Speed

Hardware solves the I/O problem, but configuration solves the efficiency problem. A common mistake I see is improper Docker layer caching.

1. The Order Matters

Docker caches layers based on the instructions. If you copy your source code before installing dependencies, you invalidate the cache every time you change a line of code.

Bad Implementation:

FROM node:18-alpine
WORKDIR /app
COPY . .
RUN npm install
CMD ["npm", "start"]

Optimized Implementation:

FROM node:18-alpine
WORKDIR /app
# Copy only package.json first to leverage cache
COPY package.json package-lock.json ./
RUN npm ci --only=production
# Then copy the source
COPY . .
CMD ["npm", "start"]

This ensures that npm install only runs when dependencies actually change, not when you fix a typo in index.js.

2. Use the Overlay2 Driver

Ensure your runner is using the overlay2 storage driver. It is the preferred storage driver for all currently supported Linux distributions. Check it with:

If it returns devicemapper or vfs, you are operating with one hand tied behind your back.

Tuning the GitLab Runner

GitLab is the standard for many Nordic dev teams. The default config.toml is rarely optimized for high-performance dedicated instances like those offered by CoolVDS.

If you have a dedicated KVM instance with 8 vCPUs, you should not be limiting your concurrency to 1. However, you also don't want to starve the host OS. Here is a battle-tested configuration for a high-load runner:

concurrent = 10
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "CoolVDS-NVMe-Runner-01"
  url = "https://gitlab.com/"
  token = "REDACTED"
  executor = "docker"
  # Environment variables to force fast compression
  environment = ["FF_USE_FASTZIP=true", "ARTIFACT_COMPRESSION_LEVEL=fastest"]
  [runners.custom_build_dir]
  [runners.cache]
    Type = "s3"
    Path = "gitlab_runner"
    Shared = true
    [runners.cache.s3]
      ServerAddress = "minio.local:9000"
      AccessKey = "minio-access-key"
      SecretKey = "minio-secret-key"
      BucketName = "runner-cache"
      Insecure = true
  [runners.docker]
    tls_verify = false
    image = "docker:23.0.5"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
    shm_size = 2147483648
    pull_policy = "if-not-present"

Pro Tip: Notice the shm_size (Shared Memory). The default Docker container gets 64MB. If you are running heavy tests (like Selenium or Puppeteer) or building large C++ projects, they will crash randomly without this bump.

Distributed Caching with MinIO

If you run multiple runners, local caching isn't enough. You need a distributed cache so Runner A can use the dependencies downloaded by Runner B. In 2023, the standard for self-hosted object storage is MinIO. It's S3 compatible and incredibly fast.

Deploying a local MinIO instance on your CoolVDS environment keeps traffic on the local network (low latency) and avoids egress fees from public clouds. Here is a quick deployment snippet using Docker Compose:

version: '3'
services:
  minio:
    image: quay.io/minio/minio:RELEASE.2023-05-18T00-05-36Z
    command: server --console-address ":9001" /data
    ports:
      - "9000:9000"
      - "9001:9001"
    environment:
      MINIO_ROOT_USER: "admin"
      MINIO_ROOT_PASSWORD: "SuperSecretPassword123"
    volumes:
      - ./data:/data
    restart: always

Point your GitLab Runner to this instance (as shown in the TOML above) to instantly share cache artifacts across your fleet.

The Reliability Factor: KVM vs. Containers

Why do we insist on KVM virtualization at CoolVDS? Because container-based VPS (like OpenVZ/LXC) often suffer from "noisy neighbor" issues where another customer's load spikes affect your kernel performance. In a CI/CD context, consistency is key. You cannot debug a flaky test if the flakiness comes from the hypervisor.

KVM provides a hardware-level abstraction. Your RAM is yours. Your NVMe queues are yours. This isolation is critical when running heavy compilation jobs or integration tests that require predictable timing.

Conclusion

Optimizing your CI/CD pipeline is about removing friction. It requires a combination of smart caching, efficient container layering, and most importantly, the right underlying infrastructure. Don't let your high-performance engineering team get bogged down by low-performance hardware.

If you are tired of watching the "Pending" spinner, it's time to own your infrastructure. Deploy a high-frequency, NVMe-backed KVM instance on CoolVDS today and cut your build times in half.