Stop Watching Paint Dry: Optimizing CI/CD IOPS Bottlenecks on Linux Infrastructure

There is a dangerous myth in our industry: "My code is compiling, so I'm on a coffee break." It's nonsense. Long build times destroy flow states. If a developer waits 20 minutes for a Jenkins job to return a syntax error, they have already mentally moved on to Reddit or Hacker News. Context switching is the enemy of shipping.

In 2018, with the rise of Docker containers for everything, the bottleneck has shifted. I’ve spent the last month debugging a deployment pipeline for a client in Oslo—a standard heavy Java stack running on a generic European VPS provider. Their builds took 45 minutes.

We dropped that to 8 minutes. We didn't rewrite the code. We didn't buy a massive dedicated server. We fixed the I/O.

The Silent Killer: I/O Wait

Most DevOps engineers throw vCPUs at a slow pipeline. This is usually a waste of money. CI/CD tasks—specifically npm install, mvn clean install, or extracting Docker images—are not CPU-bound. They are I/O-bound. They involve writing thousands of tiny files to the disk.

If you are running on a standard HDD or a shared SATA SSD with a "noisy neighbor," your CPU spends most of its time doing absolutely nothing. It is just waiting for the disk controller.

Here is how you prove it. Run this during your build process:

iostat -x 1

Look at the %iowait column. If you see numbers climbing above 10-15% while your CPU usage (user/system) sits low, your storage is the bottleneck.

Pro Tip: On a shared VPS, %steal is also a metric to watch. If it's high, the hypervisor is throttling you. At CoolVDS, we use KVM with strict resource isolation to prevent this, but many budget hosts oversell their cores aggressively.

War Story: The `node_modules` Black Hole

During the project mentioned above, we noticed the pipeline stalled specifically during the dependency fetch phase. Node.js projects are notorious for this. A single project can have 40,000 files in node_modules. Writing 40,000 files to a disk with low IOPS (Input/Output Operations Per Second) is torture.

We saw write latencies spiking to 300ms. In a high-speed environment, that should be under 2ms.

Tactical Fixes for 2018

Before you migrate servers, try these configuration tweaks. They work on Jenkins, GitLab CI, or Bamboo.

1. Mount Build Directories as tmpfs (RAM Disk)

Why write temporary build artifacts to disk if you're just going to zip them up and delete them 5 minutes later? RAM is orders of magnitude faster than any NVMe drive.

If you are using Docker Compose for your test environments, map the heavy write directories to tmpfs.

version: '3'
services:
  app:
    image: node:9-alpine
    tmpfs:
      - /app/node_modules
      - /app/build
    command: npm test

For Jenkins, you can mount the workspace root as a RAM disk if you have enough memory. Be careful: if the server reboots, the data is gone. But for CI jobs, who cares? Data persistence is for artifacts, not workspaces.

2. Docker Layer Caching Strategy

Don't rebuild the world every time. With the release of Docker 17.05 last year, multi-stage builds became stable. Use them to cache dependencies.

Here is a refined Dockerfile pattern we use to prevent re-downloading dependencies unless package.json changes:

FROM node:9 AS base
WORKDIR /app

# Copy only dependency definitions first
COPY package.json package-lock.json ./

# This layer is cached unless package.json changes
RUN npm install

# Now copy the source code
COPY . .
RUN npm run build

3. Tuning the Overlay2 Driver

Ensure your Linux kernel is 4.0+ (standard on Ubuntu 16.04 now) and you are using the overlay2 storage driver. The old aufs or devicemapper drivers are performance graveyards.

Check your config:

docker info | grep "Storage Driver"

If it doesn't say overlay2, edit your daemon config:

# /etc/docker/daemon.json
{
  "storage-driver": "overlay2"
}

The Infrastructure Reality Check

Software tweaks can only take you so far. Eventually, physics wins. If the underlying disk cannot handle the IOPS, your pipeline will drag.

This is where the choice of VPS becomes critical. In Norway, data sovereignty is becoming the headline topic with GDPR enforcement landing in May. You need a host that keeps data within borders (Datatilsynet is watching), but you also need raw speed.

Standard SATA SSDs top out around 500-600 MB/s. NVMe drives, which communicate directly with the PCIe bus, can hit 3,000+ MB/s. For CI/CD, the sequential speed matters less than the Random 4K Write speed.

Benchmark Comparison

We built CoolVDS on pure NVMe arrays precisely for this reason. When you are running a git clone followed by a compile, you are hitting the disk thousands of times per second.

Network Latency and Mirrors

Finally, don't ignore the network. If your server is in Frankfurt but your developers and private Docker registry are in Oslo, you are adding latency to every docker push.

By hosting locally in Norway, you benefit from low latency connections to the NIX (Norwegian Internet Exchange). Ensure your sources.list or mirrorlist is pointing to local mirrors.

# /etc/apt/sources.list
deb http://no.archive.ubuntu.com/ubuntu/ xenial main restricted

Small changes. Massive impact.

Conclusion

Your CI/CD pipeline is the heartbeat of your engineering team. If it's slow, your culture suffers. Stop optimizing code that isn't running yet and look at your infrastructure. Use RAM disks for transience, cache your layers aggressively, and ensure your underlying hardware isn't stuck in 2015.

If you are tired of fighting for IOPS on oversold platforms, spin up a CoolVDS instance. Between the NVMe storage and the low latency to Norwegian ISPs, you might finally get that build time under 5 minutes.