Why Your CI/CD Pipeline Crawls (And How to Fix It Without Throwing Money at AWS)

It’s 15:45 on a Friday. You’ve just pushed a critical hotfix to `main`. The commit hash exists. Now you wait. And wait. By the time the `npm install` stage finishes, you've lost the flow, and by the time the Docker image pushes to the registry, the office is empty. If this sounds familiar, your pipeline isn't just slow; it's broken.

Most developers treat Continuous Integration (CI) infrastructure as an afterthought. They rely on the default, shared runners provided by GitHub or GitLab. It works fine for a "Hello World" app. But when you are compiling Rust binaries, building massive React bundles, or running integration tests against a PostgreSQL container, the bottleneck is rarely CPU clock speed. It’s almost always Disk I/O and Network Latency.

I've spent the last decade debugging pipelines that hang. Here is the reality: shared cloud runners are noisy neighbors. You are fighting for IOPS with thousands of other builds. In this guide, we are going to optimize a GitLab CI pipeline running on a dedicated Linux environment, specifically looking at how dedicated NVMe storage and local caching strategies can cut build times by 60-70%.

The Silent Killer: I/O Wait

When your build hangs on `Extracting...` or `Linking...`, your CPU is likely sitting idle. It's in a state called iowait. The processor is ready to work, but it's waiting for the disk to hand over data. On standard HDD or shared SSD setups often found in budget hosting or free tier runners, this latency stacks up.

To diagnose this on your current runner, you can use `iostat`. If `%iowait` is consistently above 5-10% during your build process, your storage is too slow.

# Install sysstat to check I/O usage
apt-get install sysstat

# Watch I/O during a build
iostat -xz 1

If you see `await` (average time for I/O requests) spiking over 10ms, you are in trouble. This is why at CoolVDS, we standardize on local NVMe storage rather than network-attached block storage (CEPH) for high-performance instances. The difference in random read/write speeds for `node_modules` or Docker layer extraction is not marginal; it is logarithmic.

Strategy 1: Docker Layer Caching Done Right

The most common mistake I see in `Dockerfile` construction is improper layer ordering. Docker builds layers from top to bottom. If a layer changes, every layer below it must be rebuilt.

The Wrong Way:

FROM node:18-alpine
WORKDIR /app
COPY . .
RUN npm ci
CMD ["node", "server.js"]

Every time you change a single line of code in `server.js` or a CSS file, Docker invalidates the `COPY . .` layer. Consequently, it invalidates the `RUN npm ci` layer. You are re-downloading the entire internet every time you fix a typo.

The Optimized Way:

FROM node:18-alpine
WORKDIR /app

# Copy only dependency definitions first
COPY package.json package-lock.json ./

# Install dependencies. This layer is cached unless package.json changes.
RUN npm ci

# Then copy the rest of the code
COPY . .
CMD ["node", "server.js"]

This is basic, but 50% of the pipelines I audit get it wrong. By utilizing the cache, we only hit the disk for the actual source code changes, sparing the I/O overhead of writing thousands of small files.

Strategy 2: The Dedicated Runner

If you are serious about speed and data sovereignty—especially here in Norway where GDPR and Schrems II compliance is critical—you need your own runners. Relying on US-based cloud runners introduces latency and potential compliance headaches. A dedicated runner in Oslo ensures your code never leaves the jurisdiction and benefits from low latency to local services.

Here is how to deploy a high-performance GitLab Runner on a Debian/Ubuntu instance (like a CoolVDS KVM slice):

# 1. Add the official GitLab repository
curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash

# 2. Install the runner
sudo apt-get install gitlab-runner

# 3. Register the runner (Use the token from your GitLab project CI/CD settings)
sudo gitlab-runner register \
  --url "https://gitlab.com/" \
  --registration-token "PROJECT_REGISTRATION_TOKEN" \
  --executor "docker" \
  --description "coolvds-nvme-runner-oslo" \
  --docker-image "docker:stable" \
  --tag-list "nvme,fast,norway" \
  --docker-privileged

Tuning `config.toml` for Concurrency

Default settings are safe, not fast. Open `/etc/gitlab-runner/config.toml`. We want to leverage the multicore capabilities of your VPS. If you have a 4 vCPU instance, allowing 4 concurrent jobs is usually safe if they aren't all memory-heavy.

concurrent = 4
check_interval = 0

[[runners]]
  name = "coolvds-nvme-runner-oslo"
  url = "https://gitlab.com/"
  token = "TOKEN_HASH"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_verify = false
    image = "docker:stable"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache", "/var/run/docker.sock:/var/run/docker.sock"]
    shm_size = 0

Pro Tip: Mounting `/var/run/docker.sock` allows the container to use the host's Docker daemon. This enables "Docker-in-Docker" usage without the overhead of nested virtualization, leveraging the host's cached images directly. This is significantly faster but has security implications; only use this on private, trusted runners.

Strategy 3: OverlayFS and Storage Drivers

In 2023, the `overlay2` storage driver is the standard, but verify it. Older installations or misconfigured VPS templates might default to `devicemapper` or `vfs`, which are performance disasters.

Check your Docker info:

docker info | grep "Storage Driver"

It should return Storage Driver: overlay2. If not, you are burning CPU cycles on storage translation. `overlay2` works best with the `xfs` backing filesystem with `d_type=true` enabled. CoolVDS images are pre-configured with `ext4` or `xfs` optimized for this, ensuring inode lookups don't become your bottleneck.

The "Norwegian Advantage"

Why does geography matter for CI/CD? Latency. If your team is in Oslo or Bergen, and your staging server is in Frankfurt, but your runner is in Virginia (US-East), you are hair-pinning traffic across the Atlantic twice for every deployment.

By placing your runner on a VPS in Norway:

• Artifact Uploads: Pushing a 500MB Docker image to a local registry happens at line speed.
• Data Privacy: Temporary build artifacts often contain PII or database dumps used for testing. Keeping this data within Norwegian borders satisfies strict Datatilsynet interpretations.
• Stability: The Norwegian power grid is one of the most stable and green in Europe. Uptime matters for build servers too.

Conclusion

Pipeline optimization isn't about finding a magic flag; it's about removing friction. The friction of slow disks, the friction of network latency, and the friction of unnecessary rebuilds. By moving your CI/CD workloads from shared, oversubscribed SaaS runners to a dedicated NVMe-backed instance, you regain control over your release cycle.

Don't let slow I/O kill your productivity. Deploy a dedicated GitLab Runner on a CoolVDS NVMe instance today and stop waiting for the progress bar.