Stop Blaming Jenkins: It's Your Infrastructure

It is March 2018. The GDPR enforcement deadline is less than two months away. Engineering teams across Oslo and Bergen are scrambling to audit data flows, but amidst the compliance panic, another fire is burning: build times. If your deployment pipeline takes 45 minutes, you aren't doing Continuous Integration; you are doing "Eventually Consistent Integration."

I recently audited a setup for a fintech startup in Stavanger. They were running GitLab CI on a budget VPS hosted somewhere in Central Europe. Their complaint? "Docker is slow."

Docker wasn't the problem. I/O wait was.

When you run npm install or compile Go binaries, you are hammering the disk with thousands of small read/write operations. On a noisy, oversold host, your CPU spends half its cycles waiting for the storage controller to respond. In this post, we are going to fix your pipeline using optimization techniques valid for the current Docker 17.12 ecosystem and the right hardware strategy.

1. Diagnosing the "Noisy Neighbor" Effect

Before you rewrite your Jenkinsfile, check your steal time. If you are on a shared platform, other tenants are stealing your cycles. Log into your CI runner and check the disk latency.

Run this during a build:

Look at the %iowait and await columns. If await is consistently over 10ms, your drive is the bottleneck. Here is a snapshot from a struggling build server I analyzed last week:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           14.50    0.00    4.20   35.40    2.10   43.80

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
vda               0.00    12.00   85.00   45.00  3450.00  1200.00    71.54     4.20   32.50   12.10   52.90   6.50  84.50

35.40% iowait. That is a third of the time spent doing absolutely nothing. The w_await (write await) is nearly 53ms. This is unacceptable for CI/CD.

This is why we standardized on NVMe storage for CoolVDS. Spinning rust (HDD) and even standard SATA SSDs over a shared bus cannot handle the random I/O storms generated by concurrent Docker builds. Switching this client to a CoolVDS instance with direct-attached NVMe dropped the w_await to sub-1ms and cut the total build time by 60%.

2. Docker Layer Caching: The 2018 Standard

If you are rebuilding the world every time you commit, you are wasting CPU. With Docker 17.05+, we finally got multi-stage builds. Use them. It keeps your production images small and leverages the build cache effectively.

Here is the pattern you should be using right now:

# Dockerfile

# Stage 1: The Build Environment
FROM node:9-alpine AS builder
WORKDIR /app

# Optimize cache: Copy package.json mostly likely won't change often
COPY package.json yarn.lock ./

# This layer is cached unless dependencies change
RUN yarn install --frozen-lockfile

COPY . .
RUN yarn build

# Stage 2: Production Artifact
FROM nginx:1.13-alpine
COPY --from=builder /app/build /usr/share/nginx/html
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

However, caching on a CI runner is tricky because containers are ephemeral. If you destroy the runner after every job, you lose the cache.

Pro Tip: If you are using GitLab CI, map the Docker socket /var/run/docker.sock (with caution) or use a persistent overlay volume for /var/lib/docker on your runners. This prevents pulling base images like node:9-alpine repeatedly over the WAN.

3. Network Latency and Repo Mirrors

Downloading dependencies from npm, Maven Central, or PyPI takes time. The geographical distance to these repositories adds up.

For Norwegian developers, routing traffic through Frankfurt or Amsterdam to reach US servers adds unnecessary milliseconds to every handshake. The solution is running a local artifact proxy like Nexus or Artifactory, or at least a caching Nginx reverse proxy.

Here is a simple Nginx config to cache aggressive static assets if you are hosting your own registry mirrors:

proxy_cache_path /var/cache/nginx/npm levels=1:2 keys_zone=npm_cache:10m max_size=10g inactive=60m use_temp_path=off;

server {
    listen 80;
    server_name npm-mirror.internal;

    location / {
        proxy_pass https://registry.npmjs.org;
        proxy_cache npm_cache;
        proxy_cache_valid 200 302 60m;
        proxy_cache_valid 404 1m;
        
        # Hide headers to keep npm CLI happy
        proxy_hide_header Set-Cookie;
        proxy_ignore_headers Set-Cookie;
    }
}

Deploying this cache on a CoolVDS instance in Oslo keeps the traffic local to the Norwegian Internet Exchange (NIX). The latency drop from ~40ms (to Central Europe) to ~2ms (local) makes a massive difference when resolving dependency trees with thousands of files.

4. The Compliance Factor: GDPR & Datatilsynet

We cannot ignore the elephant in the room. May 25th is coming. If your CI/CD pipeline processes production dumps for testing (a bad practice, but common), where is that data living?

If your build server is hosted on a US cloud provider, you are entering the complex territory of data transfer mechanisms. Hosting your CI infrastructure on Norwegian soil simplifies your compliance posture regarding the Data Inspectorate (Datatilsynet). Data stays within the legal jurisdiction.

5. Configuring the Runner for Performance

Whether you use Jenkins or GitLab CI, the default concurrency settings are often too conservative. You need to balance CPU cores against memory usage.

For a GitLab Runner config.toml running on a 4 vCore CoolVDS instance, do not be afraid to push the limits:

concurrent = 4
check_interval = 0

[[runners]]
  name = "coolvds-performance-runner"
  url = "https://gitlab.com/"
  token = "YOUR-TOKEN"
  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "docker:17.12.0-ce"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    # Use the host's high-speed /dev/shm to avoid crashes in memory-intensive builds
    shm_size = 2147483648 
    volumes = ["/cache"]
  [runners.cache]
    Insecure = false

Note the shm_size. Default Docker containers often have a tiny 64MB /dev/shm. Tools like Headless Chrome (used for frontend testing) will crash immediately without this adjustment.

Trade-offs and Conclusion

Optimization is about removing bottlenecks. First, you fix the code (caching). Then you fix the network (local mirrors). Finally, you hit the hard limit of physics: hardware speed.

You can spend weeks tweaking Makefiles, or you can migrate to infrastructure designed for high IOPS. While huge bare-metal clusters are great for enterprises, they are overkill for most dev teams. A KVM-based VPS with dedicated resource allocation hits the sweet spot between cost and raw performance.

Don't let slow I/O kill your team's momentum. Deploy a high-performance Jenkins or GitLab runner on CoolVDS today. With our Oslo datacenter, you get the low latency and data sovereignty your CTO demands, and the NVMe speeds your developers dream of.