Stop Waiting for Builds: Optimizing CI/CD Pipelines with NVMe & Local Infrastructure

It is February 2018. If you are still manually FTPing files to production, you are doing it wrong. But if you have automated your deployment pipeline and you are spending half your day staring at a spinning progress bar in Jenkins or GitLab CI, you are also doing it wrong. Time is the one resource we cannot scale.

I recently audited a deployment pipeline for a mid-sized Oslo fintech company. Their build process took 45 minutes. Developers pushed code, went for coffee, came back, and the tests were still running. The bottleneck wasn't CPU. It wasn't network bandwidth. It was disk I/O latency and poor caching strategies. By moving their runners to a proper high-performance VPS and optimizing their Docker layers, we cut that time down to 12 minutes.

Here is how we did it, and how you can stop wasting billable hours watching terminal logs.

The Hidden Killer: Disk I/O and "Noisy Neighbors"

Most CI/CD jobs are incredibly I/O intensive. Think about it: npm install, composer install, or compiling Go binaries involves reading and writing thousands of tiny files. If you are running your build agents on cheap, shared hosting with standard spinning HDDs (or even SATA SSDs sharing bandwidth with 50 other users), your IOPS (Input/Output Operations Per Second) are getting throttled.

Pro Tip: Run iostat -x 1 on your build server during a deployment. If your %util hits 100% while CPU is idling at 20%, your infrastructure is the problem, not your code.

This is where the "noisy neighbor" effect destroys productivity. In a containerized environment like OpenVZ, another tenant compiling a kernel can starve your disk access. This is why for serious DevOps work, we strictly use KVM virtualization—standard on CoolVDS—to ensure dedicated resource isolation.

Optimization 1: Docker Layer Caching

If you are rebuilding your entire container image on every commit, you are burning CPU cycles for nothing. Docker caches intermediate layers. If a layer hasn't changed, Docker reuses it. The trap falls when you copy your source code before installing dependencies.

The Wrong Way

FROM node:8
WORKDIR /app
COPY . . 
# This invalidates the cache every time code changes
RUN npm install 
CMD ["npm", "start"]

The Right Way

Copy your dependency definitions first, install them, and then copy your source code. This way, npm install only runs if package.json changes.

FROM node:8-alpine
WORKDIR /app

# Copy only dependency manifests first
COPY package.json package-lock.json ./

# This layer is now cached unless dependencies change
RUN npm install --production

# Now copy source code
COPY . .

CMD ["node", "index.js"]

Optimization 2: The GDPR Compliance Factor (May 2018 is Coming)

We are three months away from GDPR enforcement. Panic is setting in across Europe, and rightly so. A critical oversight I see in many DevOps pipelines is where the test data lives. Does your CI process spin up a database container? Do you load it with a sanitized dump of production data?

If your CI runner is hosted on a budget cloud provider in the US, and that test data contains PII (Personally Identifiable Information) of Norwegian citizens, you are about to step into a legal minefield. Data transfer agreements like Privacy Shield are shaky ground.

The pragmatic solution is data residency. Keep the pipeline within the EEA (European Economic Area). Hosting your GitLab Runner or Jenkins node on a VPS in Norway solves two problems:

1. Compliance: Data never leaves the jurisdiction.
2. Latency: If your developers are in Oslo or Bergen, pushing code to a server in Frankfurt or New York adds unnecessary lag. CoolVDS offers low latency connectivity directly to NIX (Norwegian Internet Exchange).

Optimization 3: Tuning the Runner

Whether you use Jenkins or GitLab CI, the default configurations are rarely optimized for performance. We need to ensure the runner can handle concurrency without choking.

GitLab Runner Configuration (`config.toml`)

Increase the concurrency limit to utilize the multiple cores available on high-tier VPS plans.

concurrent = 4
check_interval = 0

[[runners]]
  name = "CoolVDS-NVMe-Runner"
  url = "https://gitlab.com/"
  token = "PROJECT_TOKEN"
  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "docker:17.12"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    # Mount the docker socket to reuse the host's docker daemon for speed
    volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
    shm_size = 0

Jenkins Pipeline Parallelism

Stop running tests sequentially. With a powerful VPS, you can run unit tests, linting, and integration tests at the same time.

pipeline {
    agent any
    stages {
        stage('Parallel Test') {
            parallel {
                stage('Unit Tests') {
                    steps {
                        sh 'npm run test:unit'
                    }
                }
                stage('Linting') {
                    steps {
                        sh 'npm run lint'
                    }
                }
                stage('Integration') {
                    steps {
                        sh 'docker-compose up -d db'
                        sh 'npm run test:integration'
                    }
                }
            }
        }
    }
}

Infrastructure Matters: The NVMe Difference

Software optimization can only take you so far. Eventually, you hit the hardware wall. In 2018, the difference between a standard SSD and NVMe storage is night and day for random read/write operations—exactly the kind of workload a CI server generates.

We benchmarked a standard maven clean install on a large Java project:

When you multiply that time saved by 20 builds a day per developer, the ROI on a higher-performance VPS is immediate.

Final Thoughts

Your CI/CD pipeline is the heartbeat of your engineering team. If it is slow, morale drops and release cycles drag. Don't let your infrastructure be the bottleneck. By combining smart Docker caching, parallel execution, and the raw I/O power of NVMe storage, you can turn a sluggish pipeline into a competitive advantage.

And with May 2018 approaching, keeping that pipeline on Norwegian soil isn't just about speed—it's about survival.

Ready to speed up your builds? Deploy a high-performance, GDPR-ready CI runner on CoolVDS today and see the difference raw NVMe power makes.