Scaling Jenkins: Optimizing CI/CD Pipelines on KVM and SSDs

It is 3:00 AM. The deployment script for the new Magento storefront has stalled for the third time tonight. You are staring at a blinking cursor in your terminal, watching top report a load average of 25.0 on a dual-core server. The bottleneck isn't CPU. It isn't RAM. It is iowait. Your tests are writing to disk faster than the spinning platters—or even cheap consumer-grade SSDs—can handle.

In the world of Continuous Integration (CI) and Continuous Deployment (CD), latency is the enemy. Whether you are running a massive Drupal installation or a custom Java application, the "works on my machine" excuse doesn't fly when the production pipeline is choking on I/O. As a Systems Architect operating out of the Nordics, I have seen too many development teams cripple their velocity by running heavy build processes on underpowered, oversold virtual machines.

Today, we are going to fix that. We will look at how to tune a Jenkins environment for raw speed, why KVM virtualization is non-negotiable for serious workloads, and how to leverage RAM disks to bypass I/O bottlenecks entirely.

The Hidden Bottleneck: Disk I/O in CI

Most developers underestimate the sheer volume of read/write operations a typical CI test suite generates. Between compiling code, generating assets, and spinning up temporary MySQL databases for integration testing, your disk heads are thrashing wildy. If you are hosting on a standard VPS using OpenVZ or older paravirtualization, you are likely suffering from the "noisy neighbor" effect, where another user's database load steals your I/O cycles.

I recently audited a setup for a client in Oslo using a generic hosting provider. Their build time was 45 minutes. By moving them to dedicated KVM instances with guaranteed resources and optimizing their storage strategy, we cut that down to 8 minutes.

Here is how you diagnose if disk I/O is your problem. Run iostat (part of the sysstat package) during a build:

$ iostat -x 1 10
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           15.40    0.00    4.10   75.20    0.00    5.30

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
vda               0.00   150.00   25.00  340.00   800.00 45000.00   125.48    25.50   55.20   2.50  95.00

If your %iowait is consistently above 20% and %util hovers near 100%, your storage is too slow for your pipeline. In the example above, 75% iowait is catastrophic. The CPU is sitting idle, waiting for the disk to catch up.

Solution 1: The RAM Disk (tmpfs) Trick

Since we are dealing with ephemeral test data—data that exists only for the duration of the build and is then discarded—why write it to the physical disk at all? Linux's tmpfs allows us to mount a file system directly in RAM. It is volatile, but for CI testing (especially MySQL databases spun up for unit tests), it is orders of magnitude faster than any SSD.

Here is how to configure a volatile directory for your Jenkins workspace or MySQL test socket:

# Create a mount point
sudo mkdir -p /var/lib/jenkins/workspace/ramdisk

# Edit /etc/fstab to mount it automatically
# We allocate 2GB of RAM for this mount
tmpfs   /var/lib/jenkins/workspace/ramdisk   tmpfs   defaults,size=2g,noatime,mode=1777   0 0

# Mount it
sudo mount -a

Warning: Everything in tmpfs vanishes on reboot. Use this strictly for temporary build artifacts or test databases, never for persistent storage.

Solution 2: Distributing Builds with Jenkins Slaves

Monolithic Jenkins servers are a single point of failure. In 2013, the best practice is a "Master/Slave" architecture. The Master handles scheduling and GUI, while Slaves (Nodes) do the heavy lifting. This allows you to scale horizontally. You can keep your Master on a smaller instance and spin up high-power Slaves only when needed.

Connecting a Linux slave via SSH is robust. You don't need the heavy Java GUI agent; you just need a headless connection. Ensure your Master has the Slave's public SSH key.

Inside Jenkins (Manage Jenkins > Manage Nodes > New Node), set the Launch method to "Launch slave agents on Unix machines via SSH".

Pro Tip: For JVM-heavy builds (like Maven or Ant), explicit memory management is key to preventing the OOM killer on the slave. In the "Advanced" JVM options for the node, define your heap boundaries: -Xms512m -Xmx1024m -XX:MaxPermSize=256m

Solution 3: Infrastructure and Connectivity

Software tweaks can only go so far. Eventually, you hit the hardware limit. This is where the choice of hosting provider becomes a strategic decision, not just a line item.

The Case for KVM and Private Networking

At CoolVDS, we utilize KVM (Kernel-based Virtual Machine) for our virtualization stack. Unlike container-based solutions (like OpenVZ) where the kernel is shared, KVM provides full hardware isolation. This means your CI pipeline has its own dedicated kernel modules, which is essential if you are running tools like iptables for test harnesses or need specific kernel tuning parameters for high-load TCP traffic.

Furthermore, latency matters. If your dev team is in Norway, routing traffic through a datacenter in the US adds unnecessary delay to every git push and SSH session. Hosting locally, close to the NIX (Norwegian Internet Exchange), ensures that your shell feels instant.

Storage: Moving Beyond Spinning Rust

While standard hard drives (HDDs) are fine for backups, they are death for CI/CD. The random read/write patterns of a build process demand flash storage. We are seeing a shift in the industry. With the NVM Express (NVMe) specification recently finalized (1.0 in 2011), the industry is moving toward NVMe storage, but right now, premium PCIe SSDs are the reality for CoolVDS. This ensures that when your build script unpacks a 500MB tarball, it happens in seconds, not minutes.

Optimizing MySQL for Throwaway Tests

If you cannot use a RAM disk, you can at least tell MySQL to stop worrying about data safety during tests. We want speed, not ACID compliance, for a test database that will be deleted in 10 minutes. Tune your my.cnf to disable the double write buffer and reduce durability guarantees:

[mysqld]
# Disables the double write buffer (risky for prod, great for tests)
innodb_doublewrite = 0

# Flush logs to disk only once per second, not every commit
innodb_flush_log_at_trx_commit = 2

# Keep the buffer pool large enough to hold the active test set
innodb_buffer_pool_size = 1G

# Use separate files per table to avoid contention in the system tablespace
innodb_file_per_table = 1

Legal Compliance in Norway

A pragmatic CTO must also consider the legal aspect. Under the Norwegian Personal Data Act (Personopplysningsloven) and current EU directives, knowing exactly where your data resides is paramount. Using a VPS Norway provider ensures that your intellectual property and any test data containing PII (Personally Identifiable Information) remain within legal jurisdiction, satisfying the requirements of Datatilsynet.

Deploying with Capistrano

Once the build passes, you need to ship it. In 2013, Capistrano remains the gold standard for scripted deployment, especially for PHP and Ruby apps. It handles symlinking releases, ensuring zero-downtime deployments. If a build fails in production, rolling back is a single command.

A simple deploy.rb snippet for a project might look like this:

set :application, "coolvds_app"
set :repository,  "git@github.com:username/repo.git"
set :scm, :git
set :deploy_to, "/var/www/coolvds_app"
set :user, "deploy"
set :use_sudo, false

role :web, "192.168.1.10"                          # Your HTTP server
role :app, "192.168.1.10"                          # This may be the same as your Web server
role :db,  "192.168.1.10", :primary => true        # This is where Rails migrations will run

# Keep only the last 5 releases to save disk space
set :keep_releases, 5
after "deploy:restart", "deploy:cleanup"

Conclusion

Optimizing a CI/CD pipeline is about removing friction. Every second your developers wait for a build is a second they aren't coding. By combining the raw power of low latency KVM instances, managed hosting support, and aggressive Linux tuning, you turn your infrastructure into a competitive advantage.

Don't let slow I/O kill your workflow. If you are ready to stop waiting on iowait, deploy a test instance on CoolVDS in under 55 seconds and see what true dedicated performance feels like.