The problem
A CI system runs code written by someone else: a pull request from a fork, a dependency with a post-install script, a test suite you have never read. The usual way to run that code is a container, because it starts fast and every CI product supports it. The question I wanted to answer is whether a container is a safe place for a hostile job, or whether it needs a microVM such as Firecracker.
I set out to run the same hostile steps in containers and in a Firecracker microVM. Only the container half could be run, which became a finding in itself, so this article reports what ran, what refused to run, and what that means for choosing a boundary.
The environment I actually had
Everything below happened in one clean Linux sandbox. I did not assume anything about it, so the first step was to ask it what it is, and the second was to ask whether it offers the hardware virtualization that a microVM needs.
uname -a
Linux vm 6.18.44-fc-v553 #1 SMP PREEMPT_DYNAMIC @0 x86_64 x86_64 x86_64 GNU/Linux
cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04.5 LTS"
VERSION_ID="24.04"
nproc
4
free -m
Mem: 16087 365 14888 12 1062 15722
It is Ubuntu 24.04.5 with four vCPUs and about 16 GB of RAM, as printed above. I make no claim about where it runs or on what hardware. What matters is that it exposes no KVM device, as the next section shows.
The cause
Containers and microVMs differ in what an attacker must break to reach the host. A container is an ordinary Linux process with namespaces, cgroups, a capability set and a seccomp filter. It talks to the same kernel as the host, so any kernel bug or any generous configuration flag is a path out. A microVM puts a separate guest kernel behind a hypervisor.
Containers share the host kernel
The first thing to check is the kernel a container reports. If it is the host's kernel, every syscall a CI job makes lands in host kernel code that the job author can probe at will, which is the whole attack surface.
docker run --rm alpine:3.20 uname -a
Linux f52a9e125a45 6.18.44-fc-v553 #1 SMP PREEMPT_DYNAMIC @0 x86_64 Linux
The release string 6.18.44-fc-v553 is identical to the host's. Docker's default profile still narrows things: inside the container the process has a reduced capability mask and a seccomp filter attached, while the host shell has neither. I read both from /proc/self/status.
docker run --rm alpine:3.20 grep -E 'CapEff|Seccomp|NoNewPrivs' /proc/self/status
CapEff: 00000000a80425fb
NoNewPrivs: 0
Seccomp: 2
Seccomp_filters: 1
Firecracker needs a device this sandbox lacks
Firecracker is a virtual machine monitor built on KVM. The upstream getting-started guide says it needs read and write access to /dev/kvm. I checked for the device and for the CPU virtualization flags before going any further with the microVM side.
ls -l /dev/kvm
ls: cannot access '/dev/kvm': No such file or directory
grep -c -E 'vmx|svm' /proc/cpuinfo
0
No device and no vmx or svm flag: this sandbox does not offer nested virtualization. That is the practical cause of a very common surprise for people who try Firecracker on an ordinary cloud VM or inside a container: the microVM is not slow or misconfigured, it simply cannot start.
Source of truth
Two upstream documents decide the questions in this article. I read both in the Firecracker repository, which is published under the Apache License 2.0 (I printed the top of the LICENSE file shipped in the release tarball and it is the Apache 2.0 text).
- The Firecracker repository is the project itself; release v1.18.0 is the build I downloaded.
- getting-started.md states that Firecracker requires read and write access to /dev/kvm, and that the project itself uses bare-metal instance types because EC2 only offers KVM there, apart from a separate Nitro sandbox mode.
- design.md says the first isolation layer is KVM plus the Firecracker virtualization boundary, with seccomp filters, cgroups, namespaces and the jailer as further layers around the VMM process.
- prod-host-setup.md lists the host hardening Firecracker expects in production, from jailer configuration to mitigating hardware vulnerabilities.
- The gVisor documentation describes a per-sandbox application kernel that takes over the system interfaces normally implemented by the host kernel; the install guide covers the Docker runtime setup I used.
Together they confirm the architecture claim from the previous section, and they tell me exactly what to verify here, in this sandbox: KVM presence for Firecracker, and the kernel and device view from inside the sandbox for gVisor and plain containers, since that is what a CI runner exposes.
Possible solutions
There are four realistic isolation levels for an untrusted CI job, ordered here from cheapest to strongest. They are not mutually exclusive, because the cheaper ones are also the hardening you wrap around the stronger ones, so most real setups combine two of them.
- Plain container, default settings. Fast and universal, but the job shares the host kernel and whatever the CI runner exposes to it.
- Hardened container. Dropped capabilities, a read-only root filesystem, no network, no new privileges, and cgroup limits on processes and memory. Same kernel, far fewer levers.
- User-space kernel sandbox (gVisor, runsc). The container's syscalls are served by a separate application kernel instead of the host kernel. It plugs into Docker as another runtime and, as I show below, does not need /dev/kvm.
- MicroVM (Firecracker). A real guest kernel behind KVM. The strongest boundary of the four, but it needs /dev/kvm on the host and you build the rootfs, networking and job protocol yourself.
A common way a container boundary disappears in practice is giving the job the Docker socket so it can build images, or running it with the privileged flag. I tested both below, since real pipelines do this under deadline pressure.
The choice and why
My rule is to match the boundary to the trust level of the code. For code from your own team, a hardened container is usually enough. For code from strangers, such as fork pull requests on a public repository, I would choose a microVM per job, because a guest kernel means a kernel exploit lands in a throwaway VM rather than on the runner.
In this sandbox I could not follow that rule to the end, because Firecracker cannot start without KVM. So I tested what I could: how little a default container protects, what hardening buys, and gVisor as the nearest runnable alternative that also replaces the kernel surface.
I picked gVisor as the runnable comparison because it installs as a Docker runtime and, as shown below, ran here without /dev/kvm. It is not equivalent to a microVM: it is a different design with different trade-offs, and I measured its cost.
Doing it in the sandbox
Step 1: install Firecracker and try to boot a microVM
I downloaded the v1.18.0 release tarball and checked that the binary runs. Then I fetched the guest kernel that the upstream guide points to, and wrote a minimal one-vCPU config. I skipped the root filesystem on purpose, since the failure happens before any disk is opened.
curl -sL -o fc.tgz https://github.com/firecracker-microvm/firecracker/releases/download/v1.18.0/firecracker-v1.18.0-x86_64.tgz
tar xzf fc.tgz
./release-v1.18.0-x86_64/firecracker-v1.18.0-x86_64 --version
Firecracker v1.18.0
cat /tmp/vmconfig.json
{
"boot-source": {"kernel_image_path": "/tmp/vmlinux", "boot_args": "console=ttyS0 reboot=k panic=1"},
"drives": [],
"machine-config": {"vcpu_count": 1, "mem_size_mib": 256}
}
timeout 20 /tmp/release-v1.18.0-x86_64/firecracker-v1.18.0-x86_64 --no-api --config-file /tmp/vmconfig.json 2>&1 | sed 's/^[0-9T:.-]* //'
[anonymous-instance:main] Running Firecracker v1.18.0
configured on the /dev/kvm file's ACL.
Error: RunWithoutApiError(BuildMicroVMFromJson(StartMicroVM(Kvm(Kvm(Error(2))))))
[anonymous-instance:main] Firecracker exiting with error. exit_code=1
The binary starts and parses the config, then stops with a KVM error (the long error line says it could not create the KVM object, and I trimmed the log timestamps with sed): it cannot open /dev/kvm. I also tried creating the device node by hand with mknod, using major 10 and minor 232. Opening it fails with "No such device", consistent with a kernel that has no KVM driver behind the node. The microVM half cannot be reproduced here.
mknod /tmp/kvmnode2 c 10 232; head -c0 /tmp/kvmnode2
head: cannot open '/tmp/kvmnode2' for reading: No such device
[ -r /dev/kvm ] && [ -w /dev/kvm ] && echo "OK" || echo "FAIL"
FAIL
Step 2: what a default container gives a hostile job
I planted a secret file in the host's /root (it contained the text host-secret-1794902497), then ran a plain container and tried to read it. This is the control: with no special flags the host filesystem is not visible, which is the container doing its job.
docker run --rm alpine:3.20 sh -c 'cat /root/ci-host-secret.txt; ls /host' 2>&1
cat: can't open '/root/ci-host-secret.txt': No such file or directory
ls: /host: No such file or directory
Step 3: the two shortcuts that remove the boundary
Now the pipeline habits. First, a job given the Docker socket, which is how many CI setups let steps build images. From inside that container I asked the host's Docker daemon to start a sibling container with the host root mounted, read the secret, and created a file on the host.
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock docker:cli sh -c 'docker run --rm -v /:/host alpine:3.20 sh -c "cat /host/root/ci-host-secret.txt; touch /host/root/written-by-ci-job; id"' 2>&1 | tail -5
host-secret-1794902497
uid=0(root) gid=0(root) groups=0(root),0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel),11(floppy),20(dialout),26(tape),27(video)
ls /root/written-by-ci-job
/root/written-by-ci-job
The job read the host secret and wrote a root-owned file on the host. No exploit was involved: socket access is access to a root-capable daemon. Second, the privileged flag. A privileged container sees the host's device nodes (counted below) and, as I confirmed, can mount the host's disk and read the same secret.
docker run --rm alpine:3.20 sh -c 'ls /dev | wc -l'
14
docker run --rm --privileged alpine:3.20 sh -c 'ls /dev | wc -l'
108
docker run --rm --privileged alpine:3.20 sh -c 'mkdir /h; mount -t ext4 /dev/vda /h; echo mount_rc=$?; cat /h/root/ci-host-secret.txt' 2>&1
mount_rc=0
host-secret-1794902497
Step 4: harden the container
Next I applied the cheap controls together: all capabilities dropped, no new privileges, read-only root, no network, a non-root user, and limits on processes, memory and CPU. Then I tried a write, and a loop that spawns 200 background processes, which is a crude fork bomb.
docker run --rm --cap-drop ALL --security-opt no-new-privileges --read-only --network none --pids-limit 64 --memory 256m --cpus 1 --user 1000:1000 alpine:3.20 sh -c 'grep -E "CapEff|NoNewPrivs|Seccomp:" /proc/self/status; touch /x; for i in $(seq 1 200); do sleep 30 & done; cat /sys/fs/cgroup/pids/pids.max' 2>&1 | head -8
CapEff: 0000000000000000
NoNewPrivs: 1
Seccomp: 2
touch: /x: Read-only file system
sh: can't fork: Resource temporarily unavailable
The capability mask is zero, writes are refused, and the process limit stopped the fork loop: a large gain for almost no cost. The job's syscalls still hit the host kernel, though, so a kernel bug remains reachable from inside this container.
Step 5: add gVisor as a Docker runtime
I downloaded the gVisor release, verified its published SHA-512 checksum, unpacked it into /usr/local/bin, and registered it with Docker. The per-file runsc URL under the release/latest path returned HTTP 404 when I tried it, so I listed the storage bucket and used the dated 20261109 release instead.
sha512sum -c gvisor.tar.bz2.sha512
gvisor.tar.bz2: OK
runsc install
docker info 2>&1 | grep Runtimes
Runtimes: io.containerd.runc.v2 runc runsc
docker run --rm --runtime runsc alpine:3.20 uname -a
Linux 6739a5138d14 4.19.0-gvisor #1 SMP Sun Jan 10 15:06:54 PST 2016 x86_64 Linux
The same image now reports a kernel named 4.19.0-gvisor instead of the host's 6.18.44. That string is the application kernel's own banner, not the real one: the job's syscalls are served by gVisor rather than by the host kernel directly. The Docker runtime list now shows runsc next to runc, and no /dev/kvm was needed for any of this.
Step 6: measure the price of the extra layer
A sandbox that intercepts syscalls pays for it where syscalls are frequent. I used a deliberately syscall-heavy loop: dd copying one million single-byte blocks, which is two million read and write syscalls. I ran the pair twice. The first pass gave 0.35 s and 8.94 s; the second pass is shown.
for rt in runc runsc; do echo "== $rt"; docker run --rm --runtime $rt alpine:3.20 sh -c 'time dd if=/dev/zero of=/dev/null bs=1 count=1000000' 2>&1 | grep -E 'records out|real'; done
== runc
1000000+0 records out
real 0m 0.28s
== runsc
1000000+0 records out
real 0m 8.24s
That is roughly a 25 to 30 times slowdown for this worst-case pattern, on this sandbox, with the runsc default platform. Ten back-to-back starts of a no-op container, timed next, were close, so start-up was not a differentiator here. These are single observations, not benchmarks.
time (for i in 1 2 3 4 5 6 7 8 9 10; do docker run --rm --runtime runc alpine:3.20 true; done)
real 0m3.052s
time (for i in 1 2 3 4 5 6 7 8 9 10; do docker run --rm --runtime runsc alpine:3.20 true; done)
real 0m2.817s
A note on Firecracker's own attack surface
The release tarball includes the seccomp filter that Firecracker installs on itself. I counted the distinct syscalls each of its three thread classes may make: 51 for vmm, 32 for api and 27 for vcpu. That is the host-kernel surface reachable through the VMM, and it is small.
python3 -c "import json; d=json.load(open('/tmp/release-v1.18.0-x86_64/seccomp-filter-v1.18.0-x86_64.json')); [print(t, len({r['syscall'] for r in v['filter']})) for t,v in d.items()]"
vmm 51
api 32
vcpu 27
Limits and alternatives
What worked
Everything on the container side behaved as the docs imply. A default container hides the host filesystem, yet handing it the Docker socket or the privileged flag gave root-level read and write on the host within a single command each. Hardening flags closed those doors cheaply, and gVisor swapped the kernel at a measurable syscall cost.
What did not work, and the limits of this article
The headline comparison failed: Firecracker could not start because the sandbox exposes no /dev/kvm and no hardware virtualization flags. I therefore have no boot time, no memory overhead and no job throughput for a microVM, and I make no claim about them. I also did not try to break out of any runtime with a kernel exploit, so I cannot say how much each boundary resists a real attack.
When to choose another method
- Trusted team code: a hardened container is the sensible default. Never mount the Docker socket into it, and never use the privileged flag.
- Untrusted code on hosts with KVM: use Firecracker or a similar microVM per job, started through the jailer, and budget for the rootfs and networking work.
- Untrusted code where KVM is unavailable: gVisor is a practical middle path, provided your jobs are not dominated by syscalls. Test your real pipeline before committing.
- Builds that need Docker: run the daemon inside the sandbox boundary, not on the runner host.
To verify the Firecracker half yourself, run the same config on a host where the read and write test on /dev/kvm prints OK, which is the first requirement from the getting-started guide. In this sandbox, that test printed FAIL, so none of the microVM steps could run.