Console Login

Running the Cilium agent from main on a bare Linux VM

The problem

I wanted to see the eBPF datapath of Cilium work from the current main branch on a plain Linux machine with no Kubernetes cluster. The repository is Apache-2.0 licensed, and the latest push-triggered run of its "Image CI Build" workflow on main concluded with success. I did not run that workflow myself. Despite the green status, a fresh checkout did not build, and once it built, the agent died twice before it ran.

The environment

Everything below ran in a clean Linux sandbox, and I report only what it printed about itself. I ran everything as root and shallow-cloned the repository. I tested the development head of main, not a release, so a released version may behave differently. The kernel, the distribution and the commit were:

uname -a
Linux vm 6.18.44-fc-v595 #1 SMP PREEMPT_DYNAMIC @0 x86_64 x86_64 x86_64 GNU/Linux
cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04.5 LTS"
VERSION_ID="24.04"
VERSION="24.04.5 LTS (Noble Numbat)"
cd /tmp/cilium && git log -1 --format=%H
36d88648cab8c4848197419035c1cb0e6e7f9cd6

Three failures in a row

The first failure is the Go toolchain: the module wants a newer Go than the one installed. The second is that the freshly built cilium-agent refuses to start because it cannot find its BPF source directory. The third is that it aborts on a kernel without IPv6. None of these is a bug in the code. They are the gap between "it compiles in CI" and "it runs on this machine".

The cause

Each failure has a different, mundane cause, and I list them in the order I hit them, because fixing one only exposed the next. The toolchain problem is a version requirement in go.mod. The BPF problem is that go build produces only a Go binary, while the agent also needs the C sources of its datapath on disk. The IPv6 problem is that the agent enables IPv6 by default, and this kernel has none.

The Go version requirement

The Go installed in the sandbox is older than the version the module declares. These two commands show both numbers, and the third shows what a build does when automatic toolchain downloads are switched off. Since Go 1.21 the go line is a hard minimum, not a hint, so the error is expected:

cd /tmp && go version
go version go1.24.7 linux/amd64
cd /tmp/cilium && grep -n '^go ' go.mod
3:go 1.27
cd /tmp/cilium && GOTOOLCHAIN=local go build -o /tmp/x ./cilium-dbg
go: go.mod requires go >= 1.27 (running go 1.24.7; GOTOOLCHAIN=local)

No IPv6 in the kernel

The other two causes show up only at run time, so their error messages appear in the step-by-step section, next to the fixes. Briefly, the agent looks for a bpf directory below its library directory, and it tries to install an IPv6 routing rule unless told not to. For IPv6, I checked what the kernel offers: there is no ipv6 entry under /proc/sys/net.

ls /proc/sys/net
bridge
core
ipv4
mptcp
netfilter
nf_conntrack_max
unix

Source of truth

I relied on three references, plus the agent's own error messages, which were the most useful of the four. They separate documented behaviour from what I only observed in this sandbox, and they are the places to look first if your result differs from mine on another kernel or another Go version.

The Go toolchain documentation

The Go toolchains reference says the default GOTOOLCHAIN=auto runs a newer toolchain when the go line in go.mod is newer than the bundled one: it first looks in the PATH, and otherwise downloads and caches a copy. With switching disabled, the go command refuses to run in such a module. The Go 1.21 toolchain post explains why the go line became a minimum.

The repository and the host firewall guide

The repository's Makefile has an install-bpf target whose help text says it copies the BPF source files from bpf/ to /var/lib/cilium/bpf/; the agent's fatal message names the same target. For the final check I used the host firewall from the Cilium host firewall guide, which says cilium-dbg status shows the interfaces it applies to. The host policies page says host rules are a Kubernetes resource.

Possible solutions

Each problem has more than one way out. For the Go version, you can leave GOTOOLCHAIN=auto alone and let Go fetch 1.27, install a Go 1.27 tarball yourself, or build inside a container image that already has it. There is no running Docker daemon in this sandbox, so I could not try the container route.

For the BPF sources, you can run make install-bpf into the default /var/lib/cilium, install under a private DESTDIR, or populate the library directory yourself. For IPv6, you either turn it off with --enable-ipv6=false or use a kernel that has it. For isolation, you can run the agent on the host, in a throwaway network namespace, or in a separate VM. I could not change the kernel, so only the flag was open to me.

Why this fix

I chose the automatic toolchain download, a private DESTDIR for the BPF sources, the IPv6 flag, and a network namespace. Automatic download is the documented default and needs no manual install; on a locked-down network you would pin GOTOOLCHAIN or provide a mirror instead. The private DESTDIR keeps the experiment out of /var/lib, and the namespace gives the agent a realistic uplink device while the sandbox's own eth0 stays untouched.

I skipped Kubernetes entirely, because the question was whether the agent and its datapath work, and the agent has an --enable-k8s=false switch for exactly that. I also passed --enable-health-checking=false to keep the setup minimal, since the health checker is not what I wanted to test.

Step by step

Build the agent and the debug CLI

With the default toolchain setting, the build that failed above now downloads Go 1.27.0 and succeeds. I cleared the cached toolchain first, so that the download message would appear again. Inside the repository, go version then reports the new toolchain, and the agent builds from the daemon directory:

cd /tmp/cilium && go build -o /tmp/cilium-dbg ./cilium-dbg
go: downloading go1.27.0 (linux/amd64)
cd /tmp/cilium && go version
go version go1.27.0 linux/amd64
cd /tmp/cilium && go build -o /tmp/cilium-agent ./daemon

Create the isolated network

I made a namespace named cil with a veth pair. The host side is 10.99.0.1, the namespace side is 10.99.0.2, and the namespace has a default route back to the host side. The agent is later told to use veth-a as its only device:

ip netns add cil && ip link add veth-a type veth peer name veth-b && ip link set veth-a netns cil && ip addr add 10.99.0.1/24 dev veth-b && ip link set veth-b up && echo ready
ready
ip netns exec cil sh -c 'ip addr add 10.99.0.2/24 dev veth-a; ip link set veth-a up; ip link set lo up; ip route add default via 10.99.0.1; ip -br a; ip route'
lo               UNKNOWN        127.0.0.1/8
veth-a@if7       UP             10.99.0.2/24
default via 10.99.0.1 dev veth-a
10.99.0.0/24 dev veth-a proto kernel scope link src 10.99.0.2

Failure two: no BPF sources

I started the agent in the foreground with Kubernetes switched off and kept only the fatal line of its log, because it prints its full flag list first and the rest of that output is not needed here. I pointed the library directory at an empty path on purpose. The message names the fix itself:

rm -rf /tmp/run; mkdir -p /tmp/run; cd /tmp && ip netns exec cil /tmp/cilium-agent --enable-k8s=false --ipam=cluster-pool --devices=veth-a --state-dir=/tmp/run/state --lib-dir=/tmp/run/lib 2>&1 | grep 'level=fatal' | cut -c1-400
time=2026-11-18T12:52:33.096355652Z level=fatal msg="BPF template directory: NOT OK. Please run 'make install-bpf'" runDirectory=/tmp/run/state libDirectory=/tmp/run/lib BPFDirectory=/tmp/run/lib/bpf StateDirectory=/tmp/run/state/state error="stat /tmp/run/lib/bpf: no such file or directory"

I ran the target with a private DESTDIR, discarded its very long command echo, and counted the installed files. The count is only a sanity check that the whole tree was copied, not a test of its content, and nothing was written to the real /var/lib:

cd /tmp/cilium && make install-bpf DESTDIR=/tmp/run > /dev/null && find /tmp/run/var/lib/cilium/bpf -type f | wc -l
163

Failure three: no IPv6 in the kernel

With the sources in place, I pointed --lib-dir at the new location. The agent got further, then failed while installing a routing rule for IPv6, which matches the missing ipv6 entry shown earlier. As before, I kept only the fatal line of the log so the article does not need to reproduce the long flag listing:

cd /tmp && ip netns exec cil /tmp/cilium-agent --enable-k8s=false --ipam=cluster-pool --devices=veth-a --state-dir=/tmp/run/state --lib-dir=/tmp/run/var/lib/cilium 2>&1 | grep 'level=fatal' | cut -c1-400
time=2026-11-18T12:52:43.984405759Z level=fatal msg="unable to run agent: failed to start: failed to ensure local routing rule: replace local ipv6 rule: address family not supported by protocol" subsys=daemon

Adding --enable-ipv6=false removes the IPv6 rule, and the agent stays up. I started it in the background, with the same state and library directories as before, and then waited about 45 seconds for startup to finish before sending any traffic through the device:

cd /tmp && nohup ip netns exec cil /tmp/cilium-agent --enable-k8s=false --ipam=cluster-pool --enable-ipv6=false --devices=veth-a --enable-health-checking=false --state-dir=/tmp/run/state --lib-dir=/tmp/run/var/lib/cilium > /tmp/agent5.log 2>&1 &

Verify with a real request

I started a Python web server inside the namespace on 10.99.0.2:8080, then requested it from the host side of the veth pair. The request enters through veth-a, where Cilium's BPF program is attached. The server takes a moment to start, so wait before the first request:

cd /tmp && nohup ip netns exec cil python3 -m http.server 8080 --bind 10.99.0.2 > /tmp/http.log 2>&1 &
curl -s -m 5 -o /dev/null -w '%{http_code}\n' http://10.99.0.2:8080/
200

The datapath counters show packets going through Cilium's program on the host device, and the agent reports itself healthy. I used nsenter to join the agent's network and mount namespaces for the metrics command, so that the CLI looks at the same BPF filesystem the agent uses. The second command prints the agent's own status summary:

nsenter -t $(pgrep -f '^/tmp/cilium-agent') -n -m /tmp/cilium-dbg bpf metrics list
REASON      DIRECTION   PACKETS   BYTES   LINE   FILE
Interface   INGRESS     7         547     1200   bpf_host.c
Success     EGRESS      11        8922    479    nodeport_egress.h
ip netns exec cil /tmp/cilium-dbg status | grep -E 'Host firewall|Cilium:|Controller Status|Attach Mode'
Host firewall:           Disabled
Cilium:                  Ok    (v-)
Controller Status:       8/8 healthy
Attach Mode:             TCX

Prove that it enforces something

A 200 only shows that traffic passes. To see enforcement, I stopped the agent and restarted it with the host firewall on and the policy mode forced to always. I expected this to make the host endpoint default-deny for ingress, since no rule selects it. The status confirms the firewall is active on the veth device:

cd /tmp && nohup ip netns exec cil /tmp/cilium-agent --enable-k8s=false --ipam=cluster-pool --enable-ipv6=false --devices=veth-a --enable-host-firewall=true --enable-policy=always --enable-health-checking=false --state-dir=/tmp/run/state --lib-dir=/tmp/run/var/lib/cilium > /tmp/agent3.log 2>&1 &
ip netns exec cil /tmp/cilium-dbg status | grep -E 'Host firewall|Cilium:|Controller Status'
Host firewall:           Enabled    [veth-a]
Cilium:                  Ok    (v-)
Controller Status:       8/8 healthy

The same request, with the same Python server still running, now times out, although nothing about the server changed between the two runs. The status code 000 is what curl prints when no HTTP response arrived within the five-second limit I gave it:

curl -s -m 5 -o /dev/null -w '%{http_code}\n' http://10.99.0.2:8080/
000

To see why, I watched drop events with the monitor while sending one more request. The SYN from 10.99.0.1 is classified as identity 2 (the reserved "world" identity) and dropped with "Policy denied" in bpf_host.c. So the datapath, not the Python server, stops the connection. The monitor printed three drop lines for the same source port; I show one:

timeout 8 nsenter -t $(pgrep -f '^/tmp/cilium-agent') -n -m /tmp/cilium-dbg monitor -n --type drop > /tmp/monitor4.log 2>&1 & sleep 2; curl -s -m 3 -o /dev/null -w '%{http_code}\n' http://10.99.0.2:8080/; sleep 7; grep -v '^time' /tmp/monitor4.log | cut -c1-250 | head -4
xx drop (Policy denied) flow 0xf09e3cf4 to endpoint 0, ifindex 6, file bpf_host.c:915, , identity 2->0: 10.99.0.1:41326 -> 10.99.0.2:8080 tcp SYN

Unit tests

I also ran a small set of fast unit-test packages with the repository's Go 1.27 toolchain. This was not the full suite, which I did not attempt. The "(cached)" markers mean an identical run had passed earlier in the session. I show only the packages that were not cached or that matter most:

cd /tmp/cilium && go test ./pkg/labels/... ./pkg/identity/... ./pkg/option/... ./pkg/policy/api/... 2>&1 | grep -v 'no test files'
ok  	github.com/cilium/cilium/pkg/labels	(cached)
ok  	github.com/cilium/cilium/pkg/identity	(cached)
ok  	github.com/cilium/cilium/pkg/option	0.013s
ok  	github.com/cilium/cilium/pkg/policy/api	(cached)

Limits and alternatives

What worked: with Go 1.27 fetched automatically, the BPF sources installed under a private prefix, and IPv6 turned off, the agent from main ran on this Linux 6.18 machine, attached its programs through TCX to the veth device, passed a real HTTP request, and dropped it again once default-deny was on for the host endpoint.

What this does not show: it was one node, in a network namespace, without Kubernetes, pods, CNI integration or a real policy rule, so my enforcement test used default-deny, not a precise allow rule. I did not run the full test suite, the BPF unit tests, encryption or any multi-node setup. For anything beyond a lab, use the project's documented Kubernetes installation with released images. The host firewall guide recommends audit mode before enforcing host policies, and that applies here too, because a default-deny host policy on a real machine can lock you out.