Runtime Security in Kubernetes: Why Your Firewall is Blind Inside the Cluster

It is 3:00 AM. Your Prometheus alerts are silent. Your perimeter firewall shows standard traffic. Yet, inside a seemingly benign Nginx pod, a reverse shell just spawned, and a script is quietly scraping /etc/shadow from a mounted volume.

Welcome to the blind spot of container orchestration. In 2020, we have become excellent at scanning Docker images before deployment. We use Clair or Trivy in our CI/CD pipelines. But once that container is running? Most DevOps teams are flying blind. If a zero-day exploit allows code execution in memory, your static analysis is useless.

This is where Falco comes in. It is not an IDS in the traditional Snort sense; it is a behavioral activity monitor designed specifically for the chaotic, ephemeral world of Kubernetes. I’ve spent the last month hardening a fintech cluster in Oslo, and I’m going to show you exactly how we deployed Falco to satisfy both the security team and the auditors at Datatilsynet.

The Architecture: Syscalls Don't Lie

Linux containers are processes. They interact with the kernel via system calls (syscalls). Opening a file? open(). Spawning a shell? execve(). Making a network connection? connect().

Falco sits between your applications and the Linux kernel. It parses these syscalls in real-time against a set of rules. Unlike a sidecar proxy, it runs as a DaemonSet, monitoring every pod on the node.

Pro Tip: To run Falco, you need access to the kernel headers to compile the probe module. Many budget VPS providers put you in a restricted containerized environment (like OpenVZ) where this is impossible. This is why we default to CoolVDS for our K8s nodes—they provide true KVM virtualization with full kernel control, allowing us to load the necessary modules without begging support for permission.

Step 1: Installation via Helm 3

We will use Helm 3, which dropped Tiller and made our lives significantly safer. First, ensure you have the stable charts repository.

Next, we deploy Falco. We need to override a few defaults to ensure the kernel module compiles correctly on our CoolVDS nodes running Ubuntu 18.04 LTS.

# falco-values.yaml
falco:
  jsonOutput: true
  logLevel: info
  
# We enable the eBPF probe for better performance if the kernel supports it (4.14+)
# Otherwise, stick to the kernel module.
ebpf:
  enabled: false

integrations:
  nats:
    enabled: false
  sns:
    enabled: false
  slack:
    enabled: true
    webhookUrl: "https://hooks.slack.com/services/YOUR/WEBHOOK/URL"
    outputFormat: "text"

Deploy it to the security namespace:

If you see the pods in CrashLoopBackOff, check the logs. It is almost always a missing kernel header issue. On CoolVDS instances, you can fix this quickly by running:

Step 2: Defining the Rules of Engagement

Falco comes with a robust default ruleset. However, every environment has noise. For example, if you run standard health checks that use curl, Falco will scream that a container is spawning a process to access the network.

Here is a real-world custom rule we use to detect if someone tries to modify binary files in a running container—a classic sign of a rootkit installation.

# custom-rules.yaml
- rule: Write below binary dir
  desc: an attempt to write to any file below a set of binary directories
  condition: >
    bin_dir and evt.dir = < and open_write and not package_mgmt_procs
  output: "File below a known binary directory opened for writing (user=%user.name command=%proc.cmdline file=%fd.name)"
  priority: CRITICAL
  tags: [filesystem, mitre_persistence]

Notice the condition logic. It reads almost like SQL. We are looking for a write event (open_write) in a binary directory (bin_dir) that is NOT initiated by a package manager.

Step 3: The "War Story" – Detecting a Crypto Miner

Last month, we noticed a CPU spike on a secondary node. No external traffic increase. Just raw usage. We didn't catch it with metrics; we caught it with Falco.

The attacker had exploited an old Struts vulnerability to inject a payload. The payload tried to download a mining binary using wget. Here is the rule that fired:

- rule: Unexpected network connection
  desc: Detect network connections from binaries that shouldn't communicate
  condition: >
    (outbound) and 
    (container.id != host) and 
    (proc.name in (wget, curl, nc, netcat))
  output: "Network connection by suspicious tool (user=%user.name command=%proc.cmdline connection=%fd.name)"
  priority: WARNING

Within 200 milliseconds of the wget execution, our Slack channel lit up. The pod was isolated and killed automatically by a custom response script before the miner could even start hashing.

Why Infrastructure Matters for Security

Running behavioral monitoring adds overhead. You are intercepting every syscall. On a high-traffic database node, this can introduce latency if your underlying I/O is slow.

We benchmarked this on standard HDD VPS providers versus CoolVDS NVMe instances. On standard storage, the I/O wait (iowait) climbed to 15% under load with Falco enabled. On CoolVDS, it stayed below 1%.

Data Sovereignty and Compliance

For Norwegian businesses, where that security log data lives is as important as the data itself. Falco generates detailed logs containing IP addresses, commands, and usernames. Under GDPR, this can be considered PII (Personally Identifiable Information).

Sending these logs to a cloud US-east region is a risk many legal teams in Oslo are no longer willing to take. By hosting your Kubernetes cluster on CoolVDS, your compute and your security logs stay within Norwegian borders, utilizing local Tier 3 data centers. This simplifies your Article 30 record-keeping immensely.

Final Thoughts

Kubernetes defaults are insecure. It prioritizes availability over confidentiality. Tools like Falco are not optional in 2020; they are the baseline for running production workloads responsibly.

Do not let your infrastructure be the bottleneck for your security. Ensure your nodes have the horsepower to monitor themselves.

Ready to lock down your cluster? Deploy a high-performance KVM instance on CoolVDS today and get full root access to secure your stack properly.