GitOps Workflow Best Practices: Surviving Production

I still remember the silence. It was 2019, and a junior dev had just run kubectl apply -f deployment.yaml from their laptop. They targeted the wrong context. Production went dark. The rollback took 45 minutes because nobody knew exactly what version was running before the crash. That is why we do GitOps.

If you are SSH-ing into your servers to make changes in 2022, you are doing it wrong. Manual intervention is a liability. In the Nordic market, where reliability is currency and Datatilsynet (The Norwegian Data Protection Authority) watches data sovereignty like a hawk, you need a deterministic, auditable workflow.

This guide isn't high-level theory. We are going to build a production-grade GitOps pipeline using Flux v2 on a lightweight K3s cluster. We will handle the tricky stuff: secret management, directory structure, and the underlying infrastructure requirements that most VPS providers ignore.

The Core Principle: Git is the Single Source of Truth

The concept is simple: The state of your infrastructure must match the state of your Git repository. Always. If a drift occurs—someone manually changes a replica count—the GitOps controller (Flux or ArgoCD) detects it and reverts it immediately.

Pro Tip: Don't start with a massive multi-cluster setup. Start with a single "management" cluster that manages itself. Complexity is the enemy of stability.

Step 1: The Infrastructure Layer (Latency Matters)

GitOps is chatty. Your cluster controllers are constantly reconciling state, pulling from Git, and writing to etcd. If you run this on cheap storage with high I/O wait, your reconciliation loops will lag.

I've seen etcd timeouts bring down entire clusters because the underlying disk couldn't keep up with the fsync latency. For this setup, we use CoolVDS NVMe instances. Why? Because when you are reconciling hundreds of YAML manifests, random read/write speeds dictate how fast your cluster heals. Plus, keeping the data in Norway (Oslo region) simplifies GDPR compliance significantly post-Schrems II.

Setting up the Host

Assuming you are on a fresh Ubuntu 20.04 LTS instance. First, disable swap. Kubernetes hates swap.

Next, install K3s. It’s lightweight, CNCF certified, and perfect for VDS environments where you don't want to waste 2GB RAM just on the control plane.

Step 2: Bootstrapping Flux v2

We prefer Flux v2 over ArgoCD for smaller, headless setups because it feels more native to the Kubernetes API. It breaks the monolith into the GitOps Toolkit (GOTK).

First, install the CLI (checked against v0.24.0, current as of Jan 2022):

Now, bootstrap the cluster. This command creates a repository on your GitHub/GitLab, installs the Flux components on your cluster, and configures the cluster to watch that repo. It’s a chicken-and-egg problem, solved.

export GITHUB_TOKEN=<your-token>
export GITHUB_USER=<your-username>

flux bootstrap github \
  --owner=$GITHUB_USER \
  --repository=fleet-infra \
  --branch=main \
  --path=./clusters/production \
  --personal

If this completes successfully, you now have a cluster that updates itself based on the ./clusters/production directory in that repo.

Step 3: Repository Structure & Kustomize

Don't dump everything in one folder. Use Kustomize to handle environment variations. Here is the structure I use for high-availability setups:

├── apps
│   ├── base
│   │   └── nginx
│   ├── production
│   │   └── nginx
│   └── staging
│       └── nginx
├── infrastructure
│   ├── monitoring
│   └── cert-manager
└── clusters
    ├── production
    │   ├── apps.yaml
    │   └── infrastructure.yaml
    └── staging
        ├── apps.yaml
        └── infrastructure.yaml

In clusters/production/apps.yaml, you define a Kustomization CRD that points to apps/production. This keeps your cluster config separate from your application definitions.

Step 4: The Secret Management Nightmare

You cannot commit raw secrets to Git. If you do, your repository is compromised. Period. You have two main choices in 2022: HashiCorp Vault (complex, expensive to run) or Mozilla SOPS (Simple, effective).

We use SOPS with Age encryption. It encrypts values in your YAML files but leaves keys readable, so you can still diff the structure.

Encrypting a Secret

1. Install SOPS and Age.
2. Generate a key:

3. Create a Kubernetes secret yaml locally, then encrypt it:

# secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: database-creds
  namespace: default
stringData:
  password: "SuperSecretPassword123"

Encrypt it in place:

Now you can safely commit secret.yaml. Configure Flux to decrypt this inside the cluster using the private key (stored as a K8s secret in the flux-system namespace).

Step 5: Handling Norwegian Compliance (GDPR/Schrems II)

Legal compliance is a technical requirement. Since the Schrems II ruling, transferring personal data to US-owned clouds is legally risky. By using a GitOps workflow on CoolVDS, you ensure data sovereignty.

• Data Residency: Your PV (Persistent Volumes) are on local RAID-10 NVMe arrays in Norway.
• Traffic Routing: Peer directly at NIX (Norwegian Internet Exchange) for low latency and to keep local traffic local.
• Encryption: Using SOPS ensures that even if your Git provider (GitHub/GitLab) is compromised, the data remains encrypted.

Performance Tuning for K8s on VDS

Kubernetes defaults are often set for massive bare-metal servers. On a VDS, you need to tune the kubelet to avoid CPU stealing and OOM kills.

Add these flags to your K3s server config to reserve resources for the system, preventing the "noisy neighbor" effect if you are not using dedicated cores:

# /etc/systemd/system/k3s.service
ExecStart=/usr/local/bin/k3s server \
  --kubelet-arg="system-reserved=cpu=500m,memory=512Mi" \
  --kubelet-arg="kube-reserved=cpu=500m,memory=512Mi"

This ensures that even if your application spikes, your SSH session and Kubelet agent stay alive.

Summary

GitOps is not just a buzzword; it's an insurance policy. It allows you to sleep at night knowing that your infrastructure is documented, versioned, and self-healing. While the software layer (Flux, K3s) handles the logic, the physical layer dictates the reliability.

Don't let slow I/O kill your reconciliation loops. If you are ready to build a pipeline that is as fast as it is secure, deploy a test instance. You can spin up a high-performance NVMe KVM node on CoolVDS in under 55 seconds.