GitOps Architecture: Stop SSH-ing into Production

If you are still SSH-ing into your servers to run kubectl apply or, god forbid, editing files with vim directly in /etc/, you are not managing infrastructure. You are gambling. I have seen production environments in Oslo melt down simply because a senior engineer “fixed” a config drift manually and forgot to commit it to the repo. Three weeks later, the CI pipeline overwrote the fix, and the database connection pool collapsed.

It is June 2023. We have tools like Argo CD and Flux v2. There is no excuse for infrastructure drift.

This guide isn't about the philosophy of DevOps. It is about the hard technical reality of implementing GitOps in a way that handles high throughput without failing when your control plane gets busy. We are going to look at the stack, the code, and the hardware requirements that most cloud providers hide from you.

The Core Loop: Why Latency Kills GitOps

GitOps is simple in theory: Git is the single source of truth. An agent inside your cluster (like Argo CD) pulls the state from Git and forces the cluster to match it.

In practice, this creates a massive amount of I/O pressure on your Kubernetes control plane. Every reconciliation loop triggers queries to the API server, which hits etcd. If you are running this on a budget VPS with shared HDD or throttled SSDs, your etcd latency spikes. When etcd fsync latency exceeds 10ms, your cluster becomes unstable. Leader elections fail. Pods get evicted.

Pro Tip: Always benchmark your storage before deploying a GitOps controller. Use fio to test fsync performance specifically, not just throughput. On CoolVDS NVMe instances, we typically see fsync latencies under 0.5ms, which is mandatory for a healthy K8s control plane.

Benchmarking Your Disk for Etcd

Before you install Argo, run this. If your 99th percentile fdatasync latency is over 10ms, change providers immediately.

fio --rw=write --ioengine=sync --fdatasync=1 --directory=test-data --size=22m --bs=2300 --name=mytest

The Stack: Kubernetes 1.27 + Argo CD

We are using Kubernetes 1.27 (released April 2023) because of the stabilization of the ptr library and better sidecar container support. For the CD controller, we stick with Argo CD v2.7 due to its superior UI and multi-cluster visibility.

Directory Structure Strategy

Don't dump everything into one folder. Use the Kustomize Base/Overlay pattern to manage environment differences between Staging (perhaps a smaller CoolVDS instance) and Production.

├── base
│   ├── deployment.yaml
│   └── service.yaml
└── overlays
    ├── production
    │   ├── kustomization.yaml
    │   └── patch-replicas.yaml
    └── staging
        ├── kustomization.yaml
        └── patch-resources.yaml

The App of Apps Pattern

To bootstrap the cluster, we don't apply applications one by one. We define a root “Application” that points to a folder containing other Applications. This is the inception of GitOps.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: infrastructure-root
  namespace: argocd
spec:
  project: default
  source:
    repoURL: 'git@github.com:your-org/infra-repo.git'
    targetRevision: HEAD
    path: applications
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Notice the selfHeal: true flag. This is the “fix it” switch. If someone manually changes a Service definition on the cluster, Argo CD immediately reverts it. This enforces discipline.

Handling Secrets without Leaking Them

You cannot store raw secrets in Git. In 2023, the standard is Sealed Secrets or External Secrets Operator (integrating with Vault or AWS SSM). For a self-hosted environment on CoolVDS, Bitnami's Sealed Secrets is the most efficient path to minimal complexity.

1. Install the controller on the cluster.
2. Download the public key.
3. Encrypt your secret locally.

# Create a raw secret (dry run)
kubectl create secret generic database-creds --from-literal=password=SuperSecret123 --dry-run=client -o yaml > secret.yaml

# Seal it
kubeseal --format=yaml --cert=public-cert.pem < secret.yaml > sealed-secret.yaml

You can commit sealed-secret.yaml safely. Only the controller running on your secure VPS can decrypt it.

The Norwegian Context: Latency and Law

Hosting in Norway isn't just about patriotism; it's about physics and law. If your development team is in Oslo or Bergen, the round-trip time (RTT) to your GitOps controller matters. A bloated Argo CD UI loading from a US East server feels sluggish. Running it on a CoolVDS instance peered at NIX (Norwegian Internet Exchange) keeps your management plane snappy.

More importantly, Datatilsynet (The Norwegian Data Protection Authority) is increasingly strict about where data processing happens. Even if your customer data is in Norway, if your deployment metadata and secrets are transiting through non-EU servers, you are opening a compliance risk vector. Keeping the entire GitOps pipeline—GitLab Runner, Registry, and K8s Cluster—on sovereign infrastructure simplifies your GDPR posture significantly.

Infrastructure Reliability

GitOps is automated, but it is not magic. It relies on the underlying compute being rock solid. I have debugged clusters where “flaky” deployments were actually caused by CPU steal time on oversold host nodes. The Kubelet stopped reporting status, and the deployment rolled back unnecessarily.

This is where the choice of virtualization matters. We use KVM at CoolVDS to ensure strict isolation. When you run a heavy CI/CD build or a massive helm upgrade, you need guaranteed CPU cycles, not “burstable” credits that run out halfway through a compilation.

Final Verdict

GitOps is the only way to scale operations without scaling headcount. But it demands a rigorous approach to storage performance and network stability. Don't let IOPS wait times bottle-neck your deployment frequency.

Ready to build a pipeline that doesn't break? Spin up a Performance NVMe VPS on CoolVDS today and test the fio benchmarks yourself. Speed is a feature you cannot patch in later.