GitOps is Not Just "git push": Architecting Resilience for High-Traffic Nodes

I still see it happen. A senior engineer SSHs into a production node, runs kubectl edit deployment, and patches a memory leak on the fly. It fixes the immediate issue. Then, three weeks later, the cluster autoscales, the pod restarts, and the fix vanishes. The service crashes during the Friday peak.

If this sounds familiar, your infrastructure is fragile. In the Nordic market, where reliability is expected to rival the power grid and data sovereignty is legally mandated, manual interventions are a liability.

We are going to dismantle the "ClickOps" mentality. This is a technical blueprint for a GitOps workflow that actually survives production, built on the assumption that you are running on solid infrastructure like CoolVDS NVMe instances.

The Core Principle: Drift is the Enemy

GitOps isn't just about deploying code; it's about state reconciliation. The state of your infrastructure in git must match the state in the cluster. If it doesn't, the cluster is wrong, not the repo.

To achieve this in 2023, we rely on pull-based mechanisms. Your CI server (Jenkins, GitLab CI) should never have kubectl access to your production cluster. Giving CI push access creates a security blast radius that keeps CISOs awake at night. Instead, an agent inside the cluster pulls the changes.

The Stack: ArgoCD + Kustomize

For this architecture, we use ArgoCD. It handles the "last mile" of deployment better than Flux v2 for teams requiring a visual topology of their microservices.

1. The Directory Structure

Do not mix application source code with infrastructure manifests. Keep them separate to avoid infinite build loops.

├── apps/
│   ├── base/
│   │   ├── deployment.yaml
│   │   ├── service.yaml
│   └── overlays/
│       ├── production/
│       │   ├── kustomization.yaml
│       │   └── patch-replicas.yaml
│       └── staging/

2. The ArgoCD Application Manifest

Here is the exact configuration we use to ensure automated pruning (deleting resources removed from Git) and self-healing (overwriting manual changes).

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: payment-gateway-oslo
  namespace: argocd
spec:
  project: default
  source:
    repoURL: 'git@gitlab.com:your-org/infra-manifests.git'
    targetRevision: HEAD
    path: apps/overlays/production
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: payments
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - PruneLast=true

Pro Tip: Enable selfHeal: true. If a cowboy admin changes a setting manually, ArgoCD will immediately revert it. This enforces discipline through code.

The Hardware Reality: Why ETCD Needs NVMe

Kubernetes is only as fast as its brain, etcd. Etcd is incredibly sensitive to disk write latency. If fsync latency spikes, the cluster leader election fails, and your control plane goes down.

I've debugged clusters on budget VPS providers where "SSD" actually meant a throttled SATA share on a crowded SAN. The result is random API timeouts. This is why we deploy K8s control planes on CoolVDS. We get raw NVMe passthrough. The I/O latency is consistently sub-millisecond.

Here is how you verify your disk latency matches the requirement for a stable etcd (target is <10ms 99th percentile):

fio --rw=write --ioengine=sync --fdatasync=1 --directory=test-data --size=22m --bs=2300 --name=mytest

On a standard CoolVDS instance, you'll see sync times that make bare metal jealous. On legacy cloud providers, you'll see why your pods are flapping.

Handling Secrets without Leaking Them

You cannot commit secret.yaml to Git. In Feb 2023, the standard for this is Bitnami Sealed Secrets. It uses asymmetric encryption. You encrypt with a public key (safe for Git), and only the controller inside the cluster (which has the private key) can decrypt it.

Workflow:

1. Developer: kubeseal --format=yaml < secret.yaml > sealedsecret.yaml
2. Git: Commit sealedsecret.yaml.
3. ArgoCD: Deploys the SealedSecret custom resource.
4. Controller: Decrypts and creates the native K8s Secret.

The Norwegian Context: Latency and Law

Why host this in Oslo? Two reasons: Physics and Schrems II.

1. Latency: If your user base is in Scandinavia, routing traffic to Frankfurt adds 15-25ms. Routing to US-East adds 80ms+. With CoolVDS located in Oslo, your RTT (Round Trip Time) to local ISPs like Telenor or Telia is often under 5ms. For high-frequency trading or real-time gaming backends, this is non-negotiable.

2. GDPR & Datatilsynet: Since the Schrems II ruling, transferring personal data to US-owned cloud providers is a legal minefield. By running your GitOps worker nodes on local Norwegian infrastructure, you simplify your compliance posture significantly. The data rests here.

CI Integration: Separation of Concerns

Your CI pipeline (GitLab CI / GitHub Actions) has one job: Test code and build images. It should not deploy.

Here is a sanitized GitLab CI snippet. Note that it only updates the manifest repo, it does not touch the cluster.

build_image:
  stage: build
  script:
    - docker build -t registry.coolvds.com/app:$CI_COMMIT_SHA .
    - docker push registry.coolvds.com/app:$CI_COMMIT_SHA

update_manifests:
  stage: deploy
  image: alpine:3.17
  before_script:
    - apk add --no-cache git
    - eval $(ssh-agent -s)
    - echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
    - mkdir -p ~/.ssh
    - chmod 700 ~/.ssh
    - ssh-keyscan gitlab.com >> ~/.ssh/known_hosts
  script:
    - git clone git@gitlab.com:your-org/infra-manifests.git
    - cd infra-manifests/apps/overlays/production
    - kustomize edit set image app=registry.coolvds.com/app:$CI_COMMIT_SHA
    - git commit -am "Update image to $CI_COMMIT_SHA"
    - git push origin main

Disaster Recovery: The "Revert" Button

The beauty of this setup is the recovery time. If a bad configuration breaks the production ingress:

1. You identify the bad commit hash.
2. git revert <hash>
3. git push

ArgoCD sees the revert, detects the drift, and rolls the cluster back to the previous known good state. No backups to restore, no panicked SSH commands. Total resolution time: < 2 minutes.

Summary

GitOps requires discipline, but it pays dividends in sleep. However, the software layer is only as reliable as the virtualization below it. No amount of Kubernetes self-healing can fix a hypervisor with stolen CPU cycles or high I/O wait.

For your next cluster, choose infrastructure that respects the engineering rigor you put into your code. CoolVDS offers the low-latency, NVMe-backed foundation that modern DevOps workflows demand in the Nordic region.

Ready to lower your control plane latency? Deploy a high-performance KVM instance in Oslo today.