Console Login

GitOps is Not a Silver Bullet: It’s the Gun That Prevents You from Shooting Your Foot

GitOps is Not a Silver Bullet: It’s the Gun That Prevents You from Shooting Your Foot

I still remember the silence in the Slack channel. It was 2021, and a "quick fix" applied manually via kubectl had just wiped the ingress configuration for a major Norwegian e-commerce site during Black Friday. The latency graph didn't just spike; it disappeared. We spent four hours reconciling the state of the cluster with what we thought was in the repository.

That is why we do GitOps. Not because it’s trendy, but because manual intervention in production is an operational hazard.

If you are deploying to Kubernetes in 2024 without a reconciliation loop, you are effectively gambling. This guide covers a battle-tested GitOps workflow using ArgoCD, focusing on the specific needs of Norwegian infrastructure—where data sovereignty (Datatilsynet) and NIX latency actually matter.

The Architecture of Truth

The core premise is simple: Git is the single source of truth. If it’s not in the repo, it doesn’t exist in the cluster. But implementing this requires rigorous discipline and reliable underlying compute.

1. The Repository Structure

Don't dump everything into one repo. Separation of concerns saves you when your team scales. I recommend a split-repo pattern:

  • App Repo: Source code + Dockerfile + CI pipeline (Build & Push).
  • Config Repo (Infrastructure): Helm charts, Kustomize files, and ArgoCD manifests.

This separation prevents a CI loop on the application code from triggering an accidental infrastructure rollout. Here is a directory structure that survives audits:


/config-repo
├── /base
│   ├── deployment.yaml
│   ├── service.yaml
│   └── kustomization.yaml
└── /overlays
    ├── /oslo-prod
    │   ├── patch-resources.yaml
    │   └── kustomization.yaml
    └── /bergen-staging
        ├── patch-replicas.yaml
        └── kustomization.yaml

The Reconciliation Engine: ArgoCD

We use ArgoCD over Flux for one reason: visibility. The UI provides a sanity check that CLI tools lack. However, the engine is heavy. It constantly polls your Git repositories.

Pro Tip: If you host ArgoCD on a cheap, shared VPS, you will see "Context Deadline Exceeded" errors. The repo server does heavy lifting (cloning/hashing). We run our control planes on CoolVDS NVMe instances because the high IOPS ensures that git fetch operations complete before the timeout window, even with monorepos approaching 2GB.

Deploying the Application Manifest

Here is a production-grade Application manifest. Note the automated sync policy—we want the cluster to heal itself, but we disable selfHeal initially to prevent fighting during migrations.


apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: payment-gateway-oslo
  namespace: argocd
spec:
  project: default
  source:
    repoURL: 'git@gitlab.com:your-org/infra-config.git'
    targetRevision: HEAD
    path: overlays/oslo-prod
  destination:
    server: https://kubernetes.default.svc
    namespace: payments
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - PruneLast=true

The CI/CD Handshake

Your CI pipeline (GitLab CI or GitHub Actions) should never touch kubectl. Its only job is to build the container, push it to the registry, and update the image tag in the Config Repo. The GitOps operator takes it from there.

Here is how a clean GitLab CI job looks for the tag update. This script assumes you are using a strictly versioned workflow.


update_manifest:
  stage: deploy
  image: alpine:3.19
  before_script:
    - apk add --no-cache git yq
    - git config --global user.email "ci-bot@coolvds.com"
    - git config --global user.name "CI Bot"
  script:
    - git clone https://oauth2:${CI_ACCESS_TOKEN}@gitlab.com/your-org/infra-config.git
    - cd infra-config/overlays/oslo-prod
    # Use yq to safely update the tag without breaking formatting
    - yq eval ".images[0].newTag = \"$CI_COMMIT_SHORT_SHA\"" -i kustomization.yaml
    - git commit -am "Update image tag to $CI_COMMIT_SHORT_SHA"
    - git push origin main
  only:
    - main

Handling Secrets without Leaking Data

You cannot commit raw secrets to Git. If you do, your GDPR compliance is void immediately. In 2024, the standard is External Secrets Operator (ESO) integration with a vault, but for smaller setups, Sealed Secrets is acceptable.

If you are hosting data in Norway to comply with Schrems II, ensure your secret management backend (like HashiCorp Vault) is running on infrastructure within the EEA. We often deploy Vault on a dedicated CoolVDS instance with a private network link to the Kubernetes cluster. This keeps the latency between the application and the secrets engine under 1ms.

The