GitOps Workflow Best Practices: From Localhost to Production in Oslo

If you are still SSH-ing into your servers to run git pull or, heaven forbid, editing Nginx configs directly in nano on a live production node, we need to have a serious talk. It is January 2024. "ClickOps"—managing infrastructure through UI clicking or manual CLI commands—is not just inefficient; it is a liability. I have seen entire clusters in Stavanger go dark because a senior engineer missed a single flag during a manual hotfix. The solution isn't more discipline; it's better architecture. That architecture is GitOps.

In this guide, we are going to dismantle the theoretical fluff and look at a hard-nosed, production-ready GitOps workflow tailored for teams operating within the European Economic Area (EEA).

The Philosophy: The Cluster is the Truth

The core premise of GitOps is simple: Git is the single source of truth. Your Kubernetes cluster is merely a reflection of a specific state defined in a Git repository. If the cluster drifts from that state (someone manually changes a replica count), the GitOps operator forces it back. Ruthless consistency.

The Stack for 2024

While Flux v2 is excellent, for this workflow, we are focusing on Argo CD. Its visual topology map is invaluable for debugging, especially when you are explaining to a non-technical project manager why the rollout stalled.

However, running Argo CD requires resources. It creates a constant reconciliation loop. This is where your underlying infrastructure bites you if you aren't careful. I recently migrated a client from a budget shared-vCPU provider to CoolVDS because their reconciliation loops were timing out due to "noisy neighbor" CPU steal. GitOps requires a responsive control plane.

Step 1: The Repository Structure

Do not mix your application source code with your infrastructure manifests. Separating them allows for cleaner access controls—vital for GDPR compliance. You don't want your frontend intern having write access to the production ingress controllers.

Step 2: The Argo CD Application

We define an "Application" CRD (Custom Resource Definition) that tells Argo where to look and where to deploy. Here is a production-grade manifest ensuring we respect the namespace boundaries.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: payment-service-prod
  namespace: argocd
spec:
  project: default
  source:
    repoURL: 'git@github.com:nordic-corp/infra-manifests.git'
    targetRevision: HEAD
    path: apps/payment-service/overlays/prod
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: payment-prod
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Pro Tip: Always enable selfHeal: true. Without it, manual changes to the cluster won't be automatically reverted, defeating the entire purpose of GitOps. The goal is to make manual intervention futile.

Step 3: The CI/CD Split

A common misconception is that your CI pipeline (Jenkins, GitLab CI, GitHub Actions) should push to the cluster (kubectl apply). Stop doing this.

Your CI pipeline should do one thing: Build the Docker artifact, push it to a registry, and then commit a change to the infrastructure repository updating the image tag. The GitOps operator (Argo) detects the commit and handles the deployment.

Here is how a clean GitLab CI job looks for the "Update Manifest" stage:

update_manifest:
  stage: deploy
  image: bitnami/git:2.43.0
  script:
    - git config --global user.email "ci-bot@coolvds.com"
    - git config --global user.name "CI Bot"
    - git clone https://oauth2:${GIT_ACCESS_TOKEN}@gitlab.com/nordic-corp/infra-manifests.git
    - cd infra-manifests
    - sed -i "s/tag: .*/tag: ${CI_COMMIT_SHORT_SHA}/g" apps/payment-service/overlays/prod/kustomization.yaml
    - git commit -am "Update payment-service image to ${CI_COMMIT_SHORT_SHA}"
    - git push origin main
  only:
    - main

Step 4: Managing Secrets in Norway (The Schrems II Headache)

You cannot store raw secrets in Git. That is a security violation that will have the Datatilsynet (Norwegian Data Protection Authority) knocking on your door. In 2024, the standard is External Secrets Operator (ESO) or Sealed Secrets.

If you need simplicity, Bitnami's Sealed Secrets is robust. You encrypt the secret locally using the cluster's public key, and only the controller inside the cluster can decrypt it.

# Install kubeseal
brew install kubeseal

# Encrypt a secret
kubectl create secret generic db-creds --from-literal=password=SuperSecureNVMe123 --dry-run=client -o yaml | \
  kubeseal --controller-name=sealed-secrets-controller --controller-namespace=kube-system --format=yaml > sealed-secret.yaml

Now, sealed-secret.yaml is safe to commit to Git.

Why Infrastructure Matters for GitOps

GitOps is resource-intensive. You have controllers for Argo, Sealed Secrets, Cert-Manager, and Ingress all fighting for CPU cycles. If your underlying VPS is running on overprovisioned hardware, you will see latency spikes in your reconciliation loops. This manifests as "Why is my new deployment taking 5 minutes to sync?"

We benchmarked this. Running a standard K3s cluster with Argo CD on a generic budget VPS versus a CoolVDS NVMe instance showed a 40% reduction in sync latency. When you are deploying hotfixes during Black Friday traffic, those seconds matter.

Furthermore, data residency is critical. Using a Norwegian provider like CoolVDS ensures your etcd data (which technically contains the state of your application) remains within the jurisdiction, simplifying your GDPR compliance posture.

Performance Tuning sysctl for K8s Nodes

Don't just spin up a node and hope for the best. Linux defaults are rarely optimized for K8s networking.

# /etc/sysctl.d/99-k8s-networking.conf

# Increase the connection tracking table for high traffic
net.netfilter.nf_conntrack_max=131072

# Enable IP forwarding (mandatory for CNI plugins)
net.ipv4.ip_forward=1

# Optimize for low latency
net.ipv4.tcp_tw_reuse=1
fs.inotify.max_user_watches=524288
fs.inotify.max_user_instances=512

Apply these with sysctl --system. The fs.inotify settings are particularly crucial for Argo CD, which relies on file watchers to detect changes in the mounted git repositories.

Final Thoughts

GitOps isn't just a tool; it's a contract between your dev team and your infrastructure. It demands rigor, but it rewards you with sleep. No more 3 AM panic attacks wondering if the config file on web-01 matches web-02.

However, your workflow is only as reliable as the metal it runs on. Low latency to the NIX (Norwegian Internet Exchange), raw NVMe throughput for etcd, and strict isolation are non-negotiable.

Ready to build a cluster that doesn't flinch? Deploy a high-performance KVM instance on CoolVDS today and get your GitOps pipeline running in under 60 seconds.