Stop SSH-ing into Production: A Battle-Hardened Guide to GitOps Workflows in 2024

If you are still running kubectl apply -f from your local laptop, you are a security risk. I don't care how secure your VPN is or how careful you think you are. Manual intervention is the root cause of configuration drift, and in a high-stakes environment—like the fintech platforms we see emerging out of Oslo—drift means downtime.

I've spent the last decade watching servers melt because a junior dev updated a config map manually and forgot to commit it. Two weeks later, the pod restarts, pulls the old config from Git, and the application crashes. This isn't theoretical; it's the reality of non-deterministic infrastructure.

Enter GitOps. It’s not just a buzzword; it’s the only sane way to manage modern infrastructure. But implementing it requires more than just installing ArgoCD. It requires a philosophy change and, crucially, the right underlying metal.

The Architecture: Pull vs. Push

In the old days (circa 2018), we used "Push" based CI/CD. Jenkins scripts would build a docker image and then run a command to update the cluster. The problem? Jenkins has god-mode access to your cluster. If your CI server is compromised, your production environment is toast.

In 2024, the standard is the "Pull" model. Your cluster reaches out to the Git repository to see what state it should be in. The cluster agent (ArgoCD or Flux) reconciles the difference.

Pro Tip: Place your GitOps controller on a management cluster or a dedicated, high-performance node. If your application creates a resource loop and starves the node of CPU, you don't want your deployment mechanism to freeze alongside it. This is why we recommend dedicated KVM slices on CoolVDS for control planes—OpenVZ or shared containers often choke on the heavy I/O of constant Git polling.

Directory Structure: The Monorepo Myth

Don't dump everything into one repo unless you enjoy merge conflicts. A split architecture is cleaner for compliance, especially when dealing with strict GDPR requirements mandated by Datatilsynet here in Norway.

Recommended Structure

├── apps-repo/              # Source code for microservices
│   ├── service-a/
│   │   ├── src/
│   │   └── Dockerfile
│   └── .github/workflows/  # CI builds image, doesn't touch cluster
│
└── infra-repo/             # The Source of Truth (Manifests)
    ├── base/               # Vanilla K8s manifests
    └── overlays/
        ├── staging/
        │   ├── kustomization.yaml
        │   └── patch-replicas.yaml
        └── production/     # Locked down branch
            ├── kustomization.yaml
            └── patch-resources.yaml

The Engine: Configuring ArgoCD

Let's look at a proper Application manifest. We aren't just pointing to a folder; we are defining sync policies and self-healing. Without selfHeal, you aren't doing GitOps; you're just doing fancy deployments.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: payment-gateway-prod
  namespace: argocd
spec:
  project: default
  source:
    repoURL: 'git@github.com:my-org/infra-repo.git'
    targetRevision: HEAD
    path: overlays/production
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: payments
  syncPolicy:
    automated:
      prune: true      # Deletes resources not in Git. Critical.
      selfHeal: true   # Reverts manual changes instantly.
    syncOptions:
      - CreateNamespace=true
      - ApplyOutOfSyncOnly=true

If someone manually edits the deployment to change the image tag, ArgoCD detects the drift and reverts it immediately. This is the immutability we crave.

Handling Secrets: The Elephant in the Room

You cannot check raw secrets into Git. If you do, your repository is compromised forever. In 2024, the two leading solutions are HashiCorp Vault (complex, enterprise) or Bitnami Sealed Secrets (simple, K8s-native).

For most mid-sized deployments, Sealed Secrets is efficient. It uses asymmetric encryption. You encrypt with a public key locally; only the controller inside the cluster (which holds the private key) can decrypt it.

Generating a Sealed Secret:

# Create a dry-run secret
kubectl create secret generic db-creds \
  --from-literal=password=SuperSecret123 \
  --dry-run=client -o yaml > secret.yaml

# Seal it using the public key fetched from the controller
kubeseal --format=yaml < secret.yaml > sealed-secret.yaml

# Now sealed-secret.yaml is safe to commit to Git

Infrastructure Performance: Why It Matters

GitOps controllers are chatty. They are constantly cloning repositories, diffing YAML, and querying the Kubernetes API. I’ve seen GitOps implementations on budget cloud providers fail because the disk I/O couldn't keep up with the git fetch operations for 50 microservices.

This is where hardware choice becomes non-negotiable. At CoolVDS, we utilize NVMe storage arrays with high IOPS ceilings specifically to handle this kind of bursty read/write pattern. When ArgoCD needs to compare 2,000 objects against the cluster state, latency matters.

Furthermore, network stability to the Nordic region is vital. If your cluster is hosted in Frankfurt but your Git provider (e.g., a self-hosted GitLab) is in Oslo, jitter can cause sync timeouts. CoolVDS’s peering at NIX (Norwegian Internet Exchange) ensures your latency remains in the single digits.

The CI Pipeline Integration

Your CI pipeline (GitHub Actions, GitLab CI) should strictly do two things: Run tests and build images. It should never touch kubectl. Instead, it commits a change to the infrastructure repo.

Here is a clean update pattern using yq to update the image tag in the infra repo:

name: Build and Deploy
on: 
  push:
    branches: [ "main" ]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    
    - name: Build and Push Docker Image
      run: |
        docker build -t registry.coolvds.com/app:${{ github.sha }} .
        docker push registry.coolvds.com/app:${{ github.sha }}

    - name: Update Infra Repo
      run: |
        git clone https://user:${{ secrets.PAT }}@github.com/my-org/infra-repo.git
        cd infra-repo/overlays/production
        # Update the kustomization image tag
        yq e -i '(.images[] | select(.name == "my-app")).newTag = "${{ github.sha }}"' kustomization.yaml
        git config user.name "CI Bot"
        git commit -am "Update image to ${{ github.sha }}"
        git push

Monitoring and Metrics

Once deployed, you need to know if the sync is healthy. ArgoCD exposes Prometheus metrics out of the box. Ensure you are scraping port 8082.

If you see workqueue_depth spiking, your control plane is underpowered. This is a common symptom of noisy neighbor effects on shared hosting platforms. Migrating the control plane to a CoolVDS instance with dedicated CPU cores usually resolves this instantly.

Conclusion

GitOps is the standard for 2024. It separates CI from CD, improves security, and provides a full audit trail—essential for GDPR and internal compliance. But software is only half the equation. You need infrastructure that doesn't blink.

Don't let slow I/O or network latency undermine your automated workflows. Test your GitOps stack on a platform built for performance.

Deploy a high-performance CoolVDS instance today and stop fighting your infrastructure.