GitOps Workflow Best Practices: Surviving Production Without SSH

I still remember the silence in the room three years ago. A junior sysadmin had just run a manual kubectl apply to fix a hot patch on a Friday afternoon. It worked, for exactly four hours. Then the automated CI pipeline woke up, saw the "drift," and forcefully overwrote the patch with the old broken configuration. The site went dark. The database connections spiked. The phone started vibrating.

If you are still SSH-ing into your servers to make changes, or worse, manually clicking buttons in a cloud dashboard (we call that "ClickOps"), you are holding a ticking time bomb. In 2024, the only source of truth must be Git. Everything else is a mirage.

This isn't just about automation; it's about sanity. It's about knowing that the state of your infrastructure in Oslo exactly matches the code in your repository. Let's dismantle the proper way to build a GitOps workflow that respects both technical rigour and the strict data sovereignty requirements we face here in Europe.

The Pull Model vs. The Push Model

Most teams start with a Push model. Your Jenkins or GitLab CI runner builds a Docker image and then runs helm upgrade against your cluster. This is dangerous.

Why? Because you have to give your CI server cluster-admin credentials. If your CI gets compromised (and supply chain attacks are rising), the attacker owns your entire production environment. Furthermore, the CI server only knows about the deployment at the moment it runs. If the cluster drifts five minutes later, the CI server has no idea.

The Battle-Hardened Alternative: The Pull Model.

In a Pull model (GitOps), an operator inside the cluster (like ArgoCD or Flux) watches the Git repository. When it sees a change, it pulls it down and applies it. It also watches the cluster. If someone manually changes a setting, the operator detects the drift and reverts it instantly.

Pro Tip: For Norwegian enterprises dealing with sensitive data, the Pull model is a compliance superpower. It means no external credentials need to exist outside your secure VPC or private network. Your CoolVDS instance in Oslo pulls public configurations but keeps the secrets internal.

Tooling: ArgoCD Configuration

We prefer ArgoCD for its visual topology and robust RBAC integration. However, installing it blindly is a mistake. You need to configure it for high availability, especially if you are managing multiple clusters.

Here is a production-ready Application manifest. Notice the sync policy. We enable selfHeal to automatically revert manual changes, but we leave prune as a conscious decision for critical resources.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: production-payment-gateway
  namespace: argocd
spec:
  project: default
  source:
    repoURL: 'git@gitlab.com:your-org/infra-manifests.git'
    targetRevision: HEAD
    path: overlays/production
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: payments
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - Validate=false

Directory Structure: Kustomize Overlays

Don't duplicate YAML files. It creates a maintenance nightmare. Use Kustomize to handle environment differences between your staging environment and your production nodes.

A clean structure looks like this:

├── base
│   ├── deployment.yaml
│   ├── service.yaml
│   └── kustomization.yaml
└── overlays
    ├── staging
    │   ├── kustomization.yaml
    │   └── replica_patch.yaml
    └── production
        ├── kustomization.yaml
        └── resource_limits_patch.yaml

In your overlays/production/kustomization.yaml, you enforce the high-performance constraints required for your NVMe-backed instances. If you aren't defining resource requests, the scheduler will pile pods onto a single node until it chokes.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
patchesStrategicMerge:
  - resource_limits_patch.yaml
images:
  - name: my-app
    newTag: v1.4.2

The Infrastructure Factor: Why Latency Matters

Here is the uncomfortable truth software architects ignore: GitOps operators are heavy. ArgoCD's application controller is constantly diffing thousands of objects against etcd. If your underlying storage I/O is slow, your reconciliation loops lag.

I've seen "budget" VPS providers where the steal time (CPU stolen by the hypervisor) hits 20%. In that environment, ArgoCD throws timeout errors, and your cluster state becomes unknown. We build on CoolVDS because they use KVM virtualization with direct NVMe pass-through. When I run iotop, I want to see the raw speed of the drive, not a virtualized queue bottleneck.

Secret Management: The "No Secrets in Git" Rule

You cannot check secrets.yaml into Git. That is a firing offense. In 2024, the standard approach is Sealed Secrets (by Bitnami) or the External Secrets Operator.

With Sealed Secrets, you encrypt the secret on your laptop using the cluster's public key. The resulting hash is safe to commit to a public repo. Only the controller running inside your CoolVDS cluster has the private key to decrypt it.

Command to seal a secret:

This ensures that even if your git repository leaks, your database credentials remain opaque blobs of text.

Handling CI/CD with GitOps

Your CI pipeline (GitHub Actions/GitLab CI) changes its responsibility. It no longer deploys. Its only job is to:

1. Run tests.
2. Build the Docker image.
3. Commit the new image tag to the GitOps repository.

Here is a snippet for a GitLab pipeline that updates the manifest automatically. Note that we use a deploy token to keep it secure.

update_manifest:
  stage: deploy
  image: alpine:3.19
  script:
    - apk add --no-cache git
    - git config --global user.email "ci-bot@coolvds.com"
    - git clone https://oauth2:${DEPLOY_TOKEN}@gitlab.com/org/infra.git
    - cd infra/overlays/production
    - sed -i "s/newTag: .*/newTag: $CI_COMMIT_SHORT_SHA/" kustomization.yaml
    - git commit -am "Bump version to $CI_COMMIT_SHORT_SHA"
    - git push origin main

Local Compliance and Data Residency

In Norway, we have specific challenges regarding Datatilsynet and GDPR. If you are serving Norwegian customers, hosting your Kubernetes nodes on US-controlled hyperscalers can introduce legal grey areas regarding data transfer, even with the Data Privacy Framework.

By hosting your GitOps worker nodes on CoolVDS in our Oslo data centers, you ensure the physical execution of code and storage of persistent volumes remains on Norwegian soil. The git repository can be anywhere, but the state lives here.

Final Thoughts

GitOps is not a silver bullet; it is a discipline. It forces you to document your infrastructure in code. It exposes configuration drift immediately. It hurts initially, but it saves weekends.

Don't let slow I/O kill your reconciliation loops. Stability is the bedrock of automation. Deploy a high-performance KVM instance on CoolVDS today and build a platform that fixes itself while you sleep.