GitOps: Deployments You Can Audit and Trust

Somebody ran kubectl edit on production at 11pm, and now nobody knows why the payment service has three replicas instead of six. That mystery is the disease GitOps cures. When every change flows through Git and an agent reconciles the cluster against it, your deployment history becomes a commit log you can read, diff, and revert.

What GitOps actually is (and is not)

GitOps is a specific operational model with four properties. Miss any one of them and you have "CI/CD with YAML in a repo", which is fine, but it is not GitOps.

• Declarative desired state. The entire system (workloads, config, network policy, dashboards) is described declaratively in Git. Not scripts that produce a state. The state itself.
• Versioned and immutable. Git is the single source of truth. Every change is a commit with an author, a timestamp, and a diff.
• Pulled, not pushed. An agent inside the cluster pulls the desired state and applies it. Your CI system never holds cluster credentials.
• Continuously reconciled. The agent runs a loop, forever, comparing live state against Git and correcting drift. Deployment is not an event. It is a steady-state condition.

What GitOps is not: a replacement for CI. You still need pipelines that build, test, and publish artifacts. GitOps takes over exactly at the boundary where an artifact becomes a running system. If your pipelines are shaky before that boundary, fix that first (we wrote about it in CI/CD pipelines teams trust).

The reconciliation loop is the whole product

The magic is not "deploy on merge". The magic is the loop. Every few minutes (and on every webhook), the agent does three things:

1. Fetch the desired state from Git and render it (Helm, Kustomize, plain manifests).
2. Diff the rendered output against what is actually running in the cluster.
3. Converge by applying the difference, or flag it if auto-sync is off.

This loop gives you two guarantees traditional push pipelines cannot:

Drift detection. If someone hand-edits a Deployment, the agent sees the divergence within one cycle. Depending on policy, it either reverts the change or marks the app OutOfSync and alerts. Either way, the 11pm mystery edit is impossible to hide.

Self-healing recovery. Rebuild a cluster from scratch, point the agent at the repo, and it converges to production. Your disaster recovery story stops being a runbook and starts being a bootstrap command. That has direct consequences for your RTO and RPO targets: in our experience, teams that adopt GitOps typically cut cluster recovery from hours of manual reconstruction to the time it takes to restore data.

Push vs pull: why the direction matters

In a push model, CI runs kubectl apply or helm upgrade against the cluster. That means your CI system holds cluster-admin credentials, sits outside the cluster, and is a high-value target. Compromise the pipeline and you own production.

In a pull model, the agent runs inside the cluster with a read-only deploy key to Git. The credential inversion is the security win:

• CI needs write access to a Git repo and a container registry. Nothing else.
• The cluster never accepts inbound deployment traffic, which shrinks the attack surface dramatically.
• Access review becomes "who can merge to this repo", which your existing branch protection already answers.

This pairs naturally with a zero trust posture, where no external system is implicitly trusted with cluster credentials (more on that in Zero trust architecture explained).

Argo CD or Flux: an honest comparison

Both are CNCF graduated, both are production-grade, and both will outlive whatever blog post told you one of them is dead. The real differences:

Argo CD ships a full web UI, an application-centric model (Application CRD), SSO integration, and RBAC scoped per project. The UI is genuinely useful: engineers can see sync status, resource trees, and diffs without touching kubectl. The ApplicationSet controller generates apps across clusters and environments from a single template. Cost: it is a heavier install, and the UI becomes a governance surface you must secure.

Flux is a set of small controllers (source, kustomize, helm, notification, image automation) that feel native to Kubernetes. No UI by default (Weave GitOps or Capacitor fill the gap). Its image automation controllers can watch a registry and write updated image tags back to Git, closing the loop without CI involvement. Flux tends to win in platform teams that want composable primitives; Argo CD tends to win where developer self-service and visibility matter most.

A minimal Argo CD application, for flavor:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: checkout-api
  namespace: argocd
spec:
  project: ecommerce
  source:
    repoURL: https://git.example.com/platform/deploy.git
    targetRevision: main
    path: apps/checkout-api/overlays/production
  destination:
    server: https://kubernetes.default.svc
    namespace: checkout
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Three lines in that spec carry most of the operational weight. prune: true deletes resources removed from Git. selfHeal: true reverts manual drift. targetRevision: main means merge equals deploy. Each is a policy decision, not a default to copy blindly.

Repository structure that survives growth

The most common GitOps regret is repo layout. Two rules save most of the pain:

Separate application code from deployment config. App repos hold source and Dockerfiles. A deploy repo holds manifests. If manifests live next to code, every image tag bump triggers CI on the app repo, you get commit noise in your feature history, and a config-only change rebuilds the world.

Model environments as directories, not branches. Branch-per-environment sounds tidy and becomes a merge-conflict factory, because environments intentionally diverge (replicas, resources, feature flags) and Git merges are built for convergence. Use Kustomize overlays instead:

deploy/
  apps/
    checkout-api/
      base/
        deployment.yaml
        service.yaml
        kustomization.yaml
      overlays/
        staging/
        production/

Promotion becomes a pull request that copies a tested image tag from the staging overlay to the production overlay. The PR diff is one line. The reviewer sees exactly what changes. The audit trail writes itself.

Secrets: the part everyone gets wrong

You cannot commit plaintext secrets to Git, and base64 is not encryption. Three workable patterns:

• Sealed Secrets. Encrypt with the cluster's public key, commit the SealedSecret resource, and only the in-cluster controller can decrypt. Simple, but secrets are bound to one cluster and rotation means resealing.
• SOPS with age or KMS. Encrypt secret values in place, commit the encrypted file, and let Flux (native support) or an Argo CD plugin decrypt at apply time. Diffs stay reviewable because keys remain visible while values are ciphertext.
• External Secrets Operator. Git stores only a reference (ExternalSecret) to a value in Vault, AWS Secrets Manager, or GCP Secret Manager. The operator syncs the real value into the cluster. This is our default recommendation for teams already running a secrets manager, because rotation happens in one place and Git never touches sensitive material.

Whichever you pick, wire secret scanning into CI so a raw credential never lands in history. This belongs in the same guardrail set as the rest of your DevSecOps pipeline.

Progressive delivery: GitOps beyond the sync

A sync tells you the manifests applied. It does not tell you the release is healthy. Progressive delivery closes that gap. Argo Rollouts and Flagger both replace the standard Deployment rollout with a controlled canary: shift a small percentage of traffic to the new version, watch real metrics (error rate, p99 latency, business KPIs from Prometheus), and promote or roll back automatically based on thresholds.

The combination is potent. Git defines what should run. The rollout controller decides, from live telemetry, whether it is safe to keep running. Rollback becomes git revert plus an automatic traffic shift, usually completing in minutes with no human paging through dashboards.

Do not skip the observability prerequisite. Automated analysis is only as good as the metrics feeding it. If your error budget signals are noisy, the canary gate will either block everything or pass everything.

Failure modes we see in the wild

GitOps has sharp edges. These are the ones that actually cut:

• Prune disasters. Auto-prune plus a bad refactor (renaming a directory, changing a Kustomize path) can make the agent believe entire applications were deleted from Git, so it deletes them from the cluster. Mitigate with Prune=false annotations on critical stateful resources and a required diff preview on deploy-repo PRs.
• Sync loops. A mutating admission webhook rewrites a field, the agent sees drift, reverts it, the webhook rewrites it again, forever. Fix with ignoreDifferences rules for fields owned by other controllers (HPA replica counts are the classic case).
• CRD ordering. Applying a custom resource before its CRD exists fails the sync. Use sync waves or dependency ordering so CRDs and operators land first.
• Rendered-manifest surprises. Helm values plus templating logic can make a one-line values change produce a hundred-line manifest diff. Render manifests in CI and post the real diff on the PR, so reviewers approve actual changes rather than inputs to a template.
• Monorepo reconcile storms. One giant deploy repo means every commit triggers reconciliation for every application. Split by team or domain once webhook fan-out starts hurting.

A pragmatic adoption path

Do not flip the whole platform in one quarter. This sequence works:

1. Pick one stateless, low-blast-radius service and describe its full desired state declaratively.
2. Stand up the agent (Argo CD or Flux) in read-only mode: no auto-sync, drift detection only. Watch what it flags for a week. The drift you find will surprise you.
3. Enable auto-sync without prune. Merges now deploy. Deletions still require a human.
4. Add environment overlays and PR-based promotion from staging to production.
5. Solve secrets properly (External Secrets or SOPS) before onboarding anything sensitive.
6. Enable prune and self-heal once the team trusts the diffs.
7. Layer in progressive delivery for the services where a bad release is expensive.
8. Expand service by service, and delete the old push pipelines as you go. Two deployment paths to the same cluster is a standing incident invitation.

Most teams we work with land steps 1 through 4 in a few weeks. Steps 5 through 7 are where discipline pays off.

When GitOps is the wrong call

Honesty over hype. Skip GitOps, at least for now, if:

• You run two services on a single VM. A systemd unit and a deploy script beat a control plane you must operate. (Related question, same energy: do you need Kubernetes at all?)
• Your desired state cannot be made declarative. Legacy systems driven by imperative runbooks need modernization before reconciliation makes sense.
• Nobody owns the platform. The agent, its upgrades, its RBAC, and its failure modes need an owner. GitOps reduces operational toil; it does not reduce it to zero.
• Your bottleneck is review latency, not deployment safety. If PRs sit for days, adding another PR gate (promotion) slows delivery further until you fix the review culture.

For everyone else running Kubernetes with more than a handful of services and more than a couple of engineers deploying, the audit trail, the drift immunity, and the one-command recovery are worth the setup cost many times over.

How Innovation T can help

Innovation T designs and builds GitOps platforms for teams across Europe, the Gulf, and North Africa: Argo CD and Flux rollouts, environment promotion flows, secrets architecture, and progressive delivery wired to real SLOs. We have made the prune mistakes and hit the sync loops already, so you do not have to. Explore our cloud and DevOps services to see how we structure these engagements.

If your deployments still depend on the one person who knows the script, talk to us. We will audit your current pipeline and hand you a concrete, staged migration plan.