Argo CD Introduction, in depth: the GitOps reconciler, the Application resource, sync and health, waves and hooks, ApplicationSets and running it safely

Why pull instead of push

A classic pipeline deploys by pushing. The CI job holds cluster credentials and runs kubectl apply or helm upgrade at the end of a build. That works, but it has three weaknesses. The CI system becomes a high-value secret store with admin rights on production. The cluster's state is whatever the last successful job left behind, so a manual kubectl edit at 2 a.m. silently becomes the truth. And no one keeps asking whether the cluster still matches what was deployed.

Argo CD inverts the direction. It runs inside the cluster, or in a management cluster, and pulls from Git. CI builds and tests an image, then commits a new image digest to a deployment repository; it never talks to the Kubernetes API. Argo CD notices the commit, works out the difference and applies it. Because it compares continuously, it also notices drift from any source. Git history becomes your deployment log, and a revert commit becomes your rollback. The CI/CD article covers the build side: building once and promoting by digest. This page covers the reconciler that consumes those commits.

The components and how a change flows

A standard install runs several workloads in the argocd namespace. The repo-server clones repositories and renders them into plain Kubernetes manifests. It runs Helm templating, Kustomize builds or Jsonnet, and caches the output keyed by commit. The application-controller is the reconciler. It watches live objects in each target cluster, asks the repo-server for the desired manifests, computes a diff and performs syncs. argocd-server serves the web UI, the gRPC/REST API used by the argocd CLI, and the webhook endpoint that Git providers can call. Redis is a cache, not a database: the durable state lives in Kubernetes resources (Applications, AppProjects, Secrets for clusters and repositories). Dex (SSO), the ApplicationSet controller and the notifications controller are also installed.

Follow one change. A developer merges a pull request that bumps the checkout image digest in apps/checkout/overlays/prod. Argo CD learns of the commit either from a webhook, which arrives within seconds, or from polling. By default it polls every 120 seconds plus up to 60 seconds of random jitter, set by timeout.reconciliation and timeout.reconciliation.jitter in the argocd-cm ConfigMap. The controller asks the repo-server to render the new commit, diffs it against the live Deployment and marks the Application OutOfSync. With automated sync enabled it then applies the change. Kubernetes performs the rolling update, and Argo CD tracks the Deployment until its health turns Healthy.

Installing it and the first application

The upstream install is a single manifest. The documented command uses server-side apply because some Argo CD CRDs exceed the size limit of the client-side apply annotation.

kubectl create namespace argocd
kubectl apply -n argocd --server-side --force-conflicts \
  -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# initial password for the 'admin' user (rotate it, then use SSO)
argocd admin initial-password -n argocd

kubectl port-forward svc/argocd-server -n argocd 8080:443
argocd login localhost:8080

The bundled admin account exists for bootstrapping. Change its password, configure SSO, and then disable the account. For production, use the published high-availability manifest.

The Application resource, field by field

An Application is a custom resource that connects one source to one destination. Everything Argo CD does is driven by these objects. Here is a production-shaped example.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: checkout-prod
  namespace: argocd
spec:
  project: payments                     # AppProject: which repos, clusters, namespaces
  source:
    repoURL: https://git.example.com/platform/deploy.git
    targetRevision: main                # branch, tag or commit SHA
    path: apps/checkout/overlays/prod   # Kustomize overlay, Helm chart or plain YAML
  destination:
    server: https://kubernetes.default.svc
    namespace: checkout
  syncPolicy:
    automated:
      prune: true                       # delete resources removed from Git
      selfHeal: true                    # revert manual changes in the cluster
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true
      - PruneLast=true
      - RespectIgnoreDifferences=true    # leave HPA-owned replicas alone during sync too
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas                # owned by the HorizontalPodAutoscaler

source names a repository, a revision and a path. targetRevision can be a branch, which follows the tip, or a tag or commit SHA, which pins the version. Argo CD detects the tool from the path contents: a Chart.yaml means Helm, a kustomization.yaml means Kustomize, and otherwise it reads plain YAML. destination names the cluster API server and the default namespace. project places the Application in an AppProject, which limits the repositories, clusters, namespaces and resource kinds it may use. That is your multi-team guard rail.

To know which live objects belong to an Application, Argo CD writes a tracking marker on each one. The current default is the argocd.argoproj.io/tracking-id annotation. Two Applications that render the same object therefore fight over it, and the second shows a shared-resource warning. Setting FailOnSharedResource=true turns that warning into a sync failure.

Refresh, sync and the two status columns

Two verbs are easy to confuse. A refresh compares Git with the live state and updates the status; it changes nothing in the cluster. A sync applies the desired manifests so that live state matches Git.

Every Application carries two independent statuses. Sync status is Synced, OutOfSync or Unknown: does the live state match Git? Health status is Healthy, Progressing, Degraded, Suspended, Missing or Unknown: is the application actually working? Built-in health checks understand the core kinds. A Deployment is Progressing until its rollout completes, and Degraded if the rollout stalls. You can add Lua health checks for custom resources. Watch both columns. Synced and Degraded means Git is faithfully deploying a broken version. OutOfSync and Healthy usually means someone changed the cluster by hand, or Git moved ahead and sync is manual.

Automated sync, prune and self-heal

With no syncPolicy.automated block, Argo CD only reports differences, and a person presses Sync in the UI or runs argocd app sync checkout-prod. Teams usually start there, then turn on automation once they trust it. Automation has two separate switches. prune: true allows deletion of live objects that have disappeared from Git; without it they linger and the Application stays OutOfSync. selfHeal: true re-syncs when the live state drifts even though Git did not change, so a manual edit is reverted.

Sync options adjust how the apply happens. CreateNamespace=true creates the destination namespace. ServerSideApply=true uses Kubernetes server-side apply, which avoids annotation size limits and handles field ownership better. PruneLast=true defers deletions until all other resources are applied and healthy. Prune=false as a per-resource annotation protects one object, such as a PersistentVolumeClaim, from deletion.

Ordering: sync waves and hooks

By default a sync applies resources in a sensible kind order: namespaces and CRDs first, then configuration, then workloads. When you need more control, use waves and hooks. A wave is an integer in the argocd.argoproj.io/sync-wave annotation; the default is 0, and negative waves run first. Argo CD applies one wave, waits for its resources to become healthy, pauses for about 2 seconds (ARGOCD_SYNC_WAVE_DELAY) and moves to the next. A hook is a resource, usually a Job, annotated with argocd.argoproj.io/hook. The phases are PreSync, Sync, PostSync and SyncFail, plus Skip, PreDelete and PostDelete.

The canonical use is a schema migration that must finish before new Pods start:

apiVersion: batch/v1
kind: Job
metadata:
  name: checkout-db-migrate
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
spec:
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: registry.example.com/checkout@sha256:4f1c...   # same digest as the app
          args: ["migrate", "--to", "latest"]
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: checkout-config
  annotations:
    argocd.argoproj.io/sync-wave: "-1"   # before the Deployment in wave 0

If the PreSync Job fails, the sync stops and the old Deployment keeps serving. BeforeHookCreation deletes the previous run's Job before creating the new one, so the fixed name never collides. HookSucceeded and HookFailed are the other delete policies. Keep migrations backward compatible: the old Pods run against the migrated schema until the rollout finishes. A PostSync Job is a good place for a smoke test, and a SyncFail hook can post to an incident channel.

Diff noise and ignoreDifferences

Some fields are legitimately owned by something other than Git. A HorizontalPodAutoscaler changes spec.replicas. Admission webhooks inject sidecars. Operators fill in defaults. Controllers write a caBundle into webhook configurations. Each of these shows up as permanent OutOfSync, and with self-heal enabled Argo CD fights the other owner forever. The fixes, in order of preference: remove the field from Git (don't set replicas when an HPA owns it); add ignoreDifferences with a JSON pointer, JQ expression or managed-fields manager, as in the example above; and add RespectIgnoreDifferences=true so the ignored fields are also left alone during sync, not just hidden from the diff. Server-side apply reduces this noise further, because field ownership is tracked by the API server.

Many applications: app-of-apps, ApplicationSets and projects

One Application per service per environment adds up fast. The app-of-apps pattern makes Applications themselves the manifests: a root Application points at a directory of Application YAML, so adding a service is a commit, and Argo CD bootstraps itself from Git. ApplicationSet generates Applications from a template plus generators. Generators include list, cluster, Git directory or file, matrix and merge, pull request and SCM provider.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: checkout
  namespace: argocd
spec:
  generators:
    - list:
        elements:
          - env: staging
            cluster: https://staging.k8s.example.com
          - env: prod
            cluster: https://prod.k8s.example.com
  template:
    metadata:
      name: 'checkout-{{env}}'
    spec:
      project: payments
      source:
        repoURL: https://git.example.com/platform/deploy.git
        targetRevision: main
        path: 'apps/checkout/overlays/{{env}}'
      destination:
        server: '{{cluster}}'
        namespace: checkout

The pull request generator gives each open pull request a preview environment, and deletes it when the pull request closes. The cluster generator rolls a platform add-on out to every registered cluster. Pair either pattern with AppProjects. A project lists allowed source repositories, destination clusters and namespaces, and the cluster-scoped kinds its apps may create. It also defines RBAC roles, such as letting the payments team sync only payments apps. Never let tenant Applications use the default project with its wildcards.

Secrets do not belong in Git

GitOps wants everything in Git, but plaintext Secrets must not be there. Three patterns work. With encrypted secrets in Git (Sealed Secrets, or SOPS through a plugin), ciphertext is committed and decrypted in the cluster. With references in Git, the External Secrets Operator syncs an ExternalSecret object from a vault or cloud secret manager into a Kubernetes Secret. Most teams prefer this, because rotation happens in the secret store without a commit. Avoid injecting secrets at render time in the repo-server: rendered manifests are cached and shown in the UI. The repository and cluster credentials that Argo CD itself uses are Kubernetes Secrets in the argocd namespace, so protect that namespace like a production vault.

Failure modes you will meet

• Sync stuck on Progressing. A Deployment never becomes ready: a crash loop, a missing ConfigMap or an image pull error. Argo CD is waiting on health, not failing. Check events on the ReplicaSet and Pods, and set progressDeadlineSeconds so Kubernetes declares the rollout failed.
• Prune deletes something important. A refactor moves a resource to another path or Application, and the old Application prunes it before the new one creates it. Use Prune=false on stateful objects, and review the diff in a pull request.
• Self-heal versus an operator. Two controllers each correct the other every reconcile. Find the field owner and ignore that field.
• Polling lag. Without webhooks, a merge can take up to three minutes to start syncing. Configure webhooks.
• Repo-server overload. Large monorepos and heavy Helm charts make rendering slow. Scale repo-server replicas, split repositories, and watch render durations.

Trade-offs

Argo CD gives you drift detection, audit by Git history, credential-free CI and a UI that shows every object and its health. In exchange, every change takes a commit, emergency fixes need discipline (patch Git, not the cluster, or self-heal undoes your fix), and a second repository layout to design. It does not build images, run tests or orchestrate progressive delivery on its own. Argo Rollouts adds canary and blue-green strategies. If you deploy a few services to one cluster, a push pipeline may be simpler. Once you run many services, clusters and teams, a continuously reconciled source of truth pays for itself.

What to do next

1. Install Argo CD on a disposable cluster with the server-side apply command above, then retrieve and rotate the admin password.
2. Create one Application with a manual sync policy from a repository containing a Deployment and a Service, and sync it from the CLI.
3. Edit the Deployment with kubectl, watch it turn OutOfSync, then enable selfHeal and watch Argo CD revert the change.
4. Add a PreSync migration Job and a negative sync wave on a ConfigMap, then break the Job on purpose to confirm the rollout stops.
5. Create an AppProject per team, restrict sources and destinations, and move your Applications off the default project.
6. Configure Git webhooks, adopt ApplicationSets or app-of-apps for bootstrapping, and choose an external-secrets pattern before any real secret appears.
7. Wire CI to commit image digests to the deployment repository, following the branching strategy article for how that repository is branched, and containerise with the practices in the Docker introduction.