Argo CD

With Argo CD connected, every incident on an Argo-managed workload starts with the question an on-call engineer asks first: what changed? 3AM shows the revision that went out, its author and commit message, and when it synced. Argo CD also changes how a bad release is undone, because its self-heal re-applies Git: a kubectl rollback of an Argo-managed Deployment is undone within seconds (we measured 15). With this connector, 3AM rolls back through Argo CD instead. It also watches Argo CD itself: a repo-server or controller that hangs, a sync stuck on a hook, a Git host that's unreachable.

Before you start

• Admin access to Argo CD, once, to create a local account for 3AM.
• Network access from 3AM to the Argo CD API server, and to the components' metrics ports if you want 3AM to notice them hanging.
• The Kubernetes connector, so 3AM can prove why a workload fails and restart Argo CD's own components. Add the argocd namespace to its namespaces and apply the 3AM Role there.

1. Create the 3AM accountCopy Anchor Link

Add a local account with an API key, and a role that can read applications, sync them, and turn auto-sync off for a rollback:

data:
  accounts.threeam: apiKey

data:
  policy.csv: |
    p, role:threeam, applications, get, */*, allow
    p, role:threeam, applications, sync, */*, allow      # sync, roll back, stop a stuck sync
    p, role:threeam, applications, update, */*, allow    # turn auto-sync off before a rollback
    p, role:threeam, projects, get, *, allow
    g, threeam, role:threeam

Leave out sync and update for a read-only account: 3AM then diagnoses and hands fixes to a person. Scope */* to <project>/* to limit 3AM to some projects.

Create the token:

argocd account generate-token --account threeam > 3am-argocd-token

2. Add the connectorCopy Anchor Link

In the console: Settings → Add a connection → Argo CD. Or in connectors.json:

{ "name": "argocd", "kind": "argocd",
  "settings": { "url": "https://argocd-server.argocd.svc",
                "token": { "file": "/etc/3am/secrets/3am-argocd-token" },
                "ca_file": "/etc/3am/secrets/argocd-ca.pem",
                "component_urls": ["http://argocd-metrics.argocd.svc:8082",
                                   "http://argocd-repo-server.argocd.svc:8084",
                                   "http://argocd-server-metrics.argocd.svc:8083"] } }

Argo CD serves a self-signed certificate unless you gave it one. Point ca_file at the CA that signed it, rather than turning verification off.

3. VerifyCopy Anchor Link

Argo CD v3.5.3; 7 applications

If a component listed in component_urls doesn't answer, Test connection names it.

When an Argo-managed workload has a bad release, the incident's proposed fix is an Argo CD rollback, shown as rollback id=0, name=payments (via argocd), where argocd is your connector's name. It is never a kubectl rollback.

What it detectsCopy Anchor Link

Signals carry the app in the name label, like argocd_app_info. So if you already alert with the community rules (ArgoCdAppUnhealthy, ArgoCdAppSyncFailed, ArgoCdAppOutOfSync) through Prometheus, the same causes apply.

What it proves and fixesCopy Anchor Link

Fixed with approval:

Handed to a person:

How a rollback worksCopy Anchor Link

1. Rehearse. Argo CD renders the target revision. Argo refuses a dry-run rollback while auto-sync is on, so rendering is the rehearsal: it proves the old revision still produces valid manifests.
2. Approve. The request shows the commit that went out ("payments 1.1: read ledger url from config", Jane Doe) and the revision it goes back to.
3. Turn auto-sync off, so self-heal can't re-apply the bad commit.
4. Roll back to the previous deployment, and wait until the app is healthy.
5. Hand back. The incident says which commit to revert in Git, and to turn auto-sync back on (set_auto_sync, or in Argo CD).

The Kubernetes pack follows the same rule. A bad release of an Argo-managed workload is rolled back through Argo CD. If 3AM has no Argo CD connection, it hands the rollback to a person rather than using kubectl. It knows which app manages a workload from the argocd.argoproj.io/tracking-id annotation.

ActionsCopy Anchor Link

CertificationCopy Anchor Link

Live, on Argo CD v3.5.3 and v3.3.14 (the oldest supported line), with the least-privilege account above and 3AM running inside the cluster with the client RBAC (2026-10-05). 53 of 53 checks passed on each version:

• The connector. Health, discovery, the deploy behind each app (commit, author, message), and every error explained: a wrong token, an untrusted certificate, an app the account can't see.
• A bad release on an auto-syncing app. Rolled back through Argo CD with auto-sync off, still healthy 45 seconds later. A kubectl rollback is undone by self-heal within 15 seconds. The approval showed which commit went out, and the note said which commit to revert in Git.
• A release whose database was down. Left alone, with advice.
• Argo CD and Git problems. Manifests that don't render, an unreachable Git host and a resource changed by hand got advice, and nothing ran. A sync stuck on a hook was stopped. Resources deleted from a manual app were re-synced from Git after a dry run.
• Argo CD itself, hung mid-run. The repo-server and the application controller were each frozen. 3AM located them, restarted them through the Kubernetes connector and verified they were back.
• The audit trail. No token reached the ledger, and the ledger verified.

TroubleshootingCopy Anchor Link

the token is invalid or expiredCopy Anchor Link

Generate a new token for the threeam account (step 1).

the certificate is not trusted: set ca_fileCopy Anchor Link

Argo CD's certificate is self-signed or from your internal CA. Set CA certificate to the CA that signed it.

the 3AM account's Argo CD role does not allow thisCopy Anchor Link

The account lacks sync or update in argocd-rbac-cm. That's deliberate for a read-only setup; add them for fixes.

Argo CD's components aren't restartedCopy Anchor Link

The Kubernetes connector must watch the argocd namespace, with the 3AM Role applied there.