Kubernetes

What 3AM does with KubernetesCopy Anchor Link

1. Give 3AM least-privilege accessCopy Anchor Link

3AM needs one ServiceAccount and, in each namespace it watches, a Role allowing only:

• read pods, pod logs, Services, Deployments, StatefulSets, DaemonSets and ReplicaSets;
• patch Deployments, StatefulSets and DaemonSets, including their scale (approved restarts, scaling and rollbacks);
• delete a single pod, so its controller recreates it (approved restart_pod: controller-managed pods only);
• read and patch PersistentVolumeClaims (approved expand_volume: only where the StorageClass allows it);
• create Events (optional: 3AM's notes).

Cluster-wide, 3AM may only read StorageClasses, to know whether a volume may grow. No exec, no Secrets, no deleting workloads or volumes.

You can take any action away: delete its rule. The connector's health then lists it as not allowed, and 3AM hands that fix to a person instead. The manifests are in the bundle under kubernetes/. Apply them with a cluster admin's credentials:

kubectl apply -f 3am-rbac.yaml                    # namespace 3am, the ServiceAccount, its token, StorageClass read

Then once per namespace 3AM should watch:

for NS in bank-prod payments-prod; do
  sed "s/WATCHED_NAMESPACE/$NS/" 3am-role.yaml | kubectl apply -f -
done

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: {name: 3am-connector, namespace: WATCHED_NAMESPACE}
rules:
  - apiGroups: [""]
    resources: [pods, pods/log, services]
    verbs: [get, list, watch]
  - apiGroups: [apps]
    resources: [deployments, statefulsets, daemonsets, replicasets]
    verbs: [get, list, watch]
  - apiGroups: [apps]
    resources: [deployments, statefulsets, daemonsets, deployments/scale, statefulsets/scale]
    verbs: [patch]
  - apiGroups: [""]
    resources: [pods]
    verbs: [delete]                       # restart_pod
  - apiGroups: [""]
    resources: [persistentvolumeclaims]
    verbs: [get, list, patch]             # expand_volume
  - apiGroups: [""]
    resources: [events]
    verbs: [create]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: {name: 3am-connector, namespace: WATCHED_NAMESPACE}
subjects: [{kind: ServiceAccount, name: 3am-connector, namespace: 3am}]
roleRef: {apiGroup: rbac.authorization.k8s.io, kind: Role, name: 3am-connector}

2. Collect the token, the CA and the API addressCopy Anchor Link

# the token
kubectl -n 3am get secret 3am-connector-token -o jsonpath='{.data.token}' | base64 -d > 3am-k8s-token
# the cluster's CA certificate
kubectl config view --raw --minify -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 -d > k8s-ca.pem
# the API server address
kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}'; echo

3. Connect it in 3AMCopy Anchor Link

Settings → Add a connection → Kubernetes (or the wizard's Systems step). Paste the token, choose the CA file:

Test connection asks the API server, for each namespace, whether 3AM's token can do every thing it needs, and lists anything missing. A healthy result:

Connected. Kubernetes v1.30.4; all permissions present in bank-prod, payments-prod

Network: outbound HTTPS from 3AM to the API server. If 3AM runs in the same cluster, that's the kubernetes.default.svc address.

What 3AM understandsCopy Anchor Link

Every cause is proven from the cluster before anything is proposed:

Bad release and dependency down exclude each other on purpose. Restarting a pod whose database is down is the classic wrong move, and 3AM won't make it.

SignalsCopy Anchor Link

The names match the kube-prometheus alerts, so the same root causes apply whichever source reports the failure. One incident per problem per workload: three crash-looping replicas are one incident.

ActionsCopy Anchor Link

Every change is rehearsed with dryRun=All: the API server validates and admits it, including your admission webhooks, and nothing changes. Two fences protect other teams: 3AM refuses namespaces outside its list, and the Role refuses them even if that list were wrong.

TroubleshootingCopy Anchor Link

CertificationCopy Anchor Link

Certified on 2026-10-04 on a real cluster (Kubernetes 1.34) with exactly this RBAC and only 3AM's token: 32 live connector checks, and 30 end-to-end checks. Two broken releases (a config panic and a missing image) were proven, rolled back and verified healthy; a release whose database was unreachable was correctly left alone; an OOM and a missing ConfigMap were diagnosed with advice and nothing executed.

Fixes, certified live on 2026-10-04 (54 of 54 checks, on a real Kubernetes cluster, with 3AM inside it under the client RBAC). Components were hung mid-run, meaning their pod was Running and answered nothing. Each was detected, proven, located, rehearsed, approved, restarted and verified:

• vmagent, a vmstorage node (only that pod), a scrape target of vmagent and one of Prometheus, Alertmanager (seen by both monitoring connectors, restarted once) and Prometheus.

A read-only storage node got a proposal to grow its volume, approved by two people. The test storage driver accepts the request but can't grow volumes, and 3AM reported the fix as failed rather than claiming success.