Resolution, restore first and drafted fixes
How 3AM resolves, not just diagnoses. The ActionEvaluator (safe, approval, forbidden), autonomy L2, mitigate first, the resolution loop, restore first when nothing is proven, the 90-second hand-over, and drafted fixes.
3AM is judged by service restored and nothing made worse. Safe moves run as soon as the evidence and the impact justify them; breaking actions always wait for people; a fix that didn't work is followed by the next one.
The ActionEvaluator
Every fix is classified:
| Tier | Meaning |
|---|---|
SAFE_AUTONOMOUS | May run without approval, at L2 |
REQUIRES_APPROVAL | Checkpointed, then the approval chain, then resumed |
FORBIDDEN | Never runs: halted, a policy deny, no declared connector |
An action is safe only when all of these hold; each that fails is a reason shown to the approver:
| Criterion | Rule |
|---|---|
| Catalogue | kubernetes.restart (Deployment only), kubernetes.restart_pod (a ReplicaSet pod), kubernetes.scale (Deployment, up only), argocd.terminate_operation, declared reversible traffic shifts; plus policy autonomous.allow, minus autonomous.never |
| Risk | Declared low or medium |
| Recoverable | Reversible, or restorative (a controller recreates what a restart replaces) |
| Non-destructive | No state-altering verb (delete, drop, kill, write, rollback, sync, deploy, patch, …) |
| No free form | No statement, query, command, script or body parameter |
| Stateless target | Not a StatefulSet or DaemonSet, no stateful name (db, mysql, postgres, redis, kafka, etcd, …), not marked stateful, not a database connector |
| Bounds | Scale up only, ≤ ×2 and ≤ 20 replicas, from a replica count read from the evidence |
| Context | The cause is proven, or the suspect is strong (its own check passed, P ≥ 0.5) and impact ≥ 0.5; fewer than 2 autonomous actions in this incident; the autonomous budget (2 per service, 6 overall per hour) not spent; not the same action within 30 min; no policy rule asking for people |
A safe action still passes policy (deny rules, halt, hours, budgets), the rehearsal and verification.
Autonomy L2
Like L1, except SAFE_AUTONOMOUS actions run without approval. Raising to L2 needs two approvers; lowering applies at
once. At shadow and L1, the record still says what L2 would have run (classification, would_run), so the
Digest shows whether L2 is earned before anyone raises it.
Mitigate first
At L2, when nothing is proven yet but a suspect's own check passed and the impact is high, the first safe fix among the
suspects is rehearsed and run (action.autonomous), validated when the alerts stop or the suspect's check no longer
passes (within 120 s). The investigation continues. If the mitigation stopped the alerts, the incident closes as
mitigated_first / mitigated, with the suspected cause flagged as still to confirm.
The resolution loop
The first candidate fix, plus the fixes of every other confirmed cause, go through the loop, up to 3 candidates:
classify, run (safe at L2: rehearse and execute; otherwise approval), re-run the proof until it clears. Resolved: stop.
Unresolved, failed or refused by the rehearsal: the next candidate, again through policy and System 1's veto. A person's
rejection stops the loop. Every transition is a resolution.state event, checkpointed in data/resolution/<id>.json.
Candidates are ordered by:
priority = impact × P(cause) × P(works) − action_risk × (1 − impact / 2)with action_risk low 0.05, medium 0.15, high 0.4, scaled by the target's own blast radius.
Restore first
When the investigation ends with nothing proven, 3AM looks for one move that fits what it saw, in this order:
| Move | When | If the alerts don't stop |
|---|---|---|
| Undo the one change just before | Exactly one fact changed within 30 minutes before the first alert: a schema change, a database setting, reference rows, a migration marked failed after it succeeded, a workload's images, an Argo CD revision | An SQL undo is taken back by 3AM; a rollback is reported with how to reverse it |
| Start what stopped | Nothing changed, and the application the alerts are about isn't running: a unit failed, a container exited, one pod not ready while a sibling is | Nothing to take back |
| Run the declared failover | Nothing changed, nothing to restart, and exactly one database the service uses reads as down | 3AM never fails back |
- Exactly one such change, or nothing is proposed: with two, 3AM doesn't guess, and both go in the hand-over.
- A running application is never restarted. A failover is never worked out by 3AM: without a declared one, none.
- Needs fix mode
draft. Approved like a drafted fix (two approvers by default), after System 1 and a rehearsal; the request is titled Restore move (cause not proven). - Just before it runs, what it acts on is read again; otherwise it isn't run (
blocked). - Verified by the alerts stopping and the fact being back:
decision: restore,outcome: mitigated, resultrestored. - If the alerts keep firing, 3AM takes its own move back where the reverse is exact, records
action.reversed, and hands over (restore_reversed). - In fix mode
advise, nothing is proposed: the move, its exact command and its reverse lead the hand-over.
The hand-over
90 seconds after the first alert with no cause proven, the people on call get the triage while the investigation
goes on: what is affected and since when, what changed just before, what is abnormal now, what is ruled out, what
couldn't be checked (handover.sent). An incident that ends handed over sooner gets the same triage with its closing
note. Shadow tells no one.
Drafted fixes
When a cause is proven but no pack has a fix, the install-wide fix mode decides: advise (default) gives the
advice to people; draft drafts a fix. Turning draft on needs the autonomy_change approvers
(POST /v1/fix-mode {"mode":"draft"}); advise applies at once.
Where a draft comes from:
- Undo the proven change (
drafted:undo), from the value recorded before it: a removed column re-added with its type and position; a MySQL setting put back; removed configuration rows re-inserted, or added ones deleted; a Deployment rolled back. It refuses whenever the old value can't be reproduced exactly (text kept as a hash, a whole table gone, a PostgreSQL setting, …). - Otherwise, the investigator (
drafted:model) fills in one declared action; free-form and read actions are refused, and every value must appear verbatim in the evidence.
Gates, on top of a pack fix's:
- Only for a proven cause; never linked, suspected, disputed, cleared, chronic or upstream symptoms, or a demoted cause.
- Quorum max(rule quorum,
drafted_fixes.quorum), default 2, never below 1. Rules can match"origin": "drafted". - System 1 reviews the change itself; the request is labelled Drafted fix … with what it undoes, the inverse, the caveats, and a warning when a recorded deploy may explain the change.
- Just before it runs, the change must still be in place and the cause still confirmed.
- Verified only if the cause cleared and, for an undo, the fact is back. Otherwise
unresolved, with the inverse in the advice; nothing is undone automatically. - No retry of the same fix on the same service within 30 minutes. Shadow drafts and rehearses it, and asks no one.
- A drafted fix that worked becomes its learned pattern's fix, still pending an admin's review.
Audit events: fix.drafted, fix.draft_refused.
Audit events
decision.classified, decision.autonomous, action.autonomous, outcome.mitigation, resolution.state,
restore.suggested, action.reversed, handover.sent, fix.drafted, fix.draft_refused.