# Resolution, restore first and drafted fixes

> How 3AM resolves, not just diagnoses. The ActionEvaluator (safe, approval, forbidden), autonomy L2, mitigate first, the resolution loop, restore first when nothing is proven, the 90-second hand-over, and drafted fixes.

Source: https://docs.3am.si/how-3am-decides/resolution · Markdown: https://docs.3am.si/how-3am-decides/resolution.md · All docs: https://docs.3am.si/llms.txt
3AM is an on-prem AI on-call engineer for regulated enterprises (https://3am.si).

3AM is judged by **service restored and nothing made worse**. Safe moves run as soon as the evidence and the impact
justify them; breaking actions always wait for people; a fix that didn't work is followed by the next one.

## The ActionEvaluator

Every fix is classified:

| Tier                | Meaning                                                    |
| ------------------- | ---------------------------------------------------------- |
| `SAFE_AUTONOMOUS`   | May run without approval, at L2                            |
| `REQUIRES_APPROVAL` | Checkpointed, then the approval chain, then resumed        |
| `FORBIDDEN`         | Never runs: halted, a policy `deny`, no declared connector |

An action is safe only when **all** of these hold; each that fails is a reason shown to the approver:

| Criterion        | Rule                                                                                                                                                                                                                                                                                               |
| ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Catalogue        | `kubernetes.restart` (Deployment only), `kubernetes.restart_pod` (a ReplicaSet pod), `kubernetes.scale` (Deployment, up only), `argocd.terminate_operation`, declared reversible traffic shifts; plus policy `autonomous.allow`, minus `autonomous.never`                                          |
| Risk             | Declared `low` or `medium`                                                                                                                                                                                                                                                                         |
| Recoverable      | Reversible, or restorative (a controller recreates what a restart replaces)                                                                                                                                                                                                                        |
| Non-destructive  | No state-altering verb (delete, drop, kill, write, rollback, sync, deploy, patch, …)                                                                                                                                                                                                               |
| No free form     | No statement, query, command, script or body parameter                                                                                                                                                                                                                                             |
| Stateless target | Not a StatefulSet or DaemonSet, no stateful name (db, mysql, postgres, redis, kafka, etcd, …), not marked `stateful`, not a database connector                                                                                                                                                     |
| Bounds           | Scale up only, ≤ ×2 and ≤ 20 replicas, from a replica count read from the evidence                                                                                                                                                                                                                 |
| Context          | The cause is proven, **or** the suspect is strong (its own check passed, P ≥ 0.5) **and** impact ≥ 0.5; fewer than 2 autonomous actions in this incident; the autonomous budget (2 per service, 6 overall per hour) not spent; not the same action within 30 min; no policy rule asking for people |

A safe action still passes policy (deny rules, halt, hours, budgets), the rehearsal and verification.

## Autonomy L2

Like L1, except `SAFE_AUTONOMOUS` actions run without approval. Raising to L2 needs two approvers; lowering applies at
once. At shadow and L1, the record still says what L2 would have run (`classification`, `would_run`), so the
[Digest](https://docs.3am.si/console/digest) shows whether L2 is earned before anyone raises it.

## Mitigate first

At L2, when nothing is proven yet but a suspect's own check passed and the impact is high, the first safe fix among the
suspects is rehearsed and run (`action.autonomous`), validated when the alerts stop or the suspect's check no longer
passes (within 120 s). **The investigation continues.** If the mitigation stopped the alerts, the incident closes as
`mitigated_first` / `mitigated`, with the suspected cause flagged as still to confirm.

## The resolution loop

The first candidate fix, plus the fixes of every other confirmed cause, go through the loop, up to **3** candidates:
classify, run (safe at L2: rehearse and execute; otherwise approval), re-run the proof until it clears. Resolved: stop.
Unresolved, failed or refused by the rehearsal: the next candidate, again through policy and System 1's veto. A person's
rejection stops the loop. Every transition is a `resolution.state` event, checkpointed in `data/resolution/<id>.json`.

Candidates are ordered by:

```text
priority = impact × P(cause) × P(works) − action_risk × (1 − impact / 2)
```

with `action_risk` low 0.05, medium 0.15, high 0.4, scaled by the target's own blast radius.

## Restore first

When the investigation ends with **nothing proven**, 3AM looks for one move that fits what it saw, in this order:

| Move                                | When                                                                                                                                                                                                           | If the alerts don't stop                                                        |
| ----------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- |
| **Undo the one change just before** | Exactly one fact changed within 30 minutes before the first alert: a schema change, a database setting, reference rows, a migration marked failed after it succeeded, a workload's images, an Argo CD revision | An SQL undo is taken back by 3AM; a rollback is reported with how to reverse it |
| **Start what stopped**              | Nothing changed, and the application the alerts are about isn't running: a unit failed, a container exited, one pod not ready while a sibling is                                                               | Nothing to take back                                                            |
| **Run the declared failover**       | Nothing changed, nothing to restart, and exactly one database the service uses reads as down                                                                                                                   | 3AM never fails back                                                            |

* Exactly one such change, or nothing is proposed: with two, 3AM doesn't guess, and both go in the hand-over.
* A running application is never restarted. A failover is never worked out by 3AM: without a declared one, none.
* Needs fix mode **`draft`**. Approved like a drafted fix (two approvers by default), after System 1 and a rehearsal;
  the request is titled *Restore move (cause not proven)*.
* Just before it runs, what it acts on is read again; otherwise it isn't run (`blocked`).
* Verified by the alerts stopping and the fact being back: `decision: restore`, `outcome: mitigated`, result `restored`.
* If the alerts keep firing, 3AM takes its own move back where the reverse is exact, records `action.reversed`, and
  hands over (`restore_reversed`).
* In fix mode `advise`, nothing is proposed: the move, its exact command and its reverse lead the hand-over.

## The hand-over

**90 seconds** after the first alert with no cause proven, the people on call get the triage while the investigation
goes on: what is affected and since when, what changed just before, what is abnormal now, what is ruled out, what
couldn't be checked (`handover.sent`). An incident that ends handed over sooner gets the same triage with its closing
note. Shadow tells no one.

## Drafted fixes

When a cause is **proven** but no pack has a fix, the install-wide **fix mode** decides: `advise` (default) gives the
advice to people; `draft` drafts a fix. Turning `draft` on needs the `autonomy_change` approvers
(`POST /v1/fix-mode {"mode":"draft"}`); `advise` applies at once.

**Where a draft comes from:**

1. **Undo the proven change** (`drafted:undo`), from the value recorded before it: a removed column re-added with its
   type and position; a MySQL setting put back; removed configuration rows re-inserted, or added ones deleted; a
   Deployment rolled back. It refuses whenever the old value can't be reproduced exactly (text kept as a hash, a whole
   table gone, a PostgreSQL setting, …).
2. **Otherwise, the investigator** (`drafted:model`) fills in one declared action; free-form and read actions are
   refused, and every value must appear verbatim in the evidence.

**Gates, on top of a pack fix's:**

* Only for a **proven** cause; never linked, suspected, disputed, cleared, chronic or upstream symptoms, or a demoted
  cause.
* **Quorum** max(rule quorum, `drafted_fixes.quorum`), default **2**, never below 1. Rules can match
  `"origin": "drafted"`.
* System 1 reviews the change itself; the request is labelled *Drafted fix …* with what it undoes, the inverse, the
  caveats, and a warning when a recorded deploy may explain the change.
* **Just before it runs**, the change must still be in place and the cause still confirmed.
* **Verified** only if the cause cleared and, for an undo, the fact is back. Otherwise `unresolved`, with the inverse in
  the advice; nothing is undone automatically.
* **No retry** of the same fix on the same service within 30 minutes. Shadow drafts and rehearses it, and asks no one.
* A drafted fix that worked becomes its learned pattern's fix, still pending an admin's review.

Audit events: `fix.drafted`, `fix.draft_refused`.

## Audit events

`decision.classified`, `decision.autonomous`, `action.autonomous`, `outcome.mitigation`, `resolution.state`,
`restore.suggested`, `action.reversed`, `handover.sent`, `fix.drafted`, `fix.draft_refused`.