# Incidents and grouping > One failure, one incident. How alerts are grouped (refire, remediation effect, same service, dependency, shared entity), the lifecycle, what a related episode concludes, priority, one ticket and one thread. Source: https://docs.3am.si/how-3am-decides/incidents · Markdown: https://docs.3am.si/how-3am-decides/incidents.md · All docs: https://docs.3am.si/llms.txt 3AM is an on-prem AI on-call engineer for regulated enterprises (https://3am.si). A real failure rarely raises one alert. The database goes read-only; during the fix and the recovery the same service raises knock-on alerts, the services that depend on it return 5xx, and the alert may flap once on its way out. 3AM groups them: an **incident** is what people see and what gets resolved; an **episode** is one investigation of one service's alerts. An incident owns every episode of the same failure, one ticket and one message thread, and outlives its fix by a monitoring window. Two questions are kept apart: * **Grouping:** does this alert belong to that incident? Decided at intake, by the criteria below. * **Attribution:** does that incident's cause explain this alert? Decided by the investigation, under the proof rules. ## Lifecycle ```text open ──(fix proposed)──▶ mitigating ──(verified)──▶ monitoring ──(quiet 30 min)──▶ resolved │ │ rejected / not resolved ▲ │ its own alert fires again └──(every alert stopped)──┼──────────────────────────────┘ └──────────────▶ open (reopened) ▼ open resolved ──(own alert within 30 min)──▶ open ``` | State | Meaning | | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------- | | `open` | Alerts are being investigated | | `mitigating` | A fix is waiting for approval or running | | `monitoring` | The fix is verified, or every alert stopped. Alerts now (the recovery's tail, a knock-on effect, the failure coming back) join this incident | | `resolved` | Quiet for `monitor_s` (30 min) | An incident still firing after its fix's monitoring window reaches people once, as an update to that incident. ## Grouping criteria New alerts are grouped by service, then scored against every live incident. The strongest relation wins; **0.6 or more joins**. Every decision is an `incident.joined` or `incident.opened` event with its reasons. | Kind | Criterion | Score | Applies while | | -------------------- | ------------------------------------------------------------------------------------------------------------ | ------------------- | -------------------------------------------------------- | | `refire` | One of the incident's own alerts fires again (same fingerprint) | 1.0 | Live, or resolved within `reopen_s` (30 min). Reopens it | | `remediation_effect` | The alert names what 3AM's fix acted on (a workload, a host), within 30 min of the fix | 0.95 | Live | | `same_service` | The alert resolves to a service already in the incident | 0.9 | Live | | `dependency` | A dependency path either way, up to 3 hops, from a service in the incident (declared, or learned and active) | 0.85 × 0.9^(hops−1) | Live, active in the last hour | | `shared_entity` | The alert names a host, node or workload the incident's alerts name | 0.75 | Live, active in the last hour | **An open incident explains new alerts only while its cause holds.** Before it absorbs an alert or counts as the same failure, its proven causes are re-run (at most every 60 s). If the evidence no longer confirms the cause, it explains nothing new (`episode.cause_cleared`). **Baseline alerts.** Alerts already firing when 3AM starts are known (chronic), not new; a re-fire after they clear is new. An episode waits 20 s for co-firing alerts before it opens. ## What a related episode concludes 1. **Settle.** If the incident's fix ran within 30 min and this isn't a refire, wait up to `settle_s` (120 s) for the alerts to stop. If they do: `settled` / `cleared`, the recovery's tail, no investigation, no page. 2. **The same cause proven again:** the fix didn't hold. `recurrence_in_incident`, a fix proposed again (with approval) inside the same incident. 3. **A different cause proven:** a **contributing** cause while the incident's cause holds (or about what the fix acted on); otherwise a **new failure**, split into its own incident (`incident.split`). 4. **Nothing proven:** linked to a proven upstream (`downstream`); explained by the incident's cause while it holds (`related`); one of its own alerts back after the fix (`recurrence`, an update); still firing after the fix and a full investigation (`unexplained_after_fix`, an update). **Grouping is never silence.** ## Priority: which incident first When investigations wait for a slot ([Sizing](https://docs.3am.si/install/sizing#how-work-runs-in-parallel)): | Rank | Episode | Base priority | | ---- | -------------------------------------------------------- | ------------------------------------------- | | 1 | The incident's root episode | 3.0 | | 2 | Refire: its own alert back after its fix | 2.5 | | 3 | Remediation effect or same service (the recovery's tail) | 1.5 | | 4 | Dependency or shared entity (the services around it) | 1.0 | | + | Upstream within the incident | +0.75 per episode that depends on it (≤ +2) | Across incidents: **+2 × impact** (blast radius, criticality, how much is firing), **+0.5** for a critical alert, **+1 per 5 minutes** waited. One slot is always kept for a new, independent incident. Every queued and started episode records its priority and each term (`episode.queued`, `episode.started`). ## One ticket, one thread The root episode opens the ticket; related episodes write to it. Messages for a related episode are titled *Update to incident …* and throttled to one every 10 minutes per incident. Approvals are only ever for a fix: a related episode with nothing of its own to fix asks no one. ## Settings | Option | Default | Meaning | | ----------- | ------- | --------------------------------------------------------------------------------------------------- | | `monitor_s` | 1800 | Quiet period before resolved; also the window in which alerts about a fix's target are its recovery | | `reopen_s` | 1800 | A resolved incident's own alert this soon reopens it | | `settle_s` | 120 | How long a related episode waits for recovery alerts to stop | Incidents are kept in `data/incidents/.json` and reloaded at start: a restart forgets no open incident.