Incidents and grouping

A real failure rarely raises one alert. The database goes read-only; during the fix and the recovery the same service raises knock-on alerts, the services that depend on it return 5xx, and the alert may flap once on its way out. 3AM groups them: an incident is what people see and what gets resolved; an episode is one investigation of one service's alerts. An incident owns every episode of the same failure, one ticket and one message thread, and outlives its fix by a monitoring window.

Two questions are kept apart:

• Grouping: does this alert belong to that incident? Decided at intake, by the criteria below.
• Attribution: does that incident's cause explain this alert? Decided by the investigation, under the proof rules.

LifecycleCopy Anchor Link

open ──(fix proposed)──▶ mitigating ──(verified)──▶ monitoring ──(quiet 30 min)──▶ resolved
  │                         │ rejected / not resolved      ▲  │ its own alert fires again
  └──(every alert stopped)──┼──────────────────────────────┘  └──────────────▶ open (reopened)
                            ▼
                          open                     resolved ──(own alert within 30 min)──▶ open

An incident still firing after its fix's monitoring window reaches people once, as an update to that incident.

Grouping criteriaCopy Anchor Link

New alerts are grouped by service, then scored against every live incident. The strongest relation wins; 0.6 or more joins. Every decision is an incident.joined or incident.opened event with its reasons.

An open incident explains new alerts only while its cause holds. Before it absorbs an alert or counts as the same failure, its proven causes are re-run (at most every 60 s). If the evidence no longer confirms the cause, it explains nothing new (episode.cause_cleared).

Baseline alerts. Alerts already firing when 3AM starts are known (chronic), not new; a re-fire after they clear is new. An episode waits 20 s for co-firing alerts before it opens.

What a related episode concludesCopy Anchor Link

1. Settle. If the incident's fix ran within 30 min and this isn't a refire, wait up to settle_s (120 s) for the alerts to stop. If they do: settled / cleared, the recovery's tail, no investigation, no page.
2. The same cause proven again: the fix didn't hold. recurrence_in_incident, a fix proposed again (with approval) inside the same incident.
3. A different cause proven: a contributing cause while the incident's cause holds (or about what the fix acted on); otherwise a new failure, split into its own incident (incident.split).
4. Nothing proven: linked to a proven upstream (downstream); explained by the incident's cause while it holds (related); one of its own alerts back after the fix (recurrence, an update); still firing after the fix and a full investigation (unexplained_after_fix, an update). Grouping is never silence.

Priority: which incident firstCopy Anchor Link

When investigations wait for a slot (Sizing):

Across incidents: +2 × impact (blast radius, criticality, how much is firing), +0.5 for a critical alert, +1 per 5 minutes waited. One slot is always kept for a new, independent incident. Every queued and started episode records its priority and each term (episode.queued, episode.started).

One ticket, one threadCopy Anchor Link

The root episode opens the ticket; related episodes write to it. Messages for a related episode are titled Update to incident … and throttled to one every 10 minutes per incident. Approvals are only ever for a fix: a related episode with nothing of its own to fix asks no one.

SettingsCopy Anchor Link

Incidents are kept in data/incidents/<id>.json and reloaded at start: a restart forgets no open incident.