# The investigation > The flow of methods behind every incident, the evidence gatherer, the dynamic RCA engine, Bayesian inference and the next test, causal discovery, the tree search, and the incident profile. Source: https://docs.3am.si/how-3am-decides/investigation · Markdown: https://docs.3am.si/how-3am-decides/investigation.md · All docs: https://docs.3am.si/llms.txt 3AM is an on-prem AI on-call engineer for regulated enterprises (https://3am.si). An incident is investigated by a **flow**: a graph of methods, each gathering evidence or testing it. It ends in an **incident profile**: the chain from root cause to symptom, how sure 3AM is, and everything it tried. Only a *proven* profile leads to an action. ## The flow | Node | What it does | Next | | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ | | `checks` | Runs the checks of every pack cause whose signals include the alerts | proven → `context`; otherwise → `widen` | | `context` | A proven cause still gets its context for the approver: changes, dependencies, drift, signatures, evidence | → `profile` | | `widen` | All at once: dependencies, recent changes, **what changed**, **what is abnormal**, **new errors**, the hosts, telemetry, and **the alert's own questions** (the evidence gatherer) | proven or linked → `profile`; a lead → `follow`; otherwise → `rca` | | `follow` | Hands each lead (a full disk, a host down) to the packs as the signal it would raise | proven or linked → `profile`; otherwise → `rca` | | `rca` | The dynamic RCA engine (below) | proven or cleared → `profile`; otherwise → `mitigate` | | `mitigate` | At L2 only: a safe mitigation before proof, when the suspect is strong and the impact high ([Resolution](https://docs.3am.si/how-3am-decides/resolution#mitigate-first)) | → `infer` | | `infer` | Ranks every hypothesis, runs the tests that best tell them apart, promotes a change when the rules allow | proven → `profile`; otherwise → `recheck` | | `recheck` | Every 30 s for 180 s: new alerts join, the packs run again, dependencies are walked again | proven, linked or cleared → `profile`; otherwise → `infer_again` | | `investigate` | The investigator reads through declared read actions ([Models](https://docs.3am.si/how-3am-decides/models)) | proven or cleared → `profile`; otherwise → `infer_last` | | `profile` | Builds the incident profile | end | **Guards:** 24 steps; a time budget (`investigate_s`, 240 s) after which the flow goes to `profile`. Every node is a `flow.node` event, and the current one is shown live. If the alerts stop during the investigation, it closes as `cleared` and nothing is acted on. ## The evidence gatherer For every incident packs don't prove at once, 3AM asks the questions the alert itself implies, in parallel with the sensors: | Kind | What | Dialects | | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------ | | Metrics | Targets down; 5xx ratio and p99 latency now against an hour ago (abnormal at ≥ 2× and above a floor); restarts in 15 min; OOM kills; pods over 90% of their memory limit | PromQL (Prometheus, VictoriaMetrics, Mimir, Thanos), Datadog | | Logs | Error lines in the last 10–15 min | LogQL (Loki), Datadog, SPL (Splunk) | | Traces | Failing spans | TraceQL (Tempo) | | State | Every read action whose required parameters the alert's labels supply (`namespace` + `deployment` → workload health; `pod` → logs; `instance` → host status) | Any connector's declared read actions | It only reads; free-form parameters (statement, query, command, body) are never bound; each item is a sentence plus the exact query; a query that errors or times out (30 s) is recorded, never fatal. ## The dynamic RCA engine When nothing is proven yet: 1. **Impact:** the blast radius (each dependent affected with p = 0.85^hops, or 1 when firing), business impact (from tier or weight), technical impact (severity, share firing). 2. **System 1 scores what the other methods found:** every hypothesis gets its fit against the whole evidence state, and its **coverage** (the share of failing services a fault there would explain). 3. **The investigator proposes a differential:** up to 5 distinct mechanisms across categories (change, capacity, dependency, infrastructure, data or state, the application), each with a confirm check. 4. **Test by information gain:** most informative first, up to 4 per round, each through the proof rules. 5. **Adapt:** if nothing is proven, the investigator sees what was refuted and proposes again (2 rounds). Fit and coverage change what is tested and shown first; they never decide what is acted on. ## Inference Every piece of evidence becomes a hypothesis (pack verdicts, drift, state changes, anomalies, new signatures, an upstream incident, learned patterns, the investigator's ideas, causal links), scored in log space: | Term | Meaning | | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Prior | Learned from your incidents (how often this kind of cause was right here), else a default: pack 0.5, upstream 0.5, learned 0.4, drift 0.2, signature 0.15, model 0.15, anomaly 0.08, state 0.06 | | Timing | A cause starts before its symptom: a bonus that decays with the gap, a penalty if it began after | | Topology | The most probable dependency path to the alerting service (one hop 0.85; no path 0.05) | | Evidence | A check, System 1's verdict, an anomaly's strength, System 1's fit, coverage | | Corroboration | Two methods naming the same entity | Normalised into a posterior with its entropy. **The next test** is the candidate with the highest expected information gain, run read-only, then Bayes updates the posterior (up to 6 tests). The console shows every hypothesis with its posterior and the tests chosen. **The posterior decides what to test and show first, never what to act on.** ## Causal discovery (`--causal`) With `3am-causal` running, the series that shifted on the service and its neighbours are related over the last hour by **PCMCI+**. Links between services become findings, dependency paths for this incident's ranking, and evidence for the learned map of your estate (earning or losing confidence from outcomes). A weekly graph-network job (`ml.dgnn`) learns how failures propagate from your history. Artifacts are checked by digest, and by signature with `THREEAM_ML_TRUST`. ## The tree search (MCTS) `--investigator mcts` (or Helm `investigator: mcts`): Monte Carlo tree search over reads, with System 2 proposing, System 1 scoring and pruning, and the full proof ending the search. Same tools, same rules. The default is the step loop; switch once replays of your recorded incidents show the search proving more, in fewer calls. ## The proof rules | Path | Proven when | | ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | A pack cause | Its required checks pass, **and** it wasn't already true 30 minutes before the first alert. If System 1 then reads the evidence as contradicting it at ≥ 0.8, it is `disputed` and goes to a person | | A change (drift) | In the top 3 with posterior ≥ 0.3, **and** a re-read confirms it's still in place, **and** a second, independent line agrees, **and** System 1 reads "this change explains the incident" as supported. It then becomes a cause whose check is the fact itself, so a fix is verified by the fact going back | | The investigator's idea | Its own check passes twice, **and** System 1 verifies it at ≥ 0.8, **and** an independent line agrees | | Upstream | A proven incident on a service this one depends on: **linked**, not proven, and no second fix | **A second line** must come from the effect side (the alerts, telemetry, anomalies, signatures, health changes, confirmed pack causes), gathered by methods that never saw the claim. The investigator agreeing with a change never counts: it was shown the change. **State facts** (a target down, a host unreachable) are leads, never a proven root. ## The incident profile | Field | Meaning | | --------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | | `confidence` | `proven`, `linked`, `suspected`, `disputed`, `unproven` or `cleared` | | `why` | The chain: root (or a change as root plus a `via` link), then the symptom; each link with evidence sentences and System 1's verdict | | `ruled_out`, `chronic`, `open`, `not_checked` | Refuted; already true before; inconclusive; unable to run here (with why) | | `tried`, `findings` | Every method, what it did and found, and how long it took | | `ranked`, `tests` | Every hypothesis with its posterior; the tests chosen | | `knowledge` | Runbook excerpts about the proven cause, each verified by System 1 | Evidence is written as sentences ("Check rate_403 on prometheus returned 0.316 (expected > 0.02: condition met)"): System 1 reads sentences reliably, raw data badly.