The investigation
The flow of methods behind every incident, the evidence gatherer, the dynamic RCA engine, Bayesian inference and the next test, causal discovery, the tree search, and the incident profile.
An incident is investigated by a flow: a graph of methods, each gathering evidence or testing it. It ends in an incident profile: the chain from root cause to symptom, how sure 3AM is, and everything it tried. Only a proven profile leads to an action.
The flow
| Node | What it does | Next |
|---|---|---|
checks | Runs the checks of every pack cause whose signals include the alerts | proven → context; otherwise → widen |
context | A proven cause still gets its context for the approver: changes, dependencies, drift, signatures, evidence | → profile |
widen | All at once: dependencies, recent changes, what changed, what is abnormal, new errors, the hosts, telemetry, and the alert's own questions (the evidence gatherer) | proven or linked → profile; a lead → follow; otherwise → rca |
follow | Hands each lead (a full disk, a host down) to the packs as the signal it would raise | proven or linked → profile; otherwise → rca |
rca | The dynamic RCA engine (below) | proven or cleared → profile; otherwise → mitigate |
mitigate | At L2 only: a safe mitigation before proof, when the suspect is strong and the impact high (Resolution) | → infer |
infer | Ranks every hypothesis, runs the tests that best tell them apart, promotes a change when the rules allow | proven → profile; otherwise → recheck |
recheck | Every 30 s for 180 s: new alerts join, the packs run again, dependencies are walked again | proven, linked or cleared → profile; otherwise → infer_again |
investigate | The investigator reads through declared read actions (Models) | proven or cleared → profile; otherwise → infer_last |
profile | Builds the incident profile | end |
Guards: 24 steps; a time budget (investigate_s, 240 s) after which the flow goes to profile. Every node is a
flow.node event, and the current one is shown live. If the alerts stop during the investigation, it closes as
cleared and nothing is acted on.
The evidence gatherer
For every incident packs don't prove at once, 3AM asks the questions the alert itself implies, in parallel with the sensors:
| Kind | What | Dialects |
|---|---|---|
| Metrics | Targets down; 5xx ratio and p99 latency now against an hour ago (abnormal at ≥ 2× and above a floor); restarts in 15 min; OOM kills; pods over 90% of their memory limit | PromQL (Prometheus, VictoriaMetrics, Mimir, Thanos), Datadog |
| Logs | Error lines in the last 10–15 min | LogQL (Loki), Datadog, SPL (Splunk) |
| Traces | Failing spans | TraceQL (Tempo) |
| State | Every read action whose required parameters the alert's labels supply (namespace + deployment → workload health; pod → logs; instance → host status) | Any connector's declared read actions |
It only reads; free-form parameters (statement, query, command, body) are never bound; each item is a sentence plus the exact query; a query that errors or times out (30 s) is recorded, never fatal.
The dynamic RCA engine
When nothing is proven yet:
- Impact: the blast radius (each dependent affected with p = 0.85^hops, or 1 when firing), business impact (from tier or weight), technical impact (severity, share firing).
- System 1 scores what the other methods found: every hypothesis gets its fit against the whole evidence state, and its coverage (the share of failing services a fault there would explain).
- The investigator proposes a differential: up to 5 distinct mechanisms across categories (change, capacity, dependency, infrastructure, data or state, the application), each with a confirm check.
- Test by information gain: most informative first, up to 4 per round, each through the proof rules.
- Adapt: if nothing is proven, the investigator sees what was refuted and proposes again (2 rounds).
Fit and coverage change what is tested and shown first; they never decide what is acted on.
Inference
Every piece of evidence becomes a hypothesis (pack verdicts, drift, state changes, anomalies, new signatures, an upstream incident, learned patterns, the investigator's ideas, causal links), scored in log space:
| Term | Meaning |
|---|---|
| Prior | Learned from your incidents (how often this kind of cause was right here), else a default: pack 0.5, upstream 0.5, learned 0.4, drift 0.2, signature 0.15, model 0.15, anomaly 0.08, state 0.06 |
| Timing | A cause starts before its symptom: a bonus that decays with the gap, a penalty if it began after |
| Topology | The most probable dependency path to the alerting service (one hop 0.85; no path 0.05) |
| Evidence | A check, System 1's verdict, an anomaly's strength, System 1's fit, coverage |
| Corroboration | Two methods naming the same entity |
Normalised into a posterior with its entropy. The next test is the candidate with the highest expected information gain, run read-only, then Bayes updates the posterior (up to 6 tests). The console shows every hypothesis with its posterior and the tests chosen. The posterior decides what to test and show first, never what to act on.
Causal discovery (--causal)
With 3am-causal running, the series that shifted on the service and its neighbours are related over the last hour by
PCMCI+. Links between services become findings, dependency paths for this incident's ranking, and evidence for the
learned map of your estate (earning or losing confidence from outcomes). A weekly graph-network job (ml.dgnn) learns
how failures propagate from your history. Artifacts are checked by digest, and by signature with THREEAM_ML_TRUST.
The tree search (MCTS)
--investigator mcts (or Helm investigator: mcts): Monte Carlo tree search over reads, with System 2 proposing,
System 1 scoring and pruning, and the full proof ending the search. Same tools, same rules. The default is the step
loop; switch once replays of your recorded incidents show the search proving more, in fewer calls.
The proof rules
| Path | Proven when |
|---|---|
| A pack cause | Its required checks pass, and it wasn't already true 30 minutes before the first alert. If System 1 then reads the evidence as contradicting it at ≥ 0.8, it is disputed and goes to a person |
| A change (drift) | In the top 3 with posterior ≥ 0.3, and a re-read confirms it's still in place, and a second, independent line agrees, and System 1 reads "this change explains the incident" as supported. It then becomes a cause whose check is the fact itself, so a fix is verified by the fact going back |
| The investigator's idea | Its own check passes twice, and System 1 verifies it at ≥ 0.8, and an independent line agrees |
| Upstream | A proven incident on a service this one depends on: linked, not proven, and no second fix |
A second line must come from the effect side (the alerts, telemetry, anomalies, signatures, health changes, confirmed pack causes), gathered by methods that never saw the claim. The investigator agreeing with a change never counts: it was shown the change. State facts (a target down, a host unreachable) are leads, never a proven root.
The incident profile
| Field | Meaning |
|---|---|
confidence | proven, linked, suspected, disputed, unproven or cleared |
why | The chain: root (or a change as root plus a via link), then the symptom; each link with evidence sentences and System 1's verdict |
ruled_out, chronic, open, not_checked | Refuted; already true before; inconclusive; unable to run here (with why) |
tried, findings | Every method, what it did and found, and how long it took |
ranked, tests | Every hypothesis with its posterior; the tests chosen |
knowledge | Runbook excerpts about the proven cause, each verified by System 1 |
Evidence is written as sentences ("Check rate_403 on prometheus returned 0.316 (expected > 0.02: condition met)"): System 1 reads sentences reliably, raw data badly.
Incidents and grouping
One failure, one incident. How alerts are grouped (refire, remediation effect, same service, dependency, shared entity), the lifecycle, what a related episode concludes, priority, one ticket and one thread.
Evidence
What 3AM reads, what counts as proof, how your data is protected, and what's kept for every check.