How 3AM decides
Root causes and packs
How 3AM turns an alert into a proven cause and a proposed fix.
3AM's knowledge of each technology comes in a pack: a set of root-cause classes. A class has:
- the signals it explains (alert names, such as
MySQLReadOnlyorKubePodCrashLooping); - read-only checks that prove or rule it out;
- which checks must pass to confirm it, and which only add support;
- the remediation: an exact, executable fix, or advice for a person when the fix needs judgement.
A pack applies when 3AM finds its technology in your estate. A firing alert that names a pack's signal turns that pack on as well, so a gap in discovery never makes 3AM blind to an alert.
From alert to fix
An alert opens an incident. 3AM works out the service, its owner, repositories and dependencies.
It selects the classes whose signals match the alerts, and runs their checks against live systems, in parallel.
Symptoms build up, so if nothing is proven yet, it looks again before giving up.
A class is confirmed when its required checks pass, refuted when they all fail, otherwise inconclusive.
For the first confirmed class with an executable fix, 3AM fills the fix in from the evidence and the alert, for example which session to kill, or which Deployment to roll back.
The policy decides: record it (shadow), ask for approval (L1), or refuse (with the reason).
After an approved change, 3AM re-runs the class's checks until the cause has cleared.
Packs available today
| Pack | Root causes |
|---|---|
| Kubernetes | Bad release, dependency down, memory limit too low, missing config, image unavailable, unschedulable |
| Prometheus / Alertmanager | Target unreachable, target refusing credentials, target answering garbage, config reload failed, rules failing, no Alertmanager, notifications failing |
| MySQL / MariaDB | Table lock blocking, row-lock contention, connection exhaustion, database unreachable, read-only primary, replication lag |
| nginx edge | Rate limit too strict, timeouts too low, expired TLS certificate, authentication failures, authorisation failures |
| ActiveMQ | Broker down, consumer stopped or slow, dead letters growing |
| JVM / Tomcat | Instance crashing, heap pressure, JDBC pool exhaustion, version drift between instances |
| Apache Fineract | Stuck batch job, failing batch job, GL mapping gaps, unusable GL accounts, unbalanced journals, duplicate postings, disabled event hooks, maker-checker backlog |
Packs are sealed and licensed (see Licensing).