How 3AM decides
Evidence
What a check is, what counts as proof, and why 3AM never guesses.
Every check is a query and a comparison that a person could rerun:
| Kind | Example | Runs through |
|---|---|---|
| Metric | max(mysql_global_variables_read_only) > 0 | Prometheus, Splunk or Datadog: each pack carries the query for each backend, and alternatives for different exporters |
| SQL | SHOW GLOBAL VARIABLES WHERE Variable_name = 'read_only' contains ON | A read-only database connection |
| Read | The workload's revisions and crash logs, a config file, scrape targets | A connector's read-only actions |
Rules of evidence:
- Only live data counts. Runbooks and past incidents can suggest where to look; only checks against live systems confirm a cause.
- Missing data is not proof. A check with no data, or no way to run on your install, is inconclusive and says why. It is never treated as passed.
- Look-alikes are kept apart. Classes that look the same from the outside have checks that exclude each other. A crash after a release is a bad release only if the logs show no dependency errors.
- Values from alerts are validated. A namespace or workload name taken from an alert must be a plain name before any check uses it. Anything else is refused and never sent.
- Everything is recorded. Each check's exact query, its raw result and the verdict go to the audit log under the incident's id.