# 3AM Docs

> Documentation for 3AM (3am.si), the on-prem AI on-call engineer for regulated enterprises running systems that can’t
> be rewritten. 3AM watches alerts, proves the root cause with read-only checks against live systems, and fixes it only
> after a person approves, inside the customer’s own network.

How to read these docs as an LLM or an agent:
- Every page is available as Markdown: add `.md` to its URL (for example https://docs.3am.si/connectors/kubernetes.md), or send `Accept: text/markdown`.
- All pages in one file: https://docs.3am.si/llms-full.txt
- Each Markdown page starts with its title, a one-line summary and its canonical URL; connector settings are included as tables.
- Product overview, security and pricing questions: https://3am.si/llms.txt

## Start here

- [Overview](https://docs.3am.si/index.md): 3AM is the on-prem AI on-call engineer. Install it inside your network, connect your tools, and let it prove causes and fix incidents with your approval.

## Get started

- [How 3AM works](https://docs.3am.si/getting-started.md): What 3AM does with an alert, from the moment it fires to a verified fix, told through one real incident.
- [Requirements](https://docs.3am.si/getting-started/requirements.md): What to have ready before installing 3AM. A Linux server or a Kubernetes cluster, somewhere for the models to run, and access to the tools it should watch.
- [Quickstart](https://docs.3am.si/getting-started/quickstart.md): Install 3AM on one Linux server with a free trial, open the console, connect Prometheus and Slack, and see your first incident. About 30 minutes.
- [Architecture and security](https://docs.3am.si/getting-started/architecture.md): What runs where, what connects to what, where 3AM keeps its data, and how access, secrets and the audit log are protected.

## Install

- [Install](https://docs.3am.si/install.md): One command installs 3AM on a Linux server or into a Kubernetes cluster. Sites without internet install from a signed bundle.
- [Linux](https://docs.3am.si/install/linux.md): Install 3AM on a Linux server or VM with one command. What the installer asks, how to open the console, the options worth knowing, RHEL, and Compose.
- [Kubernetes](https://docs.3am.si/install/kubernetes.md): Install 3AM into a Kubernetes or OpenShift cluster, with one command or with Helm directly. Models, the console, the values you'll change, and upgrades.
- [Air-gapped sites](https://docs.3am.si/install/air-gapped.md): Install 3AM where servers can't reach the internet. Download a signed release elsewhere, check it, carry it in, and install it with nothing fetched.
- [Models](https://docs.3am.si/install/models.md): Where 3AM's models run. Your own endpoint, OpenRouter, or a GPU in the server, never the CPU. How to choose, and how to set each one up.
- [Sizing](https://docs.3am.si/install/sizing.md): How big the 3AM server should be, why it can be small, what limits how many incidents it works at once, and GPU sizing for running the models yourself.
- [Upgrade](https://docs.3am.si/install/upgrade.md): Upgrade 3AM with the same command you installed it with. Your data and settings are kept, and the previous version comes back if the new one doesn't start.
- [Backup and restore](https://docs.3am.si/install/backup-restore.md): Back up 3AM's data volume, and restore it on the same server or a new one.
- [Uninstall](https://docs.3am.si/install/uninstall.md): Remove 3AM, keeping its data or deleting it.
- [Configuration files](https://docs.3am.si/install/configuration-files.md): Keep 3AM's connections and policy in files, for GitOps or Ansible, instead of setting them in the console. And how to reference secrets.

## Licensing

- [How licensing works](https://docs.3am.si/licensing.md): What a 3AM licence is, the three tiers, how services are counted, and what happens when a licence changes or runs out.
- [Start a trial](https://docs.3am.si/licensing/trial.md): A free 90-day trial for your company (a limited offer; usually 30 days), on up to 10 installs. The code is sent to your work email.
- [Get a licence](https://docs.3am.si/licensing/request.md): Moving on from a trial, or buying 3AM. You send one install's fingerprint once, and add every other install yourself.
- [Install the licence](https://docs.3am.si/licensing/install.md): Upload a licence file in the console, keep it in your configuration folder, or mount it as a Kubernetes Secret.
- [Add and remove installs](https://docs.3am.si/licensing/installs.md): Add a DR site, staging or a new region to your licence yourself, from the console, with no email to us. And take one off.
- [Keychain](https://docs.3am.si/licensing/keychain.md): Set 3AM up once, and every other install of yours takes the same setup and stays in step. Enterprise and Strategic.
- [Renew](https://docs.3am.si/licensing/renew.md): How renewals reach your installs, what the grace period is, and how to renew an air-gapped install.
- [Licence troubleshooting](https://docs.3am.si/licensing/troubleshooting.md): What each licence, trial code, install and Keychain message means, and what to do about it.

## Console

- [The console](https://docs.3am.si/console.md): A map of the 3AM console. Every page, what it's for, and the keyboard shortcut that opens it.
- [Setup wizard](https://docs.3am.si/console/setup-wizard.md): The steps a new install walks you through, from sign-in and the licence to connecting your tools and discovering your estate. No files to edit.
- [Overview](https://docs.3am.si/console/overview.md): The console's home page. What 3AM is judged by (ended by 3AM, time given back, actions verified, hand-overs, causes proven), what needs you, and 3AM's own health.
- [Incidents](https://docs.3am.si/console/incidents.md): Every incident from the first alert to its result, live while 3AM investigates; the diagnosis, the hand-over, what your team did; every decision, outcome and result explained.
- [Approvals page](https://docs.3am.si/console/approvals.md): The record of every approval request, its channels, who decided, and withdrawing one that is pending.
- [Digest](https://docs.3am.si/console/digest.md): What 3AM did, or in shadow would have done, next to what your team did; agreement, readiness for L1, and how to tell 3AM what your team did.
- [Impact and the lookback](https://docs.3am.si/console/impact.md): The business report in your own money (what 3AM returned, where it came from, how the figures are made), saved as PDF, and the lookback over your last 90 days of incidents.
- [Estate](https://docs.3am.si/console/estate.md): Your services, owners, teams, dependencies, alerts, workloads, databases and repositories, where each link came from, what isn't placed yet, and how discovery never drops anything.
- [Study](https://docs.3am.si/console/study.md): Before anything breaks, 3AM rehearses every cause it could be asked to prove, records what normal looks like, reads who talks to whom, and lists exactly what stands in the way.
- [Autonomy, drafted fixes and the halt switch](https://docs.3am.si/console/autonomy.md): What 3AM may do, service by service (off, shadow, L1, L2), readiness for L1, drafted fixes, and how to stop everything at once.
- [Connections](https://docs.3am.si/console/connections.md): Everything 3AM reads, acts through and asks for approval on, tested at any time.
- [Learned](https://docs.3am.si/console/learned.md): What 3AM has learned on your estate, how accurate it has been, and the learned fixes waiting for you.
- [Audit log](https://docs.3am.si/console/audit-log.md): Every signal, check, approval and action, append-only, hash-chained and signed.
- [3AM Doctor](https://docs.3am.si/console/doctor.md): Is 3AM itself healthy? Every check (core, licence, intake, models and their placement, connections, approvals, accuracy, the audit log, disk) with what to do, in the console, the CLI and the API.
- [Team](https://docs.3am.si/console/team.md): People, roles, the team hierarchy that owns services and approves fixes, who may sign in, and passwords for the rare person who can't receive a code.
- [Branding](https://docs.3am.si/console/branding.md): Put your organisation's name, logo and look on the console, and point Help at your own support.
- [Keychain page](https://docs.3am.si/console/keychain.md): The console's Keychain page, with its two tabs, what it shows on the home and on a member, and who can use it.
- [Settings](https://docs.3am.si/console/settings.md): The licence card, your numbers, the policy (default autonomy, quorums, drafted fixes, budgets), approval escalation, sign-in and e-mail, and the policy rules.
- [Guide](https://docs.3am.si/console/guide.md): What would make 3AM more useful next, what it can see of your estate (coverage, service by service), what each page is for, and the tips.
- [Your account and appearance](https://docs.3am.si/console/account.md): Your profile, role and teams, chat handles, time zone, your own look (theme, accent, density, motion), and signing out.
- [Keyboard shortcuts](https://docs.3am.si/console/keyboard.md): Every console shortcut. The command bar, search, going to any page with g, moving through lists, and the rules (Tab, Enter, Escape, arrows) every page follows.

## Access

- [Sign-in](https://docs.3am.si/access/sign-in.md): How people sign in to 3AM, how the first owner is created, what the admin token is for, setting up email, and who may sign in.
- [Single sign-on](https://docs.3am.si/access/sso.md): Let people sign in with your identity provider (OIDC) or directory (LDAP, Active Directory), with roles and teams coming from directory groups.
- [Roles and teams](https://docs.3am.si/access/roles.md): The four roles and what each can do, how teams own services, and who is asked to approve a fix.

## Approvals

- [Approvals](https://docs.3am.si/approvals.md): Where people approve fixes, who may, how requests escalate, and when two approvers are needed.

## How 3AM decides

- [How 3AM finds the cause](https://docs.3am.si/how-3am-decides.md): The investigation behind every incident, what "proven" means, and the confidence 3AM reports.
- [Incidents and grouping](https://docs.3am.si/how-3am-decides/incidents.md): One failure, one incident. How 3AM groups the alerts a failure sets off, an incident's lifecycle, and which incident is worked first.
- [The investigation](https://docs.3am.si/how-3am-decides/investigation.md): How 3AM investigates an incident, stage by stage, the rules for calling a cause proven, and what the incident profile records.
- [Evidence](https://docs.3am.si/how-3am-decides/evidence.md): What 3AM reads, what counts as proof, how your data is protected, and what's kept for every check.
- [Packs](https://docs.3am.si/how-3am-decides/packs.md): How a pack defines a cause, how 3AM proves it, and how the fix is chosen and carried out.
- [Resolution, restore first and drafted fixes](https://docs.3am.si/how-3am-decides/resolution.md): How 3AM gets a service back, not just diagnosed. Which fixes may run without asking, trying the next fix, restoring service when nothing is proven, the hand-over, and fixes 3AM drafts itself.
- [Safety rails](https://docs.3am.si/how-3am-decides/safety.md): Shadow mode, per-service autonomy, policy rules, rehearsals, budgets and the halt switch, with the settings for each.
- [Models](https://docs.3am.si/how-3am-decides/models.md): The two models 3AM uses, what each is allowed to do, the rules 3AM enforces on them, and what a hosted model sees.
- [Learning](https://docs.3am.si/how-3am-decides/learning.md): How 3AM gets better on your estate, what your team's answers do, and what it will never learn on its own.

## Operations

- [Running 3AM](https://docs.3am.si/operations.md): Day-to-day care of a 3AM install. Checking its health, the audit log, the halt switch and troubleshooting.
- [Health, Doctor and preflight](https://docs.3am.si/operations/health.md): Check that 3AM is up, that every part of it works, that its host is fit, and that it can reach every tool.
- [Logs and the audit log](https://docs.3am.si/operations/logs-and-audit.md): Where 3AM logs, how to search its decisions, and how to prove nothing was altered.
- [Halt switch](https://docs.3am.si/operations/halt.md): Stop every action at once, and resume.
- [Troubleshooting](https://docs.3am.si/operations/troubleshooting.md): Common problems with a running 3AM, what causes them, and how to fix them. Start with Doctor.

## Connectors

- [Connectors](https://docs.3am.si/connectors.md): What each connector gives 3AM, what "certified" means, and how to add one.
- [Alert webhooks (any SIEM)](https://docs.3am.si/connectors/webhooks.md): Push alerts to 3AM from any monitoring tool or SIEM with one webhook (Alertmanager, Grafana, Datadog, Splunk, PagerDuty, Azure Monitor and Sentinel, Elastic and OpenSearch, or generic JSON), no connector needed.
- [PagerDuty](https://docs.3am.si/connectors/pagerduty.md): Pages become incidents, 3AM writes its diagnosis on the page, and on-call engineers approve fixes with a note.
- [Kubernetes](https://docs.3am.si/connectors/kubernetes.md): Pod and rollout failures as incidents, the cause read from the cluster, and approved rollbacks, restarts and volume growth.
- [Prometheus and Alertmanager](https://docs.3am.si/connectors/prometheus.md): Alerts in, PromQL as evidence for every pack, approved silences, and fixes when your monitoring itself breaks.
- [VictoriaMetrics](https://docs.3am.si/connectors/victoriametrics.md): Alerts from vmalert or Alertmanager, MetricsQL as evidence, and fixes when VictoriaMetrics itself is in trouble. Single-node and cluster.
- [Argo CD](https://docs.3am.si/connectors/argocd.md): Deploy history as evidence, the GitOps-correct rollback, and fixes when Argo CD itself breaks.
- [ServiceNow](https://docs.3am.si/connectors/servicenow.md): Incidents and change records kept current, recent changes as evidence, knowledge articles in every diagnosis, fixes ordered through the catalogue, and ServiceNow's own failures handled.
- [Databases (MySQL, PostgreSQL)](https://docs.3am.si/connectors/databases.md): Database evidence for every incident, fixes on the database itself, and the right response when the database is the one that's down.
- [Servers (SSH)](https://docs.3am.si/connectors/servers.md): Diagnose and fix applications on Linux servers over SSH, through a gate that runs only what each server allows. 3AM never gets a shell.
- [More connectors](https://docs.3am.si/connectors/more.md): Connectors available today whose live certification is under way, with the settings each one needs.

## Reference

- [Reference](https://docs.3am.si/reference.md): The cheat sheet, the HTTP API, the query language, every audit event, recipes, the command line, every environment variable, the network, and the glossary.
- [Cheat sheet](https://docs.3am.si/reference/cheat-sheet.md): Everything on one page. Install, upgrade, models, Doctor, CLI, the API calls you use most, Keychain, licence, keyboard shortcuts, query language, files and ports.
- [HTTP API](https://docs.3am.si/reference/api.md): Every endpoint of 3AM's API (status, incidents, search, approvals, autonomy, policy, connections, estate, study, impact, lookback, licence, installs, Keychain, access, ledger, events, alerts) with who may call it and examples.
- [Query language](https://docs.3am.si/reference/query-language.md): One line to search incidents, approvals and the audit log on the server, across years: fields, operators, wildcards, regular expressions, numbers, durations, dates and the time window.
- [Audit events](https://docs.3am.si/reference/audit-events.md): Every event type 3AM writes to its audit log, by family, with when it is written. Search them in the console or follow them from your SIEM.
- [Recipes](https://docs.3am.si/reference/recipes.md): Copy-paste automation with the HTTP API, from CI pipelines to your SIEM.
- [Command line](https://docs.3am.si/reference/cli.md): Every 3am-core command, with what it prints.
- [Environment variables](https://docs.3am.si/reference/environment.md): Every THREEAM_* variable the 3AM container reads, with its default and meaning, grouped by area (core, intake and concurrency, models, investigation, causal, mail, licence, Keychain, syslog, FIPS).
- [Network](https://docs.3am.si/reference/network.md): The ports 3AM listens on and the connections it makes.
- [Glossary](https://docs.3am.si/reference/glossary.md): Every 3AM term in one place, A to Z, with where to read more. Admin token, autonomy, Keychain, member, pack, proven, restore move, site key, System 1, Warden and the rest.

## Optional

- [3AM home](https://3am.si/): what 3AM is and who it is for
- [Security](https://3am.si/security): deployment, data handling and the audit ledger
- [For banks](https://3am.si/banking): core-banking operations