Servers (SSH)

When an application on a server goes down, 3AM does what an on-call engineer would. It reads the service's status and logs, finds the error line, checks disk, memory and its port, and looks up the error in your runbooks and knowledge base. It then fixes the proven cause, with approval:

The guardrailsCopy Anchor Link

3AM never gets a shell, and no model ever writes a command. Three separate layers each stop a command nobody allowed:

1. 3AM. Hosts are on an allow-list with pinned host keys. Arguments are plain words only. Actions come only from the server's own catalogue, and 3AM refuses anything destructive (rm -rf, mkfs, dd, reboot, curl | sh, ...) even if a server allows it.
2. The server. 3AM's key may only run 3am-gate. The gate runs read-only diagnostics, plus the actions listed in /etc/3am/gate.conf, each parameter checked against its pattern. Everything else is refused and logged to syslog.
3. sudo. It allows exactly the privileged commands those actions need, and nothing more.

Reads change nothing and run without approval. Every action is rehearsed, approved and recorded. Secrets in logs (passwords, tokens, keys, card numbers) are masked before anything leaves the connector.

1. Install the gate on each serverCopy Anchor Link

As root, with the files from the bundle's ssh/ folder (3am-gate, gate.conf.example, sudoers.example); copy them to the server first:

useradd -m -s /bin/sh threeam
usermod -aG systemd-journal threeam                 # read the units' journals
install -o root -g root -m 0755 3am-gate /usr/local/bin/3am-gate
install -d -o root -g root -m 0755 /etc/3am
install -o root -g root -m 0644 gate.conf.example /etc/3am/gate.conf        # then edit it
visudo -f /etc/sudoers.d/threeam                    # from sudoers.example
install -d -o threeam -g threeam -m 0700 /home/threeam/.ssh
echo 'restrict,command="/usr/local/bin/3am-gate" ssh-ed25519 AAAA... 3am' > /home/threeam/.ssh/authorized_keys
chown threeam:threeam /home/threeam/.ssh/authorized_keys && chmod 0600 /home/threeam/.ssh/authorized_keys

restrict turns off port, agent and X11 forwarding and terminals for that key. The gate needs only POSIX sh, systemctl, journalctl and coreutils.

2. Say what 3AM may read and doCopy Anchor Link

log /var/log/payments/*.log
unit payments.service
path /opt/payments/*.jar
java_user payments
action restart_service unit=^payments\.service$ :: sudo -n /usr/bin/systemctl restart {unit}
action clean_rotated_logs dir=^/var/log/payments$ :: find {dir} -name '*.log.*' -type f -mtime +0 -delete

threeam ALL=(root) NOPASSWD: /usr/bin/systemctl restart payments.service
threeam ALL=(payments) NOPASSWD: /usr/bin/jcmd

log, unit and path say what 3AM may read. java_user is the account Java services run as, for thread dumps. The packs use the action names restart_service and clean_rotated_logs: keep them, and change the patterns and commands to fit your servers. An action that isn't listed can't run, whatever 3AM asks for.

3. Add the connectorCopy Anchor Link

{ "name": "servers", "kind": "ssh",
  "settings": { "hosts": ["app-01.bank.internal"], "user": "threeam",
                "key_file": "/etc/3am/secrets/3am-ssh-key", "known_hosts": "/etc/3am/secrets/known_hosts",
                "apps": { "payments": { "hosts": ["app-01.bank.internal"], "unit": "payments.service",
                                        "logs": ["/var/log/payments/app.log"], "artifact": "/opt/payments/app.jar",
                                        "port": 8080, "java": true, "start_marker": "payments starting" } } } }

known_hosts pins each server's host key. A key that changes is refused, and 3AM opens a HostKeyChanged incident instead of connecting. start_marker is a line the application logs when it starts. 3AM reads only the current run's logs, so yesterday's error isn't mistaken for today's cause.

4. VerifyCopy Anchor Link

1 host through 3am-gate (v1), 1 app

If a server's gate.conf allows an action that 3AM refuses to run, Test connection names it.

What it detectsCopy Anchor Link

Probe and monitoring alerts that carry app and host labels (SyntheticProbeFailing, EndpointDown, ...) reach the same causes. That's how 3AM finds a deadlocked service: it's running, but not answering.

RunbooksCopy Anchor Link

When the checks find the error line, 3AM searches your knowledge, such as ServiceNow knowledge bases and the runbooks in your repositories, for that exact error. The matching articles are named in the approval request and the incident note.

CertificationCopy Anchor Link

Live, on a real Linux server (Ubuntu 24.04 with systemd and sshd, a Java service on OpenJDK 21, sudo rules and the gate), with 3AM inside a Kubernetes cluster reaching it only through the gate. Every scenario passed (2026-10-05):

• The guardrails. 3AM reached the server only through the gate. It refused reboot_now, although that server's gate.conf allows it, and a restart of a unit outside the allowed pattern. Seven direct attempts with 3AM's own key to get past the gate (a shell, /etc/shadow, a path traversal, a disallowed unit, ;id) were all refused by the server. Port forwarding was refused, and so was a file the gate doesn't list. A password in the service's log reached 3AM masked.
• A crash. Nothing in the logs a restart wouldn't fix: restarted through the gate, and verified running.
• Out of memory. Proven from the OutOfMemoryError line and restarted. The knowledge base article for that error was cited in the approval request and the incident note.
• A full disk. Its old rotated logs were deleted (the action that server allows), then it was restarted.
• Its port taken. Advice naming the holder, though 3AM's unprivileged account can't see another user's process.
• A bad deploy. Advice to roll back, and no restart into the broken build.
• Its database down. Read from the service's own log, checked on the database: advice, and the service left alone.
• Its database back, the service given up. Restarted.
• A deadlock. Running but not answering: restarted, with the thread dump kept for the developers.
• The host key changed. A security incident, and 3AM stopped connecting to the server.

TroubleshootingCopy Anchor Link