Databases (MySQL, PostgreSQL)

With a database connected, 3AM reads the database directly: whether it's up and why not, who holds the connections, which session is blocking the others, how replication is doing. It uses that evidence in two ways:

• For the database's own incidents. It ends the session holding a lock, closes a leaking client's idle sessions, makes a primary writable again, or restarts replication that stopped cleanly. Each fix is rehearsed and approved.
• For other incidents. When an application's log says it can't reach its database, 3AM checks the database itself instead of trusting the log line.

When the database itself is down, 3AM proves why before touching anything:

MySQL and MariaDB, and PostgreSQL, are fully supported. Oracle and SQL Server support reads and approved data changes.

Before you start

• One connection per database instance (or cluster endpoint) you want 3AM to watch.
• Two accounts on each: a read account for evidence, and an admin account for approved fixes.
• For databases in Kubernetes, the Kubernetes connector with the database's namespace, so 3AM can restart its pod or grow its volume.

1. Create the accountsCopy Anchor Link

-- evidence: sessions, locks, replication, and the application's tables for packs that read them
CREATE USER 'threeam_read'@'%' IDENTIFIED BY '…';
GRANT PROCESS, REPLICATION CLIENT ON *.* TO 'threeam_read'@'%';
GRANT SELECT ON performance_schema.* TO 'threeam_read'@'%';
GRANT SELECT, EXECUTE ON sys.* TO 'threeam_read'@'%';
GRANT SELECT ON orders.* TO 'threeam_read'@'%';            -- the application's schema

-- approved fixes: end sessions, read-only off, start replication
CREATE USER 'threeam_admin'@'%' IDENTIFIED BY '…';
GRANT PROCESS, REPLICATION CLIENT, CONNECTION_ADMIN, SYSTEM_VARIABLES_ADMIN, REPLICATION_SLAVE_ADMIN ON *.* TO 'threeam_admin'@'%';
GRANT SELECT ON performance_schema.* TO 'threeam_admin'@'%';
GRANT SELECT, EXECUTE ON sys.* TO 'threeam_admin'@'%';
GRANT SELECT ON orders.* TO 'threeam_admin'@'%';

2. Add the connectorCopy Anchor Link

One connector per instance. In connectors.json:

{ "name": "ledger-db", "kind": "sql",
  "settings": { "engine": "postgres", "host": "ledger-db-0.ledger-db.bank-prod.svc", "database": "ledger",
                "username": "threeam_admin", "password": { "file": "/etc/3am/secrets/ledger-db-admin" },
                "read_username": "threeam_read", "read_password": { "file": "/etc/3am/secrets/ledger-db-read" },
                "service": "ledger", "expected_role": "primary" } }

expected_role matters. 3AM makes a read-only database writable again only if it's declared the primary. A demoted old primary also looks like "not a replica", and making it writable would give you two primaries.

When an alert names a database (its instance or host label), 3AM checks and fixes that database, and no other. If it can't tell which of several databases an alert means, it asks instead of guessing.

3. VerifyCopy Anchor Link

postgres 17.11, primary, 1/30 connections

Test connection also tries the admin account, and names any view the read account can't see.

What it detectsCopy Anchor Link

Alerts from Prometheus exporters (MySQLDown, MySQLTooManyConnections, PostgresqlDown, ...) reach the same causes.

What it proves and fixesCopy Anchor Link

ActionsCopy Anchor Link

CertificationCopy Anchor Link

Live on MySQL 8.4 and PostgreSQL 17, with the accounts above and 3AM inside a Kubernetes cluster: 60 of 60 checks passed (2026-10-05). The oldest supported versions, MySQL 8.0 and PostgreSQL 14, passed the same scenarios.

• The connector. Both accounts on primaries and replicas; the read account can't write; a wrong password and a name that doesn't resolve are explained.
• Locks. An idle transaction held a row lock on each engine. 3AM ended the blocking session, and the waiting update went through.
• Connection leaks. A client leaked connections until none were left. On MySQL, 3AM read and fixed it through the admin account's reserved slot. On both engines it closed exactly the leaking client's idle sessions.
• Read-only primary. Made writable again and checked with a write. The replica, read-only by design, raised nothing.
• Replication. Stopped cleanly: started again. Broken by a duplicate key: advice, with the real error in the evidence.
• The database down. A frozen database and a killed database process were each restarted through Kubernetes and verified. A full disk crashed PostgreSQL, which recovered with writes failing. 3AM proved it from the log and didn't restart it.

TroubleshootingCopy Anchor Link