Prometheus and Alertmanager

What 3AM does with PrometheusCopy Anchor Link

1. Prepare accessCopy Anchor Link

3AM needs read access to Prometheus' HTTP API, and to Alertmanager's. It needs write access to Alertmanager only if it may create silences (always approved).

Adding a basic-auth user for 3AM to Prometheus' web.yml (the password must be a bcrypt hash):

htpasswd -nbBC 10 3am 'a-long-random-password' | cut -d: -f2

# web.yml
basic_auth_users:
  3am: $2y$10$…          # the hash from above

2. Exporters 3AM's checks useCopy Anchor Link

3AM's checks read standard exporter metrics. A stock exporter covers the core checks; a few need an extra flag:

When a metric is missing, the check is inconclusive, never passed. The MySQL pack confirms through SQL where a metric isn't standard.

3. Connect it in 3AMCopy Anchor Link

Settings → Add a connection → Prometheus / Alertmanager (or the wizard's Monitoring step):

Test connection runs a real query (not just a ping) and checks Alertmanager with its own credentials. A healthy result shows what 3AM found:

Connected. Prometheus 3.15.0 + Alertmanager 0.34.1
jobs: 6 (mysql, prometheus, …)   services: …   alert rules: 95   targets: 32

Network: outbound from 3AM to Prometheus and Alertmanager. Nothing connects to 3AM.

4. Check it end to endCopy Anchor Link

Create a test rule that always fires, and watch it reach 3AM:

# a rule file Prometheus loads
groups:
  - name: 3am-test
    rules:
      - alert: ThreeAMPipelineTest
        expr: vector(1)
        labels: {severity: critical, service: monitoring}

Reload Prometheus. Within a minute the console shows an incident for ThreeAMPipelineTest (3AM will find nothing to prove, and say so). Remove the rule afterwards.

What 3AM understands about monitoringCopy Anchor Link

3AM raises these itself from Prometheus' status, even when you have no alert rule for them. If your own rule for the same thing is already firing, 3AM uses yours instead:

Restarts and volume growth go through the Kubernetes connector, which finds the workload behind Prometheus', Alertmanager's or the target's address. Without it the same causes are handed to a person. Configuration causes (credentials, reload, rules, receivers) stay with a person, with the error attached.

SilencesCopy Anchor Link

TroubleshootingCopy Anchor Link

CertificationCopy Anchor Link

Certified on 2026-10-04 against Prometheus 3.15 (HTTPS with an internal CA, basic auth) and Alertmanager 0.34 behind a bearer-token proxy, with the real mysqld_exporter 0.20: 46 live checks. End to end: the database was set read-only, this Prometheus fired MySQLReadOnly through Alertmanager, 3AM proved it with this Prometheus' metric, fixed it (approved), and the alert resolved.

Fixes, certified live on 2026-10-04 (54 of 54 checks, on a real Kubernetes cluster, with 3AM inside it under the client RBAC). Components were hung mid-run, meaning their pod was Running and answered nothing. Each was detected, proven, located, rehearsed, approved, restarted and verified:

• vmagent, a vmstorage node (only that pod), a scrape target of vmagent and one of Prometheus, Alertmanager (seen by both monitoring connectors, restarted once) and Prometheus.

A read-only storage node got a proposal to grow its volume, approved by two people. The test storage driver accepts the request but can't grow volumes, and 3AM reported the fix as failed rather than claiming success.