VictoriaMetrics

3AM reads firing alerts from Alertmanager (silence-aware) or straight from vmalert, and runs every pack's PromQL on VictoriaMetrics unchanged. It also watches VictoriaMetrics itself, through each component's own /metrics. When storage turns read-only, vmagent can't deliver, a storage node drops out or a component hangs, 3AM proves which, and fixes it through the Kubernetes connector where it can.

Before you start

• Read access to the query endpoint and to each component's /metrics.
• Write access to Alertmanager silences and to /-/reload on vmagent and vmalert, only if 3AM may use them. Both are approved every time.
• Outbound access from 3AM to every URL below.

1. Create credentialsCopy Anchor Link

A read-only vmauth user for a cluster:

users:
  - username: 3am
    password: a-long-random-password
    url_map:
      - src_paths: ["/select/.*"]
        url_prefix: "http://vmselect:8481"

2. Decide what to connectCopy Anchor Link

The query endpoint alone can't show disk, queues or limits. Those live in each component's /metrics, so give 3AM the components' own addresses too.

On Kubernetes, list each vmstorage pod through its headless Service (vmstorage-0.vmstorage.monitoring.svc:8482, …) so 3AM can tell which node is in trouble, and restart only that one.

If you leave out the vmagents, 3AM can't see scrape targets or remote write. Leave out the cluster components and it can't see storage nodes. Checks that need them report inconclusive, never passed.

3. Add the connectorCopy Anchor Link

In the console: Settings → Add a connection → VictoriaMetrics. Or in connectors.json:

{ "name": "vm", "kind": "victoriametrics",
  "settings": { "url": "http://vmselect.monitoring.svc:8481/select/0/prometheus",
                "vmagent_urls": ["http://vmagent.monitoring.svc:8429"],
                "vmalert_url": "http://vmalert.monitoring.svc:8880",
                "component_urls": ["http://vminsert.monitoring.svc:8480",
                                   "http://vmstorage-0.vmstorage.monitoring.svc:8482",
                                   "http://vmstorage-1.vmstorage.monitoring.svc:8482"],
                "alertmanager_url": "http://alertmanager.monitoring.svc:9093",
                "username": "3am", "password": { "file": "/etc/3am/secrets/vm-password" } } }

4. VerifyCopy Anchor Link

Test connection runs a query and reads every component's /metrics. Each component is named with its version:

Connected. vmselect v1.153.0-cluster + vmagent v1.153.0 + vmalert v1.153.0 + vminsert v1.153.0-cluster
+ vmstorage v1.153.0-cluster + vmstorage v1.153.0-cluster + Alertmanager 0.34.1

For the whole path, load a rule that always fires into vmalert and reload it:

groups:
  - name: 3am-test
    rules:
      - alert: ThreeAMPipelineTest
        expr: vector(1)
        labels: { severity: critical, service: monitoring }

curl -X POST http://vmalert:8880/-/reload

Within a minute the console shows an incident for ThreeAMPipelineTest. Remove the rule afterwards.

What it proves and fixesCopy Anchor Link

3AM raises these itself, even with no rule for them. If you run VictoriaMetrics' own alerting rules (RowsRejectedOnIngestion, TooManyRemoteWriteErrors, ConfigurationReloadFailure, …), 3AM uses yours and proves the cause behind them.

Fixed with approval (through the Kubernetes connector):

Handed to a person with the evidence (the right fix is a judgement call):

A few things worth knowing about how it decides:

• The fix targets the proven culprit. If vmagent's alert opens the incident but a storage node is read-only, 3AM grows that node's volume, not vmagent's.
• Lost vs delayed. vmagent records each storage answer's exact status code, so 3AM can tell whether data is dropped or only queued.
• No guessing. "Storage fails writes" is only concluded when storage is visible, writable and complete. Otherwise the read-only disk or the lost node is the cause.
• Only what's happening now. Counters are compared with 3AM's earlier readings, so last week's dropped rows aren't today's incident.
• Partial results are never evidence. With a storage node down, vmselect still answers, flagged partial; 3AM refuses those answers.

ActionsCopy Anchor Link

TroubleshootingCopy Anchor Link

VictoriaMetrics: authentication failedCopy Anchor Link

Check the user and password, or the bearer token.

Alertmanager: authentication failedCopy Anchor Link

Alertmanager has its own credentials. Fill in the alertmanager_* settings.

the certificate is not trusted: set ca_fileCopy Anchor Link

Set CA certificate to your internal CA, in PEM format.

answers, but partially: a storage node is unreachableCopy Anchor Link

A vmstorage node is down. 3AM refuses partial results as evidence until it's back.

vmagent (…): HTTP 404Copy Anchor Link

That URL isn't a vmagent, or a proxy hides /metrics. Give 3AM the vmagent's own address.

CertificationCopy Anchor Link

Live, on VictoriaMetrics v1.153.0 and the long-term-support v1.136.0, 97 checks on each (2026-10-04).

The estate:

• single-node over HTTPS with an internal CA and basic auth;
• a cluster (vminsert, vmselect, two vmstorage);
• 7 vmagents and 2 vmalerts;
• Alertmanager behind a bearer-token proxy, and the real mysqld_exporter.

One thing was broken per component. Targets:

• refusing connections, rejecting credentials or answering garbage;
• carrying too many labels, or churning series.

vmagents writing:

• to a dead port, or with wrong credentials;
• through a proxy that rejects the data;
• into read-only storage, or past a series limit.

Alerting and storage:

• a dead datasource, a broken rule, a dead notifier and a broken reload;
• a storage node and a vmagent stopped mid-run.

Every cause was proven from live evidence. Read-only storage was never taken for failing storage, and partial answers were refused. On Kubernetes, hung vmagent, vmstorage and exporter pods were restarted and verified (details).