PagerDuty
Pages become 3AM incidents, 3AM writes its diagnosis on the page, and people approve with a note.
What 3AM does with PagerDuty
| In short | |
|---|---|
| Pages in | Open incidents on the services you choose become 3AM incidents. The alert name and labels come from the integration that raised the page (Alertmanager, Prometheus, Datadog…) |
| Notes out | When 3AM is allowed to act on the service, it writes its diagnosis as a note on the page that woke someone up. In shadow mode, nothing in PagerDuty changes |
| Approvals | 3AM raises an approval incident on a service you choose; approvers answer with a note saying approve or reject |
Before you start
You need a PagerDuty user with the Admin or Account Owner role to create an API key, and the services you want 3AM to watch.
1. Create a REST API key
3AM.You don't need an App or OAuth
Add New App (OAuth, Scoped OAuth) is for apps published to the PagerDuty marketplace. 3AM uses one API key from inside your network; skip the app screens.
2. Choose the user 3AM acts as
Notes and approval incidents appear under a PagerDuty user. Create a service user such as [email protected]
(People → Users → Add Users), or use an existing one. You'll enter its e-mail below.
3. Choose the services to watch
Open each service whose pages 3AM should handle; the ID is in the address bar:
https://<you>.pagerduty.com/service-directory/PXXXXXX. Leave the list empty to watch every service.
4. Create the approval service (optional)
For approvals, create a service for approval incidents: Services → Service Directory → New Service:
- Name:
3AM approvals - Escalation policy: one that reaches the people who may approve 3AM's changes
- Integrations: none (only 3AM creates incidents here)
Note its ID. Without an approval service, PagerDuty is a source of pages only.
5. Connect it in 3AM
In the console, Settings → Add a connection → PagerDuty (or the setup wizard's Monitoring or Approvals step):
kind: pagerdutyalerts inapprovals| Setting | What to enter | Required | Default |
|---|---|---|---|
api_url | REST API base URL (EU: https://api.eu.pagerduty.com) | yes | "https://api.pagerduty.com" |
api_tokensecret | REST API key | yes | — |
from_email | E-mail of the PagerDuty user 3AM acts as | yes | — |
watch_services | Service IDs whose pages 3AM handles (empty: every service) | no | — |
service_id | Service for approval incidents (optional: approvals off without it) | no | — |
approvers | E-mails of users allowed to approve | no | — |
Select Test connection. A healthy result looks like:
Connected. authenticated; 3AM approvals (approvals); core-banking-dbNetwork: 3AM needs outbound HTTPS to api.pagerduty.com (EU accounts: api.eu.pagerduty.com). PagerDuty never
connects to 3AM.
6. Check it end to end
Trigger a test page on a watched service, for example from its Events API v2 integration (copy the integration key from the service's Integrations tab):
curl -s https://events.pagerduty.com/v2/enqueue -H 'Content-Type: application/json' -d '{
"routing_key": "<integration key>", "event_action": "trigger", "dedup_key": "3am-test-1",
"payload": {"summary": "MySQLReadOnly core-banking-db (3AM test)", "source": "db-1", "severity": "critical",
"custom_details": {"alertname": "MySQLReadOnly", "service": "core-banking-db"}}}'Within one poll interval (15 seconds by default) the console shows a new incident for MySQLReadOnly. Then resolve
the test page by sending the same request with "event_action": "resolve".
How it behaves
- Which pages: triggered and acknowledged incidents on the watched services. 3AM's own approval incidents are never read as pages.
- One page, one signal: each incident is fingerprinted by the incident itself, even with alert grouping (one
incident holding several alerts). The first alert's dedup key is kept as the
dedup_keylabel. - Severity: from the priority (P1/P2 critical, P3/P4 warning, P5 info), otherwise from urgency.
- Approvals:
- Each request opens an incident with a dedup key, so a retry can't create a duplicate.
- High-risk requests ask for high urgency; services with dynamic notifications decide urgency themselves.
- A note from a listed approver saying approve or reject is a decision. The note's author is resolved to their e-mail (PagerDuty notes carry names).
- Two approvers on the same incident make a quorum of two.
- When the request closes, 3AM adds a note and resolves the approval incident.
- Limits: on rate limits (HTTP 429) 3AM waits as PagerDuty asks; on 5xx it backs off and retries. It never retries authentication or validation errors.
Troubleshooting
| Test connection says | Fix |
|---|---|
| the API key is wrong or revoked | Create a new key (step 1) and paste it again |
| the key lacks permission | The key is read-only; create one with Read-only unticked |
| no PagerDuty user with e-mail … | The user 3AM acts as doesn't exist or the e-mail differs; check People → Users |
| service not found | A service ID is wrong; copy it from the service's address bar |
| Pages don't appear | The service isn't in Services to watch, or the page was raised before 3AM started (chronic alerts are ignored until they fire again) |
| Approvals aren't counted | The note's author isn't in Approvers (compare e-mails), or the note doesn't contain approve / reject |
Certification
Certified on 2026-10-04 against a live PagerDuty account: 16 live checks and 15 contract checks. A real page raised through Events API v2 arrives with its alert name and labels; 3AM's note lands on the incident; a real approval by a second user is attributed to that user's e-mail and closed.