Sizing
How big the 3AM host should be (8 vCPU, 32 GB is the recommended minimum), what was measured, how work runs in parallel, and GPU sizing for large estates.
The short answer
8 vCPU, 32 GB RAM, 100 GB SSD, no GPU, models behind endpoints. That is a GCP e2-standard-8, what we run 3AM on
every day, and the recommended minimum.
| On it | Comfortably |
|---|---|
| Connectors | ~1,000, any number of them alert sources polled every 15 s |
| Investigations at once | 16 by default, 64 safe |
| Incidents | Hundreds per hour (bounded by the model endpoint and your systems, not the host) |
| CPU used | Under 1 core at 64 concurrent investigations (measured: 0.54) |
| Memory used | ~4–5 GB at 1,000 connectors and 64 investigations |
| Estate | Connectors | Investigations at once | Host | Disk |
|---|---|---|---|---|
| Recommended minimum | up to 1,000 | 16–64 | 8 vCPU, 32 GB | 100 GB SSD |
| Very large | over 1,000 | 64+ | 16 vCPU, 64 GB | 500 GB SSD |
The host never runs a model on its CPU: the investigator and System 1 are on a GPU or behind endpoints (Models), and the endpoint sets the ceiling on how many hard incidents are worked per minute.
What was measured
build/bench_capacity.py drives the real orchestrator, evidence gathering, RCA engine, proof rules and audit log, with
connectors and models that wait as long as real ones would (a Kubernetes or SQL read 0.15 s, a metrics query 0.05 s, a
model step 2 s, a System 1 call 0.3 s). Every incident took the expensive path. On 4 vCPU:
| Concurrency | 64 incidents, wall time | CPU per incident | Cores busy | Incidents/min |
|---|---|---|---|---|
| 8 | 34.9 s | 0.073 s | 0.13 | 110 |
| 16 | 18.9 s | 0.070 s | 0.24 | 203 |
| 32 | 11.5 s | 0.073 s | 0.41 | 334 |
| 64 | 8.5 s | 0.072 s | 0.54 | 451 |
CPU per incident is flat; throughput rises almost linearly with concurrency, because an investigation spends about 98% of its time waiting. The sizing above applies a 10× safety factor to CPU and generous memory allowances.
What limits throughput
- The model endpoint. A hard incident takes about 4 model steps. At ~4 s per step and 8 calls at once
(
THREEAM_MODEL_CONCURRENCY), that is ≈ 30 hard incidents a minute. Raise the concurrency to what the endpoint is sized for. Incidents a pack proves don't call the model at all. - Your systems. Each connector has at most 4 calls in flight however many incidents run
(
THREEAM_CONNECTOR_CONCURRENCY, ormax_in_flighton the connector): a burst never floods a database or an API. - The re-check window. An incident nothing proves at once is watched for 180 s while its symptoms build up,
holding its slot. Raise
THREEAM_MAX_CONCURRENTto 32–64 if bursts are common.
How work runs in parallel
| What | At once | Setting |
|---|---|---|
| Investigations | 2 per vCPU, between 4 and 32 (16 on 8 vCPU); the rest queue by priority | THREEAM_MAX_CONCURRENT |
| Alert-source polls | 16 (one slow source never delays the others: 20 s cap, then connector.down) | THREEAM_POLL_WORKERS |
| Connector calls | 4 per connector | THREEAM_CONNECTOR_CONCURRENCY, max_in_flight |
| Model calls | 8 | THREEAM_MODEL_CONCURRENCY |
| System 1 calls | 16 | THREEAM_S1_CONCURRENCY |
Which investigation runs first when every slot is busy: cause before symptom (an incident's root first, then its
own alert coming back, then the recovery's tail and neighbours; upstream before downstream), then impact, severity, and
time waited so nothing starves. One slot is always kept for a new, independent incident. Every wait and start is in
the audit log (episode.queued, episode.started) with its priority.
With GPUs
For large estates, the models run on your own GPUs (vLLM or Red Hat AI Inference Server). Estimates from first
principles; check them by replaying recorded 3AM prompts (THREEAM_MODEL_CASSETTE) on your GPUs before buying.
| Component | Runs on | Needs |
|---|---|---|
3am-core | CPU nodes, one replica (the audit log has one writer) | 8 vCPU / 32 GB (medium), 16 vCPU / 64 GB (large), fast SSD |
| System 2 (Llama 3.3 70B or Qwen2.5 72B) | GPU nodes, vLLM | 2× H100 80 GB per replica (FP8), or 1× H200; at least 2× A100 80 GB with 4-bit weights |
| System 1 (tev1, 4B) | Small GPUs | 1× L4 24 GB or L40S 48 GB per replica |
3am-causal | CPU, stateless, HPA | 2–4 vCPU per replica |
| Preference learning (DPO) | GPUs borrowed off-peak | 2× H100 80 GB with QLoRA for a 70B base |
| Medium | Large | |
|---|---|---|
| Estate | ~500 services, ~200 investigations/day | ~2,000 services, ~1,000 investigations/day |
| Peak | ~30 concurrent, ~12 reaching System 2 | ~100 concurrent, ~40 reaching System 2 |
| System 2 | 2 replicas = 4× H100 (or 2× H200) | 3 replicas = 6× H100 (or 3× H200) |
| System 1 | 2× L4 | 2× L40S |
3am-core | 8 vCPU / 32 GB, 250 GB SSD | 16 vCPU / 64 GB, 500 GB SSD |
| Audit log growth | ~0.3 GB/day | ~1.2 GB/day |
Why 2× H100 per replica: Llama 3.3 70B is ~71 GB in FP8. At 0.9 utilisation, 2× 80 GB leaves ~65 GB for the KV cache, about 425k tokens (160 KB per token with an FP8 KV cache): ~35 investigations of ~12k tokens in parallel.
Check your host
./install.sh --doctor # placement, endpoints, connectors
python3 build/bench_capacity.py --incidents 64 --concurrency 8,16,32,64 --connectors 200 --jsonModels: GPU or endpoint
Where 3AM's two models run. A GPU on the host, or endpoints you point it at (OpenRouter, vLLM, Ollama, your LLM gateway). Never the CPU. Every option, with copy-paste installs.
Kubernetes
Install 3AM with the Helm chart; models on GPU nodes, vLLM or endpoints; every chart value; OpenShift; upgrades.