# Sizing > How big the 3AM host should be (8 vCPU, 32 GB is the recommended minimum), what was measured, how work runs in parallel, and GPU sizing for large estates. Source: https://docs.3am.si/install/sizing · Markdown: https://docs.3am.si/install/sizing.md · All docs: https://docs.3am.si/llms.txt 3AM is an on-prem AI on-call engineer for regulated enterprises (https://3am.si). ## The short answer **8 vCPU, 32 GB RAM, 100 GB SSD, no GPU, models behind endpoints.** That is a GCP `e2-standard-8`, what we run 3AM on every day, and the recommended minimum. | On it | Comfortably | | ---------------------- | -------------------------------------------------------------------------------- | | Connectors | \~1,000, any number of them alert sources polled every 15 s | | Investigations at once | 16 by default, 64 safe | | Incidents | Hundreds per hour (bounded by the model endpoint and your systems, not the host) | | CPU used | Under 1 core at 64 concurrent investigations (measured: 0.54) | | Memory used | \~4–5 GB at 1,000 connectors and 64 investigations | | Estate | Connectors | Investigations at once | Host | Disk | | ----------------------- | ----------- | ---------------------- | ----------------- | ---------- | | **Recommended minimum** | up to 1,000 | 16–64 | **8 vCPU, 32 GB** | 100 GB SSD | | Very large | over 1,000 | 64+ | 16 vCPU, 64 GB | 500 GB SSD | The host never runs a model on its CPU: the investigator and System 1 are on a GPU or behind endpoints ([Models](https://docs.3am.si/install/models)), and **the endpoint sets the ceiling** on how many hard incidents are worked per minute. ## What was measured `build/bench_capacity.py` drives the real orchestrator, evidence gathering, RCA engine, proof rules and audit log, with connectors and models that wait as long as real ones would (a Kubernetes or SQL read 0.15 s, a metrics query 0.05 s, a model step 2 s, a System 1 call 0.3 s). Every incident took the expensive path. On 4 vCPU: | Concurrency | 64 incidents, wall time | CPU per incident | Cores busy | Incidents/min | | ----------- | ----------------------- | ---------------- | ---------- | ------------- | | 8 | 34.9 s | 0.073 s | 0.13 | 110 | | 16 | 18.9 s | 0.070 s | 0.24 | 203 | | 32 | 11.5 s | 0.073 s | 0.41 | 334 | | 64 | 8.5 s | 0.072 s | 0.54 | 451 | CPU per incident is flat; throughput rises almost linearly with concurrency, because an investigation spends about 98% of its time waiting. The sizing above applies a 10× safety factor to CPU and generous memory allowances. ## What limits throughput 1. **The model endpoint.** A hard incident takes about 4 model steps. At \~4 s per step and 8 calls at once (`THREEAM_MODEL_CONCURRENCY`), that is ≈ 30 hard incidents a minute. Raise the concurrency to what the endpoint is sized for. Incidents a pack proves don't call the model at all. 2. **Your systems.** Each connector has at most 4 calls in flight however many incidents run (`THREEAM_CONNECTOR_CONCURRENCY`, or `max_in_flight` on the connector): a burst never floods a database or an API. 3. **The re-check window.** An incident nothing proves at once is watched for 180 s while its symptoms build up, holding its slot. Raise `THREEAM_MAX_CONCURRENT` to 32–64 if bursts are common. ## How work runs in parallel | What | At once | Setting | | ------------------ | ----------------------------------------------------------------------------- | ------------------------------------------------ | | Investigations | 2 per vCPU, between 4 and 32 (16 on 8 vCPU); the rest queue by priority | `THREEAM_MAX_CONCURRENT` | | Alert-source polls | 16 (one slow source never delays the others: 20 s cap, then `connector.down`) | `THREEAM_POLL_WORKERS` | | Connector calls | 4 per connector | `THREEAM_CONNECTOR_CONCURRENCY`, `max_in_flight` | | Model calls | 8 | `THREEAM_MODEL_CONCURRENCY` | | System 1 calls | 16 | `THREEAM_S1_CONCURRENCY` | **Which investigation runs first** when every slot is busy: cause before symptom (an incident's root first, then its own alert coming back, then the recovery's tail and neighbours; upstream before downstream), then impact, severity, and time waited so nothing starves. One slot is always kept for a new, independent incident. Every wait and start is in the audit log (`episode.queued`, `episode.started`) with its priority. ## With GPUs For large estates, the models run on your own GPUs (vLLM or Red Hat AI Inference Server). Estimates from first principles; check them by replaying recorded 3AM prompts (`THREEAM_MODEL_CASSETTE`) on your GPUs before buying. | Component | Runs on | Needs | | --------------------------------------- | ----------------------------------------------------- | ------------------------------------------------------------------------------------------ | | `3am-core` | CPU nodes, one replica (the audit log has one writer) | 8 vCPU / 32 GB (medium), 16 vCPU / 64 GB (large), fast SSD | | System 2 (Llama 3.3 70B or Qwen2.5 72B) | GPU nodes, vLLM | **2× H100 80 GB per replica** (FP8), or 1× H200; at least 2× A100 80 GB with 4-bit weights | | System 1 (tev1, 4B) | Small GPUs | 1× L4 24 GB or L40S 48 GB per replica | | `3am-causal` | CPU, stateless, HPA | 2–4 vCPU per replica | | Preference learning (DPO) | GPUs borrowed off-peak | 2× H100 80 GB with QLoRA for a 70B base | | | Medium | Large | | ---------------- | ---------------------------------------- | -------------------------------------------- | | Estate | \~500 services, \~200 investigations/day | \~2,000 services, \~1,000 investigations/day | | Peak | \~30 concurrent, \~12 reaching System 2 | \~100 concurrent, \~40 reaching System 2 | | System 2 | 2 replicas = **4× H100** (or 2× H200) | 3 replicas = **6× H100** (or 3× H200) | | System 1 | 2× L4 | 2× L40S | | `3am-core` | 8 vCPU / 32 GB, 250 GB SSD | 16 vCPU / 64 GB, 500 GB SSD | | Audit log growth | \~0.3 GB/day | \~1.2 GB/day | **Why 2× H100 per replica:** Llama 3.3 70B is \~71 GB in FP8. At 0.9 utilisation, 2× 80 GB leaves \~65 GB for the KV cache, about 425k tokens (160 KB per token with an FP8 KV cache): \~35 investigations of \~12k tokens in parallel. ## Check your host ```bash ./install.sh --doctor # placement, endpoints, connectors python3 build/bench_capacity.py --incidents 64 --concurrency 8,16,32,64 --connectors 200 --json ```