# Models: GPU or endpoint

> Where 3AM's two models run. A GPU on the host, or endpoints you point it at (OpenRouter, vLLM, Ollama, your LLM gateway). Never the CPU. Every option, with copy-paste installs.

Source: https://docs.3am.si/install/models · Markdown: https://docs.3am.si/install/models.md · All docs: https://docs.3am.si/llms.txt
3AM is an on-prem AI on-call engineer for regulated enterprises (https://3am.si).

3AM uses two models, and neither ever acts ([what each does](https://docs.3am.si/how-3am-decides/models)):

|       | System 1                                                                             | The investigator (System 2)                                                    |
| ----- | ------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------ |
| Model | `togethercomputer/tev1-4b-experimental` (4B)                                         | Llama 3.3 70B by default, or Qwen2.5 72B; `qwen2.5:7b-instruct` on a local GPU |
| API   | A decisions endpoint (OpenRouter's `/api/alpha/decisions`, Ollama's `/v1/systemone`) | OpenAI-compatible chat completions, ≥ 32k context, JSON output                 |
| Job   | Verifies causes, fixes and every sentence of the note                                | Investigates through read-only actions; writes the note                        |

> **Warning: Never on the CPU**
> The models run on a **GPU** or behind an **endpoint**. A host without a GPU must be given both endpoints. There is
> no CPU model profile: an earlier install's `--profile cpu` is moved to `auto` with a warning.

## Why not the CPU

* **The investigator's prompts are large:** the alerts, the evidence bundle, every method's findings, the read tools
  and each step's history.
* **CPU inference makes everything wait.** A 7B model answers in minutes per step on a CPU, with a context too short
  for those prompts. A 1.5B model is fast enough but can't hold the evidence.
* **Incidents share the CPU.** Model calls are on the critical path of every investigation; on a CPU they take the
  cores every other incident's evidence gathering needs.

## Choose

```text
Does the 3AM host have a GPU with 8 GB+ VRAM?
├─ yes ─▶ ./install.sh                                  (the gpu profile: models run on it)
└─ no ──▶ do you have a GPU server, an LLM gateway, or may you use a hosted API?
          ├─ yes ─▶ ./install.sh --model-url … --model … --s1-url …    (endpoints)
          └─ no ──▶ 3AM won't install. For a lab only: --profile none --allow-no-models
```

## Endpoints (any host)

  **OpenRouter:**

```bash
./install.sh --model-url https://openrouter.ai/api/v1 --model meta-llama/llama-3.3-70b-instruct \
  --model-key-file ~/openrouter.key \
  --model-extra '{"provider":{"data_collection":"deny"}}'
```

System 1 uses OpenRouter's decisions endpoint (`https://openrouter.ai/api/alpha/decisions`) with the same key: no
`--s1-url` needed. `data_collection: deny` routes only to providers that neither store nor train on the data. Use
hosted APIs only for data you are allowed to send; secrets and card numbers are masked first.

  **Your vLLM:**

```bash
./install.sh --model-url https://vllm.bank.internal/v1 --model meta-llama/Llama-3.3-70B-Instruct \
  --model-key-file ~/vllm.key \
  --s1-url https://s1.bank.internal/v1/systemone --s1-key-file ~/s1.key
```

On vLLM the model name is the Hugging Face id. Run it with prefix caching (`--enable-prefix-caching`) and, on
H100s, FP8 weights and KV cache ([Sizing](https://docs.3am.si/install/sizing#with-gpus)).

  **Ollama on a GPU server:**

```bash
./install.sh --model-url http://gpu01.bank.internal:11434/v1 --model qwen2.5:7b-instruct \
  --s1-url http://gpu01.bank.internal:11434/v1/systemone
```

  **An LLM gateway:**

```bash
./install.sh --model-url https://llm-gw.bank.internal/v1 --model <the name your gateway lists at /models> \
  --model-key-file ~/gw.key --s1-url <your System 1 decisions endpoint> --s1-key-file ~/gw.key
```

**Refused:** an endpoint on the 3AM host itself (`localhost`, `127.0.0.1`, `3am-model`) when the host has no GPU, and
a missing System 1 endpoint (unless the investigator is on OpenRouter). The installer says why and stops.

## A GPU on the host

| GPU                | Needs                                                                   | Runs                                                                                                   |
| ------------------ | ----------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ |
| NVIDIA, 8 GB+ VRAM | NVIDIA Container Toolkit (Docker) or CDI (`nvidia.com/gpu=all`, Podman) | The investigator (`qwen2.5:7b-instruct`) in `3am-model`, on an internal network with no published port |
| AMD, 8 GB+ VRAM    | ROCm (`/dev/kfd`, `/dev/dri`)                                           | The ROCm build of the same runtime                                                                     |
| 24 GB+ (L4, A10G)  | As above                                                                | The investigator and System 1 side by side, with room for 8–16 concurrent calls                        |

```bash
./install.sh                    # auto: a GPU is found, the gpu profile is used
./install.sh --profile gpu      # the same, and fail if no GPU is found
```

An air-gapped GPU install needs a bundle with the model runtime and weights inside; otherwise use endpoints.

## Kubernetes

| Value                                                 | Meaning                                                                      |
| ----------------------------------------------------- | ---------------------------------------------------------------------------- |
| `externalModel.url`, `.model`, `.keySecret`, `.extra` | The investigator's endpoint (no model runs in the cluster)                   |
| `systemOne.url`, `.model`, `.keySecret`               | System 1's endpoint. Empty: OpenRouter's, when `externalModel` is OpenRouter |
| `model.enabled`, `model.profile: gpu`                 | The model runtime on a GPU node (requests `nvidia.com/gpu: 1`)               |
| `vllm.*`, `systemTwo.profile`                         | System 2 on the cluster's GPUs with vLLM (`llama-3.3-70b` or `qwen2.5-72b`)  |
| `allowNoModels`                                       | Labs only                                                                    |

The chart **fails to render** a model pod that doesn't request a GPU, `externalModel.url` without `systemOne.url`, or
no models at all without `allowNoModels`. [Kubernetes](https://docs.3am.si/install/kubernetes#models).

## Settings

| Variable                                                    | Installer                                 | Meaning                                                                                                    |
| ----------------------------------------------------------- | ----------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| `THREEAM_MODEL_URL`, `THREEAM_MODEL`                        | `--model-url`, `--model`                  | The investigator's endpoint and model                                                                      |
| `THREEAM_MODEL_KEY_FILE` / `THREEAM_MODEL_KEY`              | `--model-key-file`                        | Its key (prefer the file)                                                                                  |
| `THREEAM_MODEL_EXTRA`                                       | `--model-extra`                           | JSON merged into each request                                                                              |
| `THREEAM_S1_URL`, `THREEAM_S1_MODEL`, `THREEAM_S1_KEY_FILE` | `--s1-url`, `--s1-model`, `--s1-key-file` | System 1                                                                                                   |
| `THREEAM_SYSTEM2`                                           |                                           | `llama-3.3-70b` (default) or `qwen2.5-72b`; `THREEAM_MODEL` wins                                           |
| `THREEAM_MODEL_DAILY_TOKENS`, `THREEAM_S1_DAILY_TOKENS`     |                                           | A daily token ceiling (0: none). Spent, model steps are skipped until 00:00 UTC (`model.budget_exhausted`) |
| `THREEAM_MODEL_CONCURRENCY`, `THREEAM_S1_CONCURRENCY`       |                                           | Calls at once: 8 and 16. Raise to what the endpoint is sized for                                           |
| `THREEAM_MODEL_TIMEOUT_S`, `THREEAM_S1_TIMEOUT_S`           |                                           | Request timeouts (System 1: 20 s)                                                                          |
| `THREEAM_MODEL_CASSETTE`, `THREEAM_MODEL_REPLAY`            |                                           | Record and replay model calls (`record`, `replay`, `auto`): tests and labs re-run for free                 |

All of them: [Environment variables](https://docs.3am.si/reference/environment#models).

## Check it

```bash
./install.sh --doctor | grep -i model     # "model placement" and both models answering
```

Preflight and 3AM Doctor **fail** "model placement" on a host without a GPU whose models are missing or local.