# Add and remove installs

> Add an install (DR, staging, a new region) to your licence yourself, with six digits on two screens; take one off; what Warden checks; the hash-chained install log; air-gapped.

Source: https://docs.3am.si/licensing/installs · Markdown: https://docs.3am.si/licensing/installs.md · All docs: https://docs.3am.si/llms.txt
3AM is an on-prem AI on-call engineer for regulated enterprises (https://3am.si).

Starter and Enterprise licences name their installs (Starter: production + DR; Enterprise: up to 10). You add and
remove them yourself, never by mail: an install already on the licence **vouches** for the new one, and Warden signs
the new licence by itself, within your tier's limits. Strategic is a site licence: every install is licensed already.

Nothing secret goes to Warden: only fingerprints, public keys and licences.

## Add an install

    **On the new install.** Upload your organisation's licence (Settings → Licence). It doesn't name this install, so 3AM
    keeps it aside and offers **Ask to join** (Keychain → Installs). Pick the purpose (production, DR, staging, dev, test).
    3AM makes a request signed with this install's own key and shows it twice: a code to copy (`3AMREQ1.…`), and **six
    digits**.

    **On an install already on the licence.** Keychain → Installs → **Add an install**, paste the code. 3AM checks it and
    shows what it asks: the fingerprint, the purpose, when it expires. Type the six digits shown on the new install. Only
    if they match does this install vouch for it.

    **Warden** signs a new licence naming both installs. The vouching install takes it at once; the new install picks it
    up by itself (it asks every few seconds); the others get it at their next check with Warden, or through the Keychain.

**With a Keychain** it's simpler: a member asks over the Keychain and the home shows **Add it to the licence**, with
no code to type (pairing already proved who it is). A [site key](https://docs.3am.si/licensing/keychain#site-keys) can even add the site
to the licence from its installer, with the home vouching by itself.

```bash title="The same with the API"
# on the new install
curl -s -X POST -H "Authorization: Bearer $T" -d '{"purpose":"dr","label":"Mumbai DR"}' $NEW/v1/licence/installs/ask
# → {"code": "482 913", "text": "3AMREQ1.…"}
# on an install on the licence
curl -s -X POST -H "Authorization: Bearer $T" -d '{"text":"3AMREQ1.…"}' $PROD/v1/licence/installs/look
curl -s -X POST -H "Authorization: Bearer $T" -d '{"text":"3AMREQ1.…","code":"482913"}' $PROD/v1/licence/installs/vouch
# → granted (the new licence is installed here too), pending (held, with why), or manual (Warden unreachable)
```

Owners and admins add and remove installs; everyone signed in can see them.

## Take an install off

Keychain → Installs → **Take off**, from any install on the licence (itself included). Its older licence still names
it, but Warden answers *removed* at its next check, and it stops. The licence's last install stays.

## Air-gapped

If the vouching install can't reach Warden, it hands you the signed request (`manual`). Paste it at
`https://warden.3am.si/installs` from any machine, and upload the licence you get on both installs.

## What Warden checks before it signs, cheapest first

| Check                                                                                                       | Holds against                                                                                                                              |
| ----------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| Shape, size (16 KB), time window (15 minutes, 2 minutes of clock skew)                                      | Junk, stale requests, a clock set ahead                                                                                                    |
| Proof of work: SHA-256 with 20 leading zero bits (about a second to make, one hash to check)                | Floods                                                                                                                                     |
| Throttle per caller address (20 an hour)                                                                    | Brute force from one place                                                                                                                 |
| Replay: a request already decided gets the same answer                                                      | A recorded request sent twice never makes a second licence                                                                                 |
| The new install's signature over the request                                                                | Enrolling a key nobody holds                                                                                                               |
| The contract: live, not revoked, not a trial, the request's licence in it, the same organisation            | Requests against a revoked or someone else's licence. All get one answer ("can't be verified"), so nothing leaks about other organisations |
| The sponsor: on the current licence, signing with the key pinned at its first check-in                      | A stolen copy of the licence file: without an install's private key, nobody can vouch                                                      |
| Throttle per contract (30 an hour), once the sponsor is proven                                              | Spending another organisation's budget                                                                                                     |
| The sponsor's key pinned for at least a day; its last check-in within two weeks; not flagged as a duplicate | A freshly enrolled or cloned install vouching for more                                                                                     |
| The new fingerprint isn't pinned to another key                                                             | A fingerprint being taken over                                                                                                             |
| The tier's install limit; at most 3 added by policy a day and 10 in 30 days                                 | Runaway additions, a compromised sponsor                                                                                                   |

The first checks failing refuse the request. An untrusted-yet sponsor, a fingerprint pinned elsewhere, or a limit
reached **hold the request for people**: a pending licence in Warden's approval queue, approved by two people at 3AM,
finishing the same way. A request is never refused just for being unusual, and never granted for it either.

## Six digits on two screens

The six digits are the first 20 bits of a hash of the request itself. Swap the request on its way between the two
consoles and the digits change, and a person sees it before anything is signed. To fool them, an attacker would need a
second request with the same 20 bits, each attempt costing a signature with the new install's key and a fresh proof of
work, while the person types the digits from the real screen.

## The install log

Every addition and removal is a link in a hash chain per contract: the fingerprint and purpose, which install vouched,
the request's hash, and the hash of the link before. Every licence Warden signs carries the chain's head and its latest
twelve links.

* 3AM checks each new licence against the head it last saw. A licence whose history doesn't carry on from it is
  refused: two different histories were signed.
* Keychain members compare heads with their home at every sync, so two installs shown different histories both raise
  it (`security.install_log_fork`).
* Keychain → Installs shows the history: who was added, when, and which install vouched for each.

## Audit events

In 3AM: `licence.install_vouched`, `licence.install_removal`, `config.licence_candidate`,
`security.install_code_mismatch`, `security.install_log_fork`. In Warden: `install.added`, `install.held`,
`install.removed`, `install.refused`, `install.key_mismatch`.
