open-swe/deploy/seahaven/ROTATION.md

# Sea Haven — Open SWE secret & config rotation

How secrets and config reach the running app, and how to rotate either one.

## How values flow at boot

```
AWS Secrets Manager  open-swe-<env>/*  ─┐
AWS SSM Param Store   /open-swe-<env>/* ─┤── fetch-config.sh ──▶ tmpfs /run/open-swe/.env (root:root 0600)
                                         │   (systemd ExecStartPre=+, EC2 role)        │
                                         ▼                                             ▼
                                    FAIL-FAST if a                         app symlink <APP_DIR>/.env
                                    required var is empty                  python-dotenv reads at import
```

The app reads `.env` **once, at import**. There is no hot-reload of secrets.
Therefore the rotation contract is always the same two steps:

> **Rotation = (1) update the value in Secrets Manager / SSM, then (2) restart the
> service** so `fetch-config.sh` re-materializes the `.env`.

```bash
# after updating a secret/param in AWS:
sudo systemctl restart open-swe.service
#   ExecStartPre=+  -> fetch-config.sh re-pulls + rewrites the tmpfs .env (fail-fast)
#   ExecStartPost   -> seed_store.sh re-seeds the in-memory store (team_settings + user_mappings)
```

There is **no zero-downtime path for most secrets** on the stock in-memory
runtime — a restart is required and it also wipes the in-memory store (re-seeded
by `seed_store.sh` automatically). The one secret built for zero-downtime overlap
is `TOKEN_ENCRYPTION_KEY` (see below), but even it needs the restart to load the
new key list.

## Rotating a secret (Secrets Manager)

```bash
ENV=prod            # or dev
NAME=DASHBOARD_JWT_SECRET
aws secretsmanager put-secret-value \
  --secret-id "open-swe-${ENV}/${NAME}" \
  --secret-string 'NEW_VALUE' \
  --region us-east-1
sudo systemctl restart open-swe.service        # on the box
```

(`update-secret`/`put-secret-value` both create a new version; the boot hook
always reads `AWSCURRENT`.)

## Rotating a config param (SSM)

```bash
aws ssm put-parameter --overwrite \
  --name "/open-swe-${ENV}/DASHBOARD_BASE_URL" \
  --type String --value 'https://openswe.seahaven.com' \
  --region us-east-1
sudo systemctl restart open-swe.service
```

## Per-secret rotation notes

| Secret | Rotation notes |
|---|---|
| **TOKEN_ENCRYPTION_KEY** | Fernet key(s). Supports a **comma/newline-separated list** (`agent/encryption.py`) for zero-downtime key rotation: prepend the NEW key, keep the OLD key(s) in the list. New data is encrypted with the first key; old data still decrypts with the trailing keys. After all encrypted-at-rest tokens (per-user GitHub OAuth tokens in thread metadata) have been re-encrypted/expired, drop the old key. Store the list as one secret value; `fetch-config.sh` writes it verbatim. **Never** rotate to a single new key in one step or every existing encrypted token becomes undecryptable. |
| **GITHUB_APP_PRIVATE_KEY** | Multiline PEM. Generate a new private key in the GitHub App settings (you may have **two active keys** during overlap), put the new PEM into the secret, restart, verify install-token minting + a webhook delivery, then delete the old key in GitHub. `fetch-config.sh` writes the PEM as a double-quoted multiline value (python-dotenv-safe); paste the full `-----BEGIN…-----END-----` block including newlines. |
| **GITHUB_WEBHOOK_SECRET** | Webhook HMAC. GitHub allows only **one** webhook secret per App, so this is a brief-break rotation: update the secret in AWS **and** the GitHub App webhook config, restart. Deliveries signed with the old secret during the gap will 401 (GitHub auto-redelivers). Required in **prod** (fail-fast). |
| **SLACK_SIGNING_SECRET** | Slack request-signature secret. Rotate in the Slack app config and AWS together, restart. Required in **prod** (fail-fast). A stale value silently 401s `url_verification`/events until restart (known gotcha). |
| **LINEAR_WEBHOOK_SECRET** | Linear webhook signature. Required in prod only when the Linear integration is wired (`LINEAR_API_KEY` present). Rotate in Linear + AWS together, restart. |
| **SLACK_CLIENT_SECRET / GITHUB_APP_CLIENT_SECRET** | OAuth client secrets (dashboard login / Slack OAuth). Rotate in the provider console + AWS, restart. Existing dashboard sessions are JWT-signed by `DASHBOARD_JWT_SECRET`, not these, so they survive. |
| **DASHBOARD_JWT_SECRET** | Signs dashboard session cookies. Rotating **invalidates all active sessions** (users re-login). Hard-required (RuntimeError if empty). No overlap list — single value. |
| **Model provider keys** (`FIREWORKS_API_KEY`; legacy `ANTHROPIC_API_KEY` / `OPENAI_API_KEY` / `GOOGLE_API_KEY` / `GROQ_API_KEY`) | Standard API-key rotation: issue new key, update AWS, restart, revoke old. Bedrock (the seeded Claude builder + reviewer) authenticates via the **host IAM role — no API key**; the only fail-fast-required provider key is now `FIREWORKS_API_KEY` (fallback / subagents / any non-Claude model). Keep `REQUIRED_PROVIDER_KEYS` in sync if you change the seeded models. |
| **LANGSMITH_API_KEY_PROD** | LangSmith key powering the `langsmith` sandbox (the only provider with working in-sandbox git/gh auth). Required when `SANDBOX_TYPE=langsmith` (fail-fast). Rotate in LangSmith + AWS, restart; existing sandboxes keep their already-injected proxy token until recycled. |
| **SLACK_BOT_TOKEN / LINEAR_API_KEY / GITHUB_PAT / EXA_API_KEY / DAYTONA_API_KEY / RUNLOOP_API_KEY / CORRIDOR_* / USER_ID_API_KEY_MAP / X_SERVICE_AUTH_JWT_SECRET** | Plain API-token rotation: update AWS, restart, revoke old at the provider. None support an overlap list. |

## Fail-fast safety

`fetch-config.sh` refuses to write the `.env` (exit 1) if any required var is
empty after a rotation — so a botched rotation (e.g. an empty `put-secret-value`)
stops the service at `ExecStartPre` instead of starting it with a partial `.env`.
The missing variable **names** are printed to the journal (values never are):

```bash
sudo journalctl -u open-swe.service -b | grep fetch-config
```

Required set enforced: `DASHBOARD_JWT_SECRET`, `TOKEN_ENCRYPTION_KEY`,
`GITHUB_APP_ID`, `GITHUB_APP_PRIVATE_KEY`, `GITHUB_APP_INSTALLATION_ID`,
`GITHUB_APP_CLIENT_ID`, `GITHUB_APP_CLIENT_SECRET`, the active provider keys
(`REQUIRED_PROVIDER_KEYS`, default `ANTHROPIC_API_KEY,OPENAI_API_KEY`), the
sandbox key for `SANDBOX_TYPE` (`langsmith` ⇒ `LANGSMITH_API_KEY_PROD` +
`DEFAULT_SANDBOX_SNAPSHOT_ID`), and — in **prod** — `GITHUB_WEBHOOK_SECRET`,
`SLACK_SIGNING_SECRET` (plus `LINEAR_WEBHOOK_SECRET` when Linear is wired).