From 434d0ad80c3aea3af640799c6af5ea7578011464 Mon Sep 17 00:00:00 2001 From: Adam Moussa <166072409+amoussa1229@users.noreply.github.com> Date: Fri, 26 Jun 2026 15:06:41 -0400 Subject: [PATCH] feat(deploy): AWS-sourced fetch-config + seed_store + rotation docs (PR#7) (#8) PR#7 of the AWS migration. deploy/seahaven/: fetch-config.sh materializes a service-user-owned 0600 tmpfs .env from Secrets Manager + SSM (fail-fast); seed_store.sh reseeds the in-memory store; ROTATION.md. Incorporates T5 /sh-security-review fixes: - seed_store no longer bash-sources the .env (closes the SH-INJ-001 RCE); uses a non-eval reader, jq --arg JSON bodies, and a loopback-pinned BASE. - fetch-config: .env owned by the openswe service user (app no longer runs as root); DEFAULT_REPO_OWNER hard-pinned; key-identifier validation + flat-namespace collision detection; dropped SSM --recursive. shellcheck + bash -n clean. --- deploy/seahaven/DEPLOYMENT.md | 19 +++ deploy/seahaven/ROTATION.md | 92 +++++++++++ deploy/seahaven/fetch-config.sh | 282 ++++++++++++++++++++++++++++++++ deploy/seahaven/seed_store.sh | 209 +++++++++++++++++------ 4 files changed, 550 insertions(+), 52 deletions(-) create mode 100644 deploy/seahaven/ROTATION.md create mode 100755 deploy/seahaven/fetch-config.sh diff --git a/deploy/seahaven/DEPLOYMENT.md b/deploy/seahaven/DEPLOYMENT.md index 652160ed..16cf8e2e 100644 --- a/deploy/seahaven/DEPLOYMENT.md +++ b/deploy/seahaven/DEPLOYMENT.md @@ -89,6 +89,25 @@ Start a task by mentioning **`@openswe`** in a GitHub issue comment (the documen intake — the `open-swe` *label* path needs the `Issues` event subscription). The commenter must have a `user_mappings` entry or the run is skipped. +## AWS variant — env-sourced config (.env from Secrets Manager + SSM) + +On the AWS lift-and-shift, the `.env` is **not** staged by hand. A boot hook pulls +config from AWS via the EC2 instance role and materializes a root-only `.env` on a +tmpfs before the service starts. Same stock runtime; only how `.env` is produced +changes. + +| File | Role | +|---|---| +| `fetch-config.sh ` | `ExecStartPre=+` hook. Reads all SSM params `/open-swe-/*` + all secrets `open-swe-/*`, FAIL-FAST on any missing required var, writes `/run/open-swe/.env` (tmpfs, root:root 0600), symlinks `/.env` → it. Forces `DEFAULT_REPO_OWNER`=`Sea-Haven-Industries` (never upstream `langchain-ai`). | +| `seed_store.sh ` | `ExecStartPost` hook (unchanged role). Now sources the materialized `.env`, so `default_repo`/models/user-mappings come from AWS config instead of hardcoded `OPENSWE_*`. Still re-seeds the in-memory store on **every** restart. | +| `ROTATION.md` | How to rotate any secret/param (update AWS → `systemctl restart`) + per-secret notes (`TOKEN_ENCRYPTION_KEY` overlap list, GitHub App PEM, webhook signing secrets). | + +Because the `.env` is root-only, the AWS `open-swe.service` runs **as root** (the +on-prem unit's `User=adam` cannot read it). See the header of `fetch-config.sh` for +the exact unit snippet and the tmpfs / `RUN_DEDICATED_TMPFS` options. Naming/secret +placement follows the T9 inventory: 29 secrets → Secrets Manager `open-swe-/*`, +53 config → SSM `/open-swe-/*`, each keyed by the literal env-var name. + ## Aegra (deferred) `aegra/aegra.json` + `aegra/aegra_entry.py` are the self-hosted-runtime alternative diff --git a/deploy/seahaven/ROTATION.md b/deploy/seahaven/ROTATION.md new file mode 100644 index 00000000..f895fcbd --- /dev/null +++ b/deploy/seahaven/ROTATION.md @@ -0,0 +1,92 @@ +# Sea Haven — Open SWE secret & config rotation + +How secrets and config reach the running app, and how to rotate either one. + +## How values flow at boot + +``` +AWS Secrets Manager open-swe-/* ─┐ +AWS SSM Param Store /open-swe-/* ─┤── fetch-config.sh ──▶ tmpfs /run/open-swe/.env (root:root 0600) + │ (systemd ExecStartPre=+, EC2 role) │ + ▼ ▼ + FAIL-FAST if a app symlink /.env + required var is empty python-dotenv reads at import +``` + +The app reads `.env` **once, at import**. There is no hot-reload of secrets. +Therefore the rotation contract is always the same two steps: + +> **Rotation = (1) update the value in Secrets Manager / SSM, then (2) restart the +> service** so `fetch-config.sh` re-materializes the `.env`. + +```bash +# after updating a secret/param in AWS: +sudo systemctl restart open-swe.service +# ExecStartPre=+ -> fetch-config.sh re-pulls + rewrites the tmpfs .env (fail-fast) +# ExecStartPost -> seed_store.sh re-seeds the in-memory store (team_settings + user_mappings) +``` + +There is **no zero-downtime path for most secrets** on the stock in-memory +runtime — a restart is required and it also wipes the in-memory store (re-seeded +by `seed_store.sh` automatically). The one secret built for zero-downtime overlap +is `TOKEN_ENCRYPTION_KEY` (see below), but even it needs the restart to load the +new key list. + +## Rotating a secret (Secrets Manager) + +```bash +ENV=prod # or dev +NAME=DASHBOARD_JWT_SECRET +aws secretsmanager put-secret-value \ + --secret-id "open-swe-${ENV}/${NAME}" \ + --secret-string 'NEW_VALUE' \ + --region us-east-1 +sudo systemctl restart open-swe.service # on the box +``` + +(`update-secret`/`put-secret-value` both create a new version; the boot hook +always reads `AWSCURRENT`.) + +## Rotating a config param (SSM) + +```bash +aws ssm put-parameter --overwrite \ + --name "/open-swe-${ENV}/DASHBOARD_BASE_URL" \ + --type String --value 'https://openswe.seahaven.com' \ + --region us-east-1 +sudo systemctl restart open-swe.service +``` + +## Per-secret rotation notes + +| Secret | Rotation notes | +|---|---| +| **TOKEN_ENCRYPTION_KEY** | Fernet key(s). Supports a **comma/newline-separated list** (`agent/encryption.py`) for zero-downtime key rotation: prepend the NEW key, keep the OLD key(s) in the list. New data is encrypted with the first key; old data still decrypts with the trailing keys. After all encrypted-at-rest tokens (per-user GitHub OAuth tokens in thread metadata) have been re-encrypted/expired, drop the old key. Store the list as one secret value; `fetch-config.sh` writes it verbatim. **Never** rotate to a single new key in one step or every existing encrypted token becomes undecryptable. | +| **GITHUB_APP_PRIVATE_KEY** | Multiline PEM. Generate a new private key in the GitHub App settings (you may have **two active keys** during overlap), put the new PEM into the secret, restart, verify install-token minting + a webhook delivery, then delete the old key in GitHub. `fetch-config.sh` writes the PEM as a double-quoted multiline value (python-dotenv-safe); paste the full `-----BEGIN…-----END-----` block including newlines. | +| **GITHUB_WEBHOOK_SECRET** | Webhook HMAC. GitHub allows only **one** webhook secret per App, so this is a brief-break rotation: update the secret in AWS **and** the GitHub App webhook config, restart. Deliveries signed with the old secret during the gap will 401 (GitHub auto-redelivers). Required in **prod** (fail-fast). | +| **SLACK_SIGNING_SECRET** | Slack request-signature secret. Rotate in the Slack app config and AWS together, restart. Required in **prod** (fail-fast). A stale value silently 401s `url_verification`/events until restart (known gotcha). | +| **LINEAR_WEBHOOK_SECRET** | Linear webhook signature. Required in prod only when the Linear integration is wired (`LINEAR_API_KEY` present). Rotate in Linear + AWS together, restart. | +| **SLACK_CLIENT_SECRET / GITHUB_APP_CLIENT_SECRET** | OAuth client secrets (dashboard login / Slack OAuth). Rotate in the provider console + AWS, restart. Existing dashboard sessions are JWT-signed by `DASHBOARD_JWT_SECRET`, not these, so they survive. | +| **DASHBOARD_JWT_SECRET** | Signs dashboard session cookies. Rotating **invalidates all active sessions** (users re-login). Hard-required (RuntimeError if empty). No overlap list — single value. | +| **Model provider keys** (`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `GOOGLE_API_KEY`, `GROQ_API_KEY`, `FIREWORKS_API_KEY`) | Standard API-key rotation: issue new key, update AWS, restart, revoke old. The **active** provider keys (whatever `team_settings/default` seeds — currently `ANTHROPIC_API_KEY` + `OPENAI_API_KEY`) are fail-fast-required; keep `REQUIRED_PROVIDER_KEYS` in sync if you change the seeded models. | +| **LANGSMITH_API_KEY_PROD** | LangSmith key powering the `langsmith` sandbox (the only provider with working in-sandbox git/gh auth). Required when `SANDBOX_TYPE=langsmith` (fail-fast). Rotate in LangSmith + AWS, restart; existing sandboxes keep their already-injected proxy token until recycled. | +| **SLACK_BOT_TOKEN / LINEAR_API_KEY / GITHUB_PAT / EXA_API_KEY / DAYTONA_API_KEY / RUNLOOP_API_KEY / CORRIDOR_* / USER_ID_API_KEY_MAP / X_SERVICE_AUTH_JWT_SECRET** | Plain API-token rotation: update AWS, restart, revoke old at the provider. None support an overlap list. | + +## Fail-fast safety + +`fetch-config.sh` refuses to write the `.env` (exit 1) if any required var is +empty after a rotation — so a botched rotation (e.g. an empty `put-secret-value`) +stops the service at `ExecStartPre` instead of starting it with a partial `.env`. +The missing variable **names** are printed to the journal (values never are): + +```bash +sudo journalctl -u open-swe.service -b | grep fetch-config +``` + +Required set enforced: `DASHBOARD_JWT_SECRET`, `TOKEN_ENCRYPTION_KEY`, +`GITHUB_APP_ID`, `GITHUB_APP_PRIVATE_KEY`, `GITHUB_APP_INSTALLATION_ID`, +`GITHUB_APP_CLIENT_ID`, `GITHUB_APP_CLIENT_SECRET`, the active provider keys +(`REQUIRED_PROVIDER_KEYS`, default `ANTHROPIC_API_KEY,OPENAI_API_KEY`), the +sandbox key for `SANDBOX_TYPE` (`langsmith` ⇒ `LANGSMITH_API_KEY_PROD` + +`DEFAULT_SANDBOX_SNAPSHOT_ID`), and — in **prod** — `GITHUB_WEBHOOK_SECRET`, +`SLACK_SIGNING_SECRET` (plus `LINEAR_WEBHOOK_SECRET` when Linear is wired). diff --git a/deploy/seahaven/fetch-config.sh b/deploy/seahaven/fetch-config.sh new file mode 100755 index 00000000..8cabcde0 --- /dev/null +++ b/deploy/seahaven/fetch-config.sh @@ -0,0 +1,282 @@ +#!/usr/bin/env bash +# fetch-config.sh — AWS-sourced boot hook that materializes the app's .env. +# +# The stock `langgraph dev` runtime + the Open SWE app read a plain `.env` from +# the app working directory (python-dotenv). On the AWS lift-and-shift we do NOT +# commit a .env; instead every non-sensitive value lives in SSM Parameter Store +# (`/open-swe-/*`) and every secret lives in AWS Secrets Manager +# (`open-swe-/*`). This hook is run by systemd BEFORE the service starts; it +# pulls both sources via the EC2 instance role (no static keys), assembles a +# single .env on a tmpfs, and writes it owned by the unprivileged service user +# `chmod 600` (T5 SC-01: the privileged pre-hook materializes the secret; the app +# itself then runs as that NON-root service user, not root). +# +# It is intentionally FAIL-FAST: if any required secret/param is missing or empty +# it prints the offending variable NAMES (never values) and exits 1, so the +# service never starts with a partial .env. +# +# --------------------------------------------------------------------------- +# Naming contract (source of truth: T9 env/secret/config inventory) +# SSM /open-swe-/ -> exported as ENV_VAR_NAME +# Secrets open-swe-/ -> exported as ENV_VAR_NAME +# i.e. the last path segment IS the literal environment-variable name. This is a +# deliberate (documented) deviation from the handbook's kebab-case value-name +# example (`my-stack/slack-signing`): a .env materializer needs a lossless, +# unambiguous round-trip from store key -> env var, and the env var name is the +# only key that guarantees that. The `open-swe-` stack prefix still follows +# kebab-case per naming-conventions.md. +# --------------------------------------------------------------------------- +# +# Wiring into systemd (AWS EC2 variant): +# The unit runs as the unprivileged service user (User=openswe). ONLY the +# ExecStartPre pre-hook runs as root (the `+` prefix) so it can pull from AWS, +# write the tmpfs .env, and chown it to the service user. The app (ExecStart) +# and the seeder (ExecStartPost) then run as openswe and read the openswe-owned +# 0600 .env — the agent never runs as root (T5 SC-01). Pass the env as the +# positional arg (T5 BOOT-01): +# +# [Service] +# User=openswe +# Group=openswe +# Environment=ENV_DIR=/run/open-swe SERVICE_USER=openswe +# # ExecStartPre runs as root (+) so it can chown the .env to the service user. +# ExecStartPre=+/opt/open-swe/deploy/seahaven/fetch-config.sh prod +# ExecStart=/opt/open-swe/.venv/bin/langgraph dev --host 127.0.0.1 --port 2024 \ +# --no-browser --no-reload +# ExecStartPost=/opt/open-swe/deploy/seahaven/seed_store.sh prod +# +# tmpfs: /run is already a tmpfs on systemd hosts, so ENV_DIR=/run/open-swe is +# tmpfs-backed by default (the .env never touches disk). Set RUN_DEDICATED_TMPFS=1 +# to mount a private tmpfs at ENV_DIR instead. The app's CWD `.env` is a symlink +# into ENV_DIR (created idempotently below), so python-dotenv finds it unchanged. +# +# Idempotent, re-runnable on every (re)start. No secret is ever echoed. + +set -euo pipefail +umask 077 + +# --- Inputs ------------------------------------------------------------------ +ENV="${1:-${OPENSWE_ENV:-}}" +case "$ENV" in + dev | prod) ;; + *) + echo "fetch-config: ENV must be 'dev' or 'prod' (got '${ENV:-}')" >&2 + echo "usage: fetch-config.sh (or set OPENSWE_ENV)" >&2 + exit 2 + ;; +esac + +REGION="${AWS_REGION:-${AWS_DEFAULT_REGION:-us-east-1}}" +SSM_PREFIX="/open-swe-${ENV}/" +SECRET_PREFIX="open-swe-${ENV}/" + +ENV_DIR="${ENV_DIR:-/run/open-swe}" # tmpfs-backed (/run) by default +ENV_FILE="${ENV_DIR}/.env" +APP_DIR="${APP_DIR:-/opt/open-swe}" # where the app + its CWD .env live +APP_ENV_LINK="${APP_DIR}/.env" # symlink -> ENV_FILE + +# The unprivileged service user that runs the app and OWNS the .env (T5 SC-01). +# fetch-config runs as root (ExecStartPre=+) only to chown the secret to it. +SERVICE_USER="${SERVICE_USER:-openswe}" +SERVICE_GROUP="${SERVICE_GROUP:-${SERVICE_USER}}" + +# Sea Haven override: upstream defaults DEFAULT_REPO_OWNER to "langchain-ai". +# For the Sea Haven deployment it MUST be the org. fetch-config forces this so a +# stale/blank SSM value can never point the agent at the upstream org. +SH_REPO_OWNER="${OPENSWE_REPO_OWNER:-Sea-Haven-Industries}" + +for bin in aws jq; do + command -v "$bin" >/dev/null 2>&1 || { echo "fetch-config: '$bin' not found on PATH" >&2; exit 3; } +done + +log() { echo "fetch-config[$ENV]: $*"; } # NAMES/counts only — never values +b64d() { base64 --decode; } # GNU coreutils on the EC2 host + +# Accept a store key into VARS iff it is a valid env-var identifier and not a +# duplicate. Rejects non-identifier names (T5 SH-INJ-002 / set -e DoS hardening) +# and flat-namespace collisions (T5 SSM-05). $3 = source label for logs. +accept_var() { + local key="$1" value="$2" src="$3" + if ! [[ "$key" =~ ^[A-Za-z_][A-Za-z0-9_]*$ ]]; then + log "WARNING: skipping ${src} key with non-identifier name (rejected)" + return 0 + fi + if [ -n "${VARS[$key]+set}" ]; then + echo "fetch-config[$ENV]: FAIL-FAST — duplicate key '${key}' from ${src} (flat-namespace collision)" >&2 + exit 1 + fi + VARS["$key"]="$value" +} + +# --- tmpfs ------------------------------------------------------------------ +mkdir -p "$ENV_DIR" +# Owned by the service user so the unprivileged app can traverse it (T5 SC-01). +chown "${SERVICE_USER}:${SERVICE_GROUP}" "$ENV_DIR" 2>/dev/null || true +chmod 700 "$ENV_DIR" +if [ "${RUN_DEDICATED_TMPFS:-0}" = "1" ] && ! mountpoint -q "$ENV_DIR"; then + mount -t tmpfs -o nosuid,nodev,noexec,mode=0700,size=4m tmpfs "$ENV_DIR" + log "mounted dedicated tmpfs at $ENV_DIR" +fi + +# --- Collect values into an associative array -------------------------------- +declare -A VARS=() + +# 1) SSM Parameter Store (non-sensitive config). NOT --recursive: the contract is +# a FLAT namespace /open-swe-/, so a non-recursive list returns exactly +# those keys and cannot collapse two nested paths onto one name (T5 SSM-05). aws +# CLI v2 auto-paginates NextToken. +log "reading SSM params under ${SSM_PREFIX} ..." +ssm_json="$( + aws ssm get-parameters-by-path \ + --path "$SSM_PREFIX" \ + --with-decryption \ + --region "$REGION" \ + --no-cli-pager \ + --output json +)" +# Records are base64-encoded (namevalue) so values with spaces/newlines/tabs +# survive the line-based read intact. +ssm_count=0 +while IFS=$'\t' read -r nb vb; do + [ -n "$nb" ] || continue + name="$(printf '%s' "$nb" | b64d)" + value="$(printf '%s' "$vb" | b64d; printf 'x')"; value="${value%x}" + key="${name##*/}" # strip /open-swe-/ prefix + [ -n "$key" ] || continue + accept_var "$key" "$value" "SSM" + ssm_count=$((ssm_count + 1)) +done < <(jq -r '.Parameters[] | (.Name|@base64) + "\t" + (.Value|@base64)' <<<"$ssm_json") +log "loaded ${ssm_count} config param(s) from SSM" + +# 2) Secrets Manager (sensitive values). batch-get-secret-value filters by name +# prefix and auto-paginates; one secret per env var, SecretString = the value. +log "reading secrets under ${SECRET_PREFIX} ..." +secret_count=0 +while IFS=$'\t' read -r nb vb; do + [ -n "$nb" ] || continue + name="$(printf '%s' "$nb" | b64d)" + case "$name" in + "${SECRET_PREFIX}"*) ;; # defensive: exact-prefix only + *) continue ;; + esac + value="$(printf '%s' "$vb" | b64d; printf 'x')"; value="${value%x}" + key="${name##*/}" + [ -n "$key" ] || continue + accept_var "$key" "$value" "Secrets" + secret_count=$((secret_count + 1)) +done < <( + aws secretsmanager batch-get-secret-value \ + --filters "Key=name,Values=${SECRET_PREFIX}" \ + --region "$REGION" \ + --no-cli-pager \ + --output json \ + | jq -r '.SecretValues[] | select(.SecretString != null) | (.Name|@base64) + "\t" + (.SecretString|@base64)' +) +log "loaded ${secret_count} secret(s) from Secrets Manager" + +# --- Sea Haven DEFAULT_REPO_OWNER hard pin ----------------------------------- +# T5 OSWE-OWNER-04: a HARD pin, not a deny-list. Whatever the store holds (blank, +# upstream 'langchain-ai', a case/space variant, or any other org), the owner is +# unconditionally forced to the Sea Haven org so the agent can never target the +# wrong owner. +cur_owner="${VARS[DEFAULT_REPO_OWNER]:-}" +if [ -n "$cur_owner" ] && [ "$cur_owner" != "$SH_REPO_OWNER" ]; then + log "WARNING: SSM DEFAULT_REPO_OWNER differs from the pinned org -> hard-pinning to '${SH_REPO_OWNER}'" +fi +VARS[DEFAULT_REPO_OWNER]="$SH_REPO_OWNER" + +# --- FAIL-FAST: required vars ------------------------------------------------- +# Hard-required regardless of mode: +required=( + DASHBOARD_JWT_SECRET # RuntimeError on startup if missing (oauth.py) + TOKEN_ENCRYPTION_KEY # Fernet key(s); decrypts per-user GitHub tokens + GITHUB_APP_ID # GitHub App trio (installation-token minting) ... + GITHUB_APP_PRIVATE_KEY # ... multiline PEM ... + GITHUB_APP_INSTALLATION_ID # ... used by utils/github_app.py + GITHUB_APP_CLIENT_ID # dashboard OAuth login (prod parity) + GITHUB_APP_CLIENT_SECRET # dashboard OAuth login (prod parity) +) + +# Active model-provider key(s): model selection is store-driven (team_settings), +# so fetch-config cannot infer it from .env. Default to the seeded cross-family +# pair (anthropic builder + openai reviewer). Override with a comma list. +IFS=',' read -r -a provider_keys <<<"${REQUIRED_PROVIDER_KEYS:-ANTHROPIC_API_KEY,OPENAI_API_KEY}" +for k in "${provider_keys[@]}"; do + k="${k//[[:space:]]/}" + [ -n "$k" ] && required+=("$k") +done + +# Sandbox provider key(s) — depends on SANDBOX_TYPE (default langsmith). +sandbox_type="${VARS[SANDBOX_TYPE]:-langsmith}" +case "$sandbox_type" in + langsmith) required+=(LANGSMITH_API_KEY_PROD DEFAULT_SANDBOX_SNAPSHOT_ID) ;; + daytona) required+=(DAYTONA_API_KEY) ;; + runloop) required+=(RUNLOOP_API_KEY) ;; + modal | local) ;; # no key required + *) log "WARNING: unknown SANDBOX_TYPE='${sandbox_type}' — not enforcing a sandbox key" ;; +esac + +# Prod parity: webhook-signing secrets default empty in code but are required in +# prod. Linear's is required only when the Linear integration is wired. +if [ "$ENV" = "prod" ]; then + required+=(GITHUB_WEBHOOK_SECRET SLACK_SIGNING_SECRET) + if [ -n "${VARS[LINEAR_API_KEY]:-}" ] && [ "${OPENSWE_REQUIRE_LINEAR:-1}" = "1" ]; then + required+=(LINEAR_WEBHOOK_SECRET) + fi +fi + +missing=() +for k in "${required[@]}"; do + [ -n "${VARS[$k]:-}" ] || missing+=("$k") +done +# de-dup the names for a clean report +if [ "${#missing[@]}" -gt 0 ]; then + mapfile -t missing < <(printf '%s\n' "${missing[@]}" | sort -u) + echo "fetch-config[$ENV]: FAIL-FAST — ${#missing[@]} required var(s) missing/empty:" >&2 + printf ' - %s\n' "${missing[@]}" >&2 + echo "fetch-config[$ENV]: refusing to write a partial .env; service will not start." >&2 + exit 1 +fi + +# --- Write the .env atomically (root-only on tmpfs) -------------------------- +# python-dotenv reads double-quoted values (incl. multiline PEMs). Its decoder +# unescapes ONLY backslash and double-quote (\\ -> \, \" -> "); it does NOT honor +# \$ or \` escapes, so escaping those would leave a spurious backslash. Escape +# exactly backslash then double-quote — real newlines stay literal (multiline OK). +# (Caveat: python-dotenv interpolates a literal `${VAR}` substring; the secret +# domain here — base64/hex/PEM keys — never contains one, so no extra guard.) +emit_var() { + local name="$1" value="$2" esc + esc="${value//\\/\\\\}" + esc="${esc//\"/\\\"}" + printf '%s="%s"\n' "$name" "$esc" +} + +tmp="$(mktemp "${ENV_DIR}/.env.XXXXXX")" +chmod 600 "$tmp" +{ + printf '# Generated by fetch-config.sh for env=%s at %s — DO NOT EDIT.\n' \ + "$ENV" "$(date -u +%Y-%m-%dT%H:%M:%SZ)" + printf '# Source: SSM /open-swe-%s/* + Secrets Manager open-swe-%s/*\n\n' "$ENV" "$ENV" + for k in $(printf '%s\n' "${!VARS[@]}" | sort); do + emit_var "$k" "${VARS[$k]}" + done +} >"$tmp" + +mv -f "$tmp" "$ENV_FILE" +# Owned by the unprivileged service user (T5 SC-01) so the app reads it without +# running as root. fetch-config itself runs as root (ExecStartPre=+) to chown. +chown "${SERVICE_USER}:${SERVICE_GROUP}" "$ENV_FILE" +chmod 600 "$ENV_FILE" + +# Point the app's CWD .env at the tmpfs file (idempotent). +if [ "$APP_ENV_LINK" != "$ENV_FILE" ]; then + if [ -L "$APP_ENV_LINK" ] || [ ! -e "$APP_ENV_LINK" ]; then + ln -sfn "$ENV_FILE" "$APP_ENV_LINK" + elif [ "$(readlink -f "$APP_ENV_LINK" 2>/dev/null || true)" != "$(readlink -f "$ENV_FILE")" ]; then + log "WARNING: ${APP_ENV_LINK} exists and is not a symlink to ${ENV_FILE} — leaving it untouched" + fi +fi + +total=$((ssm_count + secret_count)) +log "wrote ${ENV_FILE} (${total} vars, sandbox=${sandbox_type}) — ${SERVICE_USER}:${SERVICE_GROUP} 0600" diff --git a/deploy/seahaven/seed_store.sh b/deploy/seahaven/seed_store.sh index 320dad93..1125a760 100755 --- a/deploy/seahaven/seed_store.sh +++ b/deploy/seahaven/seed_store.sh @@ -5,70 +5,175 @@ # to it (team model settings, user mappings) is lost on every restart. This # script idempotently re-PUTs that state and is wired as a systemd # ExecStartPost on the open-swe.service unit so it runs after each start. +# IT MUST RE-RUN ON EVERY RESTART — the in-memory store starts empty each boot. # # Replace this with Postgres-backed durability (Aegra / `langgraph up`) to make # the store survive restarts and drop this script. # -# Configuration comes from the environment (set these in the service env or a -# sourced file alongside the app .env) — no real values are committed here: -# OPENSWE_BASE_URL default http://127.0.0.1:2024 -# OPENSWE_AGENT_MODEL default anthropic:claude-opus-4-8 -# OPENSWE_AGENT_EFFORT default high -# OPENSWE_REVIEWER_MODEL default openai:gpt-5.5 -# OPENSWE_REVIEWER_EFFORT default high -# OPENSWE_DEFAULT_REPO e.g. your-org/your-pilot-repo (required) -# OPENSWE_OWNER_LOGIN GitHub login of the triggering owner (required) -# OPENSWE_OWNER_EMAIL work email mapped to that login (required) +# AWS-env-aware: pass the env as $1 (dev|prod). Seed values (default repo, model +# ids, user mappings) come from the fetch-config-materialized .env. +# +# SECURITY (T5 /sh-security-review): +# - SH-INJ-001: this script NEVER `source`s the .env. python-dotenv and bash +# have incompatible escaping, and a config value like `$(cmd)` would execute +# when sourced. We extract the few single-line seed keys with a non-eval +# reader (read_env) instead. +# - SH-INJ-003: the store PUT bodies are built with `jq --arg`, so values are +# always JSON-encoded (no string interpolation into a JSON heredoc). +# - SH-INJ-004: BASE is pinned to loopback — never derived from store/SSM +# config (LANGGRAPH_URL) — so a tampered value can't redirect the PUTs. +# - SC-02: not sourcing the .env means secrets are never exported into this +# script's (or curl's) environment. +# +# Seed values read from the materialized .env (set in SSM /open-swe-/*): +# DEFAULT_REPO_OWNER / DEFAULT_REPO_NAME -> team_settings default_repo +# LLM_MODEL_ID -> default builder model (fallback) +# SEED_AGENT_MODEL / SEED_AGENT_EFFORT -> builder model + effort (optional) +# SEED_REVIEWER_MODEL / SEED_REVIEWER_EFFORT -> reviewer model + effort (optional) +# SEED_USER_MAPPINGS -> "login:email,login:email" (optional) +# CONFIGURED_ADMINS -> "login,email" fallback for the mapping +# Legacy OPENSWE_* process-env overrides are still honored (highest precedence). set -euo pipefail -BASE="${OPENSWE_BASE_URL:-http://127.0.0.1:2024}" -AGENT_MODEL="${OPENSWE_AGENT_MODEL:-anthropic:claude-opus-4-8}" -AGENT_EFFORT="${OPENSWE_AGENT_EFFORT:-high}" -REVIEWER_MODEL="${OPENSWE_REVIEWER_MODEL:-openai:gpt-5.5}" -REVIEWER_EFFORT="${OPENSWE_REVIEWER_EFFORT:-high}" -DEFAULT_REPO="${OPENSWE_DEFAULT_REPO:?set OPENSWE_DEFAULT_REPO=owner/repo}" -OWNER_LOGIN="${OPENSWE_OWNER_LOGIN:?set OPENSWE_OWNER_LOGIN=github-login}" -OWNER_EMAIL="${OPENSWE_OWNER_EMAIL:?set OPENSWE_OWNER_EMAIL=work-email}" +ENV="${1:-${OPENSWE_ENV:-}}" +case "$ENV" in + dev | prod | "") ;; # empty allowed: pure-env / on-prem backward-compat mode + *) + echo "seed_store: ENV must be 'dev' or 'prod' (got '$ENV')" >&2 + exit 2 + ;; +esac + +ENV_DIR="${ENV_DIR:-/run/open-swe}" +ENV_FILE="${ENV_FILE:-${ENV_DIR}/.env}" + +# read_env KEY -> prints the value of a SINGLE-LINE `KEY="..."` entry from the +# materialized .env WITHOUT shell evaluation (SH-INJ-001 fix). Seed keys are +# simple single-line values; multiline secrets (e.g. the PEM) are never read +# here. Returns empty if the key is absent/unreadable. +read_env() { + local key="$1" line + [ -r "$ENV_FILE" ] || return 0 + line="$(grep -m1 -- "^${key}=" "$ENV_FILE" 2>/dev/null || true)" + [ -n "$line" ] || return 0 + line="${line#*=}" + # strip one layer of surrounding double quotes (python-dotenv double-quoted form) + if [ "${line#\"}" != "$line" ]; then line="${line%\"}"; line="${line#\"}"; fi + # reverse python-dotenv double-quote escaping (only \" and \\ are escaped) + line="${line//\\\"/\"}"; line="${line//\\\\/\\}" + printf '%s' "$line" +} + +# Process-env override (legacy/on-prem) -> .env value -> default. +pick() { # pick DEFAULT OVERRIDE_VALUE FILE_KEY... + local def="$1" override="$2"; shift 2 + if [ -n "$override" ]; then printf '%s' "$override"; return; fi + local k v + for k in "$@"; do v="$(read_env "$k")"; [ -n "$v" ] && { printf '%s' "$v"; return; }; done + printf '%s' "$def" +} + +# BASE is loopback-pinned (SH-INJ-004): this on-box seeder only talks to the +# local server; OPENSWE_PORT may override the port but never the host. +BASE="http://127.0.0.1:${OPENSWE_PORT:-2024}" + +AGENT_MODEL="$(pick 'anthropic:claude-opus-4-8' "${OPENSWE_AGENT_MODEL:-}" SEED_AGENT_MODEL LLM_MODEL_ID)" +AGENT_EFFORT="$(pick 'high' "${OPENSWE_AGENT_EFFORT:-}" SEED_AGENT_EFFORT)" +REVIEWER_MODEL="$(pick 'openai:gpt-5.5' "${OPENSWE_REVIEWER_MODEL:-}" SEED_REVIEWER_MODEL)" +REVIEWER_EFFORT="$(pick 'high' "${OPENSWE_REVIEWER_EFFORT:-}" SEED_REVIEWER_EFFORT)" + +# default_repo = owner/name from AWS config (DEFAULT_REPO_OWNER is hard-pinned +# away from upstream by fetch-config.sh). +REPO_OWNER="$(pick '' '' DEFAULT_REPO_OWNER)" +REPO_NAME="$(pick '' '' DEFAULT_REPO_NAME)" +if [ -n "${OPENSWE_DEFAULT_REPO:-}" ]; then + DEFAULT_REPO="$OPENSWE_DEFAULT_REPO" +elif [ -n "$REPO_OWNER" ] && [ -n "$REPO_NAME" ]; then + DEFAULT_REPO="${REPO_OWNER}/${REPO_NAME}" +else + echo "seed_store: set OPENSWE_DEFAULT_REPO=owner/repo (or DEFAULT_REPO_OWNER + DEFAULT_REPO_NAME in .env)" >&2 + exit 1 +fi + +# user_mappings: explicit SEED_USER_MAPPINGS ("login:email,..."), then legacy +# OPENSWE_OWNER_LOGIN/EMAIL, then parse CONFIGURED_ADMINS ("login,email"). +SEED_MAP="$(pick '' "${SEED_USER_MAPPINGS:-}" SEED_USER_MAPPINGS)" +ADMINS="$(pick '' "${CONFIGURED_ADMINS:-}" CONFIGURED_ADMINS)" +declare -a MAPPINGS=() +if [ -n "$SEED_MAP" ]; then + IFS=',' read -r -a _pairs <<<"$SEED_MAP" + for p in "${_pairs[@]}"; do + p="${p//[[:space:]]/}" + [ -n "$p" ] && MAPPINGS+=("$p") + done +elif [ -n "${OPENSWE_OWNER_LOGIN:-}" ] && [ -n "${OPENSWE_OWNER_EMAIL:-}" ]; then + MAPPINGS+=("${OPENSWE_OWNER_LOGIN}:${OPENSWE_OWNER_EMAIL}") +elif [ -n "$ADMINS" ]; then + _login="" _email="" + IFS=',' read -r -a _toks <<<"$ADMINS" + for t in "${_toks[@]}"; do + t="${t//[[:space:]]/}" + [ -z "$t" ] && continue + case "$t" in + *@*) [ -z "$_email" ] && _email="$t" ;; + *) [ -z "$_login" ] && _login="$t" ;; + esac + done + [ -n "$_login" ] && [ -n "$_email" ] && MAPPINGS+=("${_login}:${_email}") +fi + +if [ "${#MAPPINGS[@]}" -eq 0 ]; then + echo "seed_store: no user mapping resolved — set SEED_USER_MAPPINGS or OPENSWE_OWNER_LOGIN/EMAIL" >&2 + exit 1 +fi + NOW="$(date -u +%Y-%m-%dT%H:%M:%S+00:00)" -# Wait for the server to accept requests (up to ~60s). +# Wait for the server to accept requests (up to ~60s). Authoritative (OSWE-SEED-03): +# if it never comes up, log and exit 0 — do NOT fail the unit into a restart loop. +READY=0 for _ in $(seq 1 30); do - [ "$(curl -s -o /dev/null -w '%{http_code}' "$BASE/ok" || true)" = "200" ] && break + if [ "$(curl -s -o /dev/null -w '%{http_code}' "$BASE/ok" || true)" = "200" ]; then READY=1; break; fi sleep 2 done +if [ "$READY" -ne 1 ]; then + echo "seed_store: server not ready at $BASE after ~60s; skipping seed (will reseed on next restart)" >&2 + exit 0 +fi -# 1) team_settings/default — builder + reviewer models (NOT read from env by the -# app; the store value wins over LLM_MODEL_ID). gpt-4.1 is NOT in this fork's -# SUPPORTED_MODELS, so the reviewer uses gpt-5.5 (cross-family vs the builder). -curl -s -X PUT "$BASE/store/items" -H "Content-Type: application/json" -d @- </dev/null \ + || echo "seed_store: WARN team_settings PUT failed (will reseed next restart)" >&2 # 2) user_mappings/ — required, or the @openswe trigger ignores the commenter. -curl -s -X PUT "$BASE/store/items" -H "Content-Type: application/json" -d @- <&2 + continue + fi + map_body="$(jq -n --arg login "$login" --arg email "$email" --arg now "$NOW" \ + '{namespace:["user_mappings"],key:$login,value:{ + github_login:$login, work_email:$email, slack_user_id:null, + source:"slack_oauth", status:"active", created_at:$now, updated_at:$now}}')" + curl -fsS -X PUT "$BASE/store/items" -H "Content-Type: application/json" -d "$map_body" >/dev/null \ + || echo "seed_store: WARN user_mapping PUT failed for $login" >&2 +done -echo "seed_store: done at $NOW" +echo "seed_store: done at $NOW (env=${ENV:-none}, repo=$DEFAULT_REPO, mappings=${#MAPPINGS[@]})"