diff --git a/infra/README.md b/infra/README.md index f3f964f4..7eb87ddd 100644 --- a/infra/README.md +++ b/infra/README.md @@ -98,6 +98,27 @@ Three buckets: > **Confirm** the 27-vs-29 count (code-only candidates not in the table: > `USER_ID_API_KEY_MAP`, `JUDGE_ANTHROPIC_BASE_URL`). + > ⚠️ **`Retain` + fixed name orphans these shells on a failed FIRST create.** + > If the stack's initial create fails and rolls back, `Retain` keeps the shells + > instead of deleting them. The stack is then gone, but the secrets survive, + > still holding the global `open-swe-/` names — so every later create + > fails with `AlreadyExists`. A plain `delete-secret` does **not** clear it + > (the name stays reserved for the 7–30 day recovery window). This bites on a + > **teardown/rebuild, a secret logical-id change/refactor, or standing up a new + > env** — never on routine updates of an already-created stack. **Recovery —** + > before re-creating the stack, force-delete the *empty* orphans so the names + > free immediately: + > ```bash + > aws secretsmanager list-secrets --region us-east-1 \ + > --filters Key=name,Values=open-swe-/ \ + > --query 'SecretList[].Name' --output text | tr '\t' '\n' | while read -r n; do + > aws secretsmanager delete-secret --secret-id "$n" \ + > --region us-east-1 --force-delete-without-recovery + > done + > ``` + > Force-delete only shells with **no value version** — a populated secret holds + > real operator material. (Hit on prod 2026-06-29; see PR #51's deploy failure.) + 2. **IaC-managed SSM config — 8 params, real values owned in code:** | Param | dev | prod | diff --git a/infra/lib/constructs/config-store.ts b/infra/lib/constructs/config-store.ts index 2f759d48..4bc79a52 100644 --- a/infra/lib/constructs/config-store.ts +++ b/infra/lib/constructs/config-store.ts @@ -227,6 +227,23 @@ export class ConfigStore extends Construct { // creates an empty secret, so the out-of-band value is never clobbered. }); // RETAIN: a stack teardown must not destroy operator-set secret material. + // + // GOTCHA — RETAIN + fixed name orphans these shells on a FAILED FIRST + // CREATE. If the stack's initial create fails and rolls back, RETAIN keeps + // the shells instead of deleting them; the stack is then gone but the + // secrets survive, still holding the global `open-swe-/` names. + // Every later create then fails with `AlreadyExists` (and a normal + // delete-secret keeps the name reserved for the 7–30 day recovery window, + // so it does NOT clear the deadlock). Recovery: before re-creating the + // stack, force-delete the orphans so the names free immediately, e.g. + // aws secretsmanager list-secrets --filters Key=name,Values=open-swe-/ \ + // --query 'SecretList[].Name' --output text | tr '\t' '\n' | while read n; do + // aws secretsmanager delete-secret --secret-id "$n" \ + // --force-delete-without-recovery; done + // Only force-delete shells that are EMPTY (no value version) — a populated + // secret holds real operator material. This bites on teardown/rebuild, a + // secret logical-id change/refactor, or standing up a new env — NOT on + // routine updates of an already-created stack. (Hit on prod 2026-06-29.) secret.applyRemovalPolicy(cdk.RemovalPolicy.RETAIN); this.secrets.push(secret); }