From f379fbdaa9ac8efb21276dc0dec0e3e5314081b4 Mon Sep 17 00:00:00 2001
From: Adam Moussa <166072409+amoussa1229@users.noreply.github.com>
Date: Mon, 29 Jun 2026 10:51:49 -0400
Subject: [PATCH] docs: document RETAIN secret-shell orphan gotcha (#52)

RETAIN + a fixed secret name means a failed FIRST create leaves empty
secret shells behind when the stack rolls back. The shells keep the
global `open-swe-<env>/<VAR>` names, so every later create fails with
`AlreadyExists`, and a plain delete-secret keeps the name reserved for
the recovery window rather than freeing it.

Record the trap and the force-delete recovery (only for empty shells)
in the config-store construct and the infra README so the next
teardown/rebuild, secret logical-id change, or new-env stand-up does
not rediscover it the hard way.

Prod's first deploy hit this on 2026-06-29: 28 orphaned shells from an
earlier failed create reserved the names and had to be force-deleted
before the stack would create.
---
 infra/README.md                      | 21 +++++++++++++++++++++
 infra/lib/constructs/config-store.ts | 17 +++++++++++++++++
 2 files changed, 38 insertions(+)

diff --git a/infra/README.md b/infra/README.md
index f3f964f4..7eb87ddd 100644
--- a/infra/README.md
+++ b/infra/README.md
@@ -98,6 +98,27 @@ Three buckets:
    > **Confirm** the 27-vs-29 count (code-only candidates not in the table:
    > `USER_ID_API_KEY_MAP`, `JUDGE_ANTHROPIC_BASE_URL`).
 
+   > ⚠️ **`Retain` + fixed name orphans these shells on a failed FIRST create.**
+   > If the stack's initial create fails and rolls back, `Retain` keeps the shells
+   > instead of deleting them. The stack is then gone, but the secrets survive,
+   > still holding the global `open-swe-<env>/<VAR>` names — so every later create
+   > fails with `AlreadyExists`. A plain `delete-secret` does **not** clear it
+   > (the name stays reserved for the 7–30 day recovery window). This bites on a
+   > **teardown/rebuild, a secret logical-id change/refactor, or standing up a new
+   > env** — never on routine updates of an already-created stack. **Recovery —**
+   > before re-creating the stack, force-delete the *empty* orphans so the names
+   > free immediately:
+   > ```bash
+   > aws secretsmanager list-secrets --region us-east-1 \
+   >   --filters Key=name,Values=open-swe-<env>/ \
+   >   --query 'SecretList[].Name' --output text | tr '\t' '\n' | while read -r n; do
+   >     aws secretsmanager delete-secret --secret-id "$n" \
+   >       --region us-east-1 --force-delete-without-recovery
+   >   done
+   > ```
+   > Force-delete only shells with **no value version** — a populated secret holds
+   > real operator material. (Hit on prod 2026-06-29; see PR #51's deploy failure.)
+
 2. **IaC-managed SSM config — 8 params, real values owned in code:**
 
    | Param | dev | prod |
diff --git a/infra/lib/constructs/config-store.ts b/infra/lib/constructs/config-store.ts
index 2f759d48..4bc79a52 100644
--- a/infra/lib/constructs/config-store.ts
+++ b/infra/lib/constructs/config-store.ts
@@ -227,6 +227,23 @@ export class ConfigStore extends Construct {
         // creates an empty secret, so the out-of-band value is never clobbered.
       });
       // RETAIN: a stack teardown must not destroy operator-set secret material.
+      //
+      // GOTCHA — RETAIN + fixed name orphans these shells on a FAILED FIRST
+      // CREATE. If the stack's initial create fails and rolls back, RETAIN keeps
+      // the shells instead of deleting them; the stack is then gone but the
+      // secrets survive, still holding the global `open-swe-<env>/<VAR>` names.
+      // Every later create then fails with `AlreadyExists` (and a normal
+      // delete-secret keeps the name reserved for the 7–30 day recovery window,
+      // so it does NOT clear the deadlock). Recovery: before re-creating the
+      // stack, force-delete the orphans so the names free immediately, e.g.
+      //   aws secretsmanager list-secrets --filters Key=name,Values=open-swe-<env>/ \
+      //     --query 'SecretList[].Name' --output text | tr '\t' '\n' | while read n; do
+      //       aws secretsmanager delete-secret --secret-id "$n" \
+      //         --force-delete-without-recovery; done
+      // Only force-delete shells that are EMPTY (no value version) — a populated
+      // secret holds real operator material. This bites on teardown/rebuild, a
+      // secret logical-id change/refactor, or standing up a new env — NOT on
+      // routine updates of an already-created stack. (Hit on prod 2026-06-29.)
       secret.applyRemovalPolicy(cdk.RemovalPolicy.RETAIN);
       this.secrets.push(secret);
     }