fix(deploy): recover SAM stacks after update rollback

This commit is contained in:
Adam Moussa 2026-08-03 14:38:39 -04:00 • committed by GitHub
parent 9a7171a855
commit 81cf168170
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
3 changed files with 25 additions and 25 deletions

View file

@ -69,7 +69,7 @@ jobs:
--stack-name "${{ inputs.stack-name }}" \
--query 'Stacks[0].StackStatus' --output text 2>/dev/null || echo "NOT_FOUND")
case "$STATUS" in
*ROLLBACK_COMPLETE|*FAILED)
ROLLBACK_COMPLETE|*FAILED)
echo "::error::Stack ${{ inputs.stack-name }} is in $STATUS — manual intervention required."
exit 1
;;

View file

@ -121,7 +121,7 @@ This ordering rule is about changing the **boundary** or the conditions that gat
- **Removing `PermissionsBoundary` from an existing role fails by design.** CloudFormation issues `DeleteRolePermissionsBoundary` for that edit, gets `AccessDenied`, and the stack update rolls back. Removing the boundary from a SAM function is a security regression, so failing loudly is intended.
- **Rollback of an update that *adds* a boundary to an existing role would also fail**, landing the stack in `UPDATE_ROLLBACK_FAILED`. This is currently unreachable — all 26 IAM roles across the five SAM stacks already carry the boundary (verified 2026-07-27), so no update can add one. It becomes reachable again only if a role is created without the boundary and given one later.
Recovery in either case is an administrator action, not a pipeline retry: clear the wedged stack with `aws cloudformation continue-update-rollback --stack-name <stack> --resources-to-skip <RoleLogicalId>`, or replace the role by renaming its logical id. Note `cd-sam`'s pre-flight hard-fails on `*ROLLBACK_COMPLETE`, so that repo's deploys stay blocked until it is cleared.
Recovery from `UPDATE_ROLLBACK_FAILED` is an administrator action, not a pipeline retry: clear the wedged stack with `aws cloudformation continue-update-rollback --stack-name <stack> --resources-to-skip <RoleLogicalId>`, or replace the role by renaming its logical id. A completed update rollback lands in `UPDATE_ROLLBACK_COMPLETE`, which is stable and can accept a corrective update; `cd-sam` blocks only first-create `ROLLBACK_COMPLETE` and failed or in-progress states.
## Setup

View file

@ -338,30 +338,17 @@ Resources:
StringEquals:
"iam:PermissionsBoundary": !Sub "arn:aws:iam::${AWS::AccountId}:policy/seahaven-lambda-execution-boundary"
# Boundary management — SET the boundary only. DELETE is NOT
# granted: for a delete, the iam:PermissionsBoundary condition key
# reflects the boundary CURRENTLY attached to the target role, so
# a StringEquals condition on the boundary ARN MATCHES exactly the
# roles the gate protects. Granting delete under that condition
# lets this role create a boundary-gated role with an inline *:*
# policy, strip the boundary, and pass the now-unbounded role to
# Lambda — defeating the primary escalation control. Verified live
# against the mgmt copy 2026-07-27 (simulate-principal-policy:
# iam:DeleteRolePermissionsBoundary = allowed).
# Boundary management is SET-only. Granting delete would let this
# role create a boundary-gated role, strip the boundary, then pass an
# unconstrained role to Lambda. SAM creation and teardown need only
# PutRolePermissionsBoundary and DeleteRole.
#
# OPERATIONAL CONSEQUENCE — read before debugging a stuck stack.
# SAM does not need the delete for the common paths: it SETS the
# boundary on roles it creates, and stack teardown calls DeleteRole.
# But there IS one path that now fails by design: updating an
# existing AWS::IAM::Role to REMOVE its PermissionsBoundary property
# makes CloudFormation call DeleteRolePermissionsBoundary, which is
# denied. The stack update fails and rolls back, and because cd-sam's
# pre-flight hard-fails on *ROLLBACK_COMPLETE, that repo's deploys
# stay blocked until it is cleared. Recovery is an out-of-band admin
# action (remove the boundary directly, or replace the role by
# renaming its logical id) — not a pipeline retry. Removing the
# boundary from a SAM function is a security regression anyway, so
# failing loudly here is the intent.
# Removing a boundary therefore fails by design. A failed rollback
# reaches UPDATE_ROLLBACK_FAILED and needs admin recovery by skipping
# or replacing the role. A successful rollback reaches the stable
# UPDATE_ROLLBACK_COMPLETE state and can accept a corrective update.
# Removing a SAM function boundary is a security regression, so
# failing loudly is intentional.
- Sid: IAMPutPermissionsBoundary
Effect: Allow
Action:
@ -479,6 +466,19 @@ Resources:
StringEquals:
"iam:PassedToService": "lambda.amazonaws.com"
# API Gateway assumes SAM authorizer invocation roles. Keep this
# separate from Lambda PassRole so each target service and role
# pattern remains independently constrained.
- Sid: IAMPassAuthorizerRole
Effect: Allow
Action:
- iam:PassRole
Resource:
- !Sub "arn:aws:iam::${AWS::AccountId}:role/cfn-managed/*AuthorizerInvokeRole-*"
Condition:
StringEquals:
"iam:PassedToService": "apigateway.amazonaws.com"
# ---------------------------------------------------------------------------
# Shared CloudFormation execution role (SAM stacks) — INFRA-97 scoped
#