fix(deploy): recover SAM stacks after update rollback

This commit is contained in:
Adam Moussa 2026-08-03 14:34:45 -04:00
parent 9a7171a855
commit 8d85b2c396
No known key found for this signature in database
3 changed files with 22 additions and 9 deletions

View file

@ -69,7 +69,7 @@ jobs:
--stack-name "${{ inputs.stack-name }}" \
--query 'Stacks[0].StackStatus' --output text 2>/dev/null || echo "NOT_FOUND")
case "$STATUS" in
*ROLLBACK_COMPLETE|*FAILED)
ROLLBACK_COMPLETE|*FAILED)
echo "::error::Stack ${{ inputs.stack-name }} is in $STATUS — manual intervention required."
exit 1
;;

View file

@ -121,7 +121,7 @@ This ordering rule is about changing the **boundary** or the conditions that gat
- **Removing `PermissionsBoundary` from an existing role fails by design.** CloudFormation issues `DeleteRolePermissionsBoundary` for that edit, gets `AccessDenied`, and the stack update rolls back. Removing the boundary from a SAM function is a security regression, so failing loudly is intended.
- **Rollback of an update that *adds* a boundary to an existing role would also fail**, landing the stack in `UPDATE_ROLLBACK_FAILED`. This is currently unreachable — all 26 IAM roles across the five SAM stacks already carry the boundary (verified 2026-07-27), so no update can add one. It becomes reachable again only if a role is created without the boundary and given one later.
Recovery in either case is an administrator action, not a pipeline retry: clear the wedged stack with `aws cloudformation continue-update-rollback --stack-name <stack> --resources-to-skip <RoleLogicalId>`, or replace the role by renaming its logical id. Note `cd-sam`'s pre-flight hard-fails on `*ROLLBACK_COMPLETE`, so that repo's deploys stay blocked until it is cleared.
Recovery from `UPDATE_ROLLBACK_FAILED` is an administrator action, not a pipeline retry: clear the wedged stack with `aws cloudformation continue-update-rollback --stack-name <stack> --resources-to-skip <RoleLogicalId>`, or replace the role by renaming its logical id. A completed update rollback lands in `UPDATE_ROLLBACK_COMPLETE`, which is stable and can accept a corrective update; `cd-sam` blocks only first-create `ROLLBACK_COMPLETE` and failed or in-progress states.
## Setup

View file

@ -355,13 +355,13 @@ Resources:
# But there IS one path that now fails by design: updating an
# existing AWS::IAM::Role to REMOVE its PermissionsBoundary property
# makes CloudFormation call DeleteRolePermissionsBoundary, which is
# denied. The stack update fails and rolls back, and because cd-sam's
# pre-flight hard-fails on *ROLLBACK_COMPLETE, that repo's deploys
# stay blocked until it is cleared. Recovery is an out-of-band admin
# action (remove the boundary directly, or replace the role by
# renaming its logical id) — not a pipeline retry. Removing the
# boundary from a SAM function is a security regression anyway, so
# failing loudly here is the intent.
# denied. If rollback also fails, the stack reaches
# UPDATE_ROLLBACK_FAILED and needs out-of-band admin recovery (remove
# the boundary directly, skip the resource during rollback, or
# replace the role by renaming its logical id). A successful rollback
# reaches UPDATE_ROLLBACK_COMPLETE and can accept a corrective update.
# Removing the boundary from a SAM function is a security regression
# anyway, so failing loudly here is the intent.
- Sid: IAMPutPermissionsBoundary
Effect: Allow
Action:
@ -479,6 +479,19 @@ Resources:
StringEquals:
"iam:PassedToService": "lambda.amazonaws.com"
# API Gateway assumes SAM authorizer invocation roles. Keep this
# separate from Lambda PassRole so each target service and role
# pattern remains independently constrained.
- Sid: IAMPassAuthorizerRole
Effect: Allow
Action:
- iam:PassRole
Resource:
- !Sub "arn:aws:iam::${AWS::AccountId}:role/cfn-managed/*AuthorizerInvokeRole-*"
Condition:
StringEquals:
"iam:PassedToService": "apigateway.amazonaws.com"
# ---------------------------------------------------------------------------
# Shared CloudFormation execution role (SAM stacks) — INFRA-97 scoped
#