* fix(iam): let the site plan role describe SSM parameters * fix(iam): address review feedback |
||
|---|---|---|
| .github | ||
| bin | ||
| examples/hcptf-workspace-iam | ||
| lib | ||
| scripts | ||
| .gitignore | ||
| AGENTS.md | ||
| cdk.json | ||
| package-lock.json | ||
| package.json | ||
| README.md | ||
| tsconfig.json | ||
seahaven-org-baseline
Organization-wide security and governance baseline for Sea Haven Industries,
managed as a single CDK TypeScript app. Covers the management account
(328440206208: primary baseline in us-east-1, secondary-region
baselines in us-east-2/us-west-2, offsite backup vault in
us-west-2) and org member accounts (first tenant:
seahaven-external-dev 396287094661, absorbed from the retired
seahaven-external-dev-baseline repo). This is where account-wide detective
and recovery controls live, so they are versioned, reviewed, and drift-checked
like any other stack.
History: this repo was
seahaven-account-baseline(management account only) until 2026-07-14, when the external-dev member baseline was merged in and the repo renamed. Deployed CloudFormation stack names are unchanged.
Stacks (normally deployed by one CD job per target account; staged exceptions are noted):
| Stack | Account | Region | Purpose |
|---|---|---|---|
seahaven-account-baseline |
328440206208 | us-east-1 | CloudTrail + detective controls (C-1) |
seahaven-dynamodb-cmk |
328440206208 | us-east-1 | Shared customer-managed KMS key for finance/PII DynamoDB tables; ARN published to SSM /seahaven/dynamodb/cmk-arn (INFRA-95 / M-3) |
seahaven-regional-baseline-us-west-2 |
328440206208 | us-west-2 | Bedrock invocation logging (INFRA-91) |
seahaven-regional-baseline-us-east-2 |
328440206208 | us-east-2 | Bedrock invocation logging + AWS Config recorder + Security Hub (INFRA-91 / INFRA-16) |
seahaven-backup |
328440206208 | us-east-1 | Primary AWS Backup vault + plan + role (C-7) |
seahaven-backup-offsite |
328440206208 | us-west-2 | Governance-locked offsite copy vault (C-7) |
seahaven-org-governance |
328440206208 | us-east-1 | AWS Organizations OU tree + SCPs (incl. the cdk-imported external-dev guardrails) |
seahaven-external-dev-baseline |
396287094661 | us-east-1 | Member-account baseline: Config, GuardDuty, Security Hub (FSBP + CIS v3.0), Access Analyzer, flow logs, budget |
seahaven-terraform-substrate |
011934824531, 710827005802 | us-east-1 | Prod/dev HCP substrate: OIDC + shared IAM manager + eight existing hcptf-* pairs (DeletionPolicy Retain). Pending PLAT-147 delete after consumer imports. New prod/dev HCP IAM is not added here. |
seahaven-site-hcptf |
011934824531 | us-east-1 | Imported hcptf-seahaven-site apply and plan roles (PLAT-225). On the prod deploy job. Do not create. |
seahaven-terraform-substrate |
396287094661 | us-east-1 | Staged manually for SHOC backend/frontend adoption; exact HCP roles and deploy boundaries referencing the existing OIDC provider. Stays (PLAT-148). |
seahaven-security-baseline |
001520130573 | us-east-1 | Member-account baseline for the delegated security-admin account (same construct set) |
seahaven-dev-baseline |
710827005802 | us-east-1 | Member-account baseline for internal dev/staging (org-managed detection — no local GuardDuty/SecurityHub) |
seahaven-prod-baseline |
011934824531 | us-east-1 | Member-account baseline for production workloads (org-managed detection; mgmt account frozen for new workloads) |
CDK app
The repo is a single AWS CDK app written in TypeScript. cdk.json is the
project config the cdk CLI reads on every command: its app key
(npx ts-node bin/app.ts) tells CDK how to synthesize the app straight from
the TypeScript source — no separate compile step needed for cdk synth /
diff / deploy — and its context block carries the AWS CDK feature flags.
| Path | Role |
|---|---|
cdk.json |
CDK config: app synth command, watch includes/excludes, context feature flags |
bin/app.ts |
App entry point — instantiates every stack with an explicit kebab-case stackName and its target env (five accounts, per-account/per-region) |
lib/*-stack.ts |
Stack definitions (one class per stack; larger stacks compose the constructs in lib/*.ts) |
tsconfig.json |
TypeScript compiler options (outDir: cdk.out) |
package.json |
Pinned aws-cdk-lib, CDK CLI, and the build / synth / diff / deploy npm scripts |
bin/app.ts synthesizes these stacks across three regions and five accounts:
| Construct id | Stack name | Account | Region | Source |
|---|---|---|---|---|
account-baseline |
seahaven-account-baseline |
328440206208 | us-east-1 | lib/account-baseline-stack.ts |
dynamodb-cmk |
seahaven-dynamodb-cmk |
328440206208 | us-east-1 | lib/dynamodb-cmk-stack.ts |
regional-baseline-us-west-2 |
seahaven-regional-baseline-us-west-2 |
328440206208 | us-west-2 | lib/regional-baseline-stack.ts |
regional-baseline-us-east-2 |
seahaven-regional-baseline-us-east-2 |
328440206208 | us-east-2 | lib/regional-baseline-stack.ts |
backup-offsite |
seahaven-backup-offsite |
328440206208 | us-west-2 | lib/backup-offsite-stack.ts |
backup |
seahaven-backup |
328440206208 | us-east-1 | lib/backup-stack.ts |
org-governance |
seahaven-org-governance |
328440206208 | us-east-1 | lib/org-governance-stack.ts |
external-dev-baseline |
seahaven-external-dev-baseline |
396287094661 | us-east-1 | lib/member-baseline-stack.ts |
security-baseline |
seahaven-security-baseline |
001520130573 | us-east-1 | lib/member-baseline-stack.ts |
dev-baseline |
seahaven-dev-baseline |
710827005802 | us-east-1 | lib/member-baseline-stack.ts (orgManagedDetection) |
prod-baseline |
seahaven-prod-baseline |
011934824531 | us-east-1 | lib/member-baseline-stack.ts (orgManagedDetection) |
deploy-substrate-prod |
seahaven-deploy-substrate |
011934824531 | us-east-1 | lib/deploy-substrate-stack.ts |
deploy-substrate-dev |
seahaven-deploy-substrate |
710827005802 | us-east-1 | lib/deploy-substrate-stack.ts |
terraform-substrate-prod |
seahaven-terraform-substrate |
011934824531 | us-east-1 | lib/terraform-substrate-stack.ts (pending PLAT-147 delete) |
terraform-substrate-dev |
seahaven-terraform-substrate |
710827005802 | us-east-1 | lib/terraform-substrate-stack.ts (pending PLAT-147 delete) |
terraform-substrate-external-dev |
seahaven-terraform-substrate |
396287094661 | us-east-1 | lib/terraform-substrate-stack.ts |
dynamodb-cmk-prod |
seahaven-dynamodb-cmk |
011934824531 | us-east-1 | lib/dynamodb-cmk-stack.ts |
alarm-topic-prod |
seahaven-alarm-topic |
011934824531 | us-east-1 | lib/alarm-topic-stack.ts |
app-web-acl-prod |
seahaven-app-web-acl |
011934824531 | us-east-1 | lib/app-web-acl-stack.ts |
Member-account stacks deploy with per-account credentials — the CD workflow
runs one job per account, each assuming that account's OIDC deploy role. Local
deploys/diffs assume OrganizationAccountAccessRole in the target account.
| Account | OIDC deploy role | Repo secret |
|---|---|---|
| 396287094661 (external-dev) | githubdeploy-seahaven-external-dev-baseline |
AWS_DEPLOY_ROLE_ARN_EXTDEV |
| 001520130573 (security) | githubdeploy-seahaven-org-baseline |
AWS_DEPLOY_ROLE_ARN_SECURITY |
| 710827005802 (dev) | githubdeploy-seahaven-org-baseline |
AWS_DEPLOY_ROLE_ARN_DEV |
| 011934824531 (prod) | githubdeploy-seahaven-org-baseline |
AWS_DEPLOY_ROLE_ARN_PROD |
Shared constructs (DetectiveControls, FlowLogs, GovernanceToggles) are
prefix-parameterized — construct ids and physical names must stay
byte-identical to the deployed stacks (logical IDs are path-derived).
Accounts enrolled by the org delegated admin (post 2026-07-14) set
orgManagedDetection: true: the GuardDuty detector + Security Hub hub come
from the org, while STANDARDS (FSBP + CIS v3.0) and the account analyzer stay
CFN-owned (org AutoEnableStandards is NONE — the DEFAULT setting enrolls
legacy CIS v1.2.0). Enrollment (member Enabled in GuardDuty + Security Hub)
is a hard precondition for such a stack's first deploy.
backup declares an explicit dependency on backup-offsite so the offsite copy
vault exists before the primary plan that copies into it. Stack names are set
explicitly to enforce kebab-case (CDK defaults to PascalCase). Synthesized
CloudFormation templates land in cdk.out/ (git-ignored).
Common commands:
npm ci # install pinned deps
npm run build # tsc type-check (compiles to cdk.out/)
npx cdk synth # synthesize CloudFormation for all stacks
npx cdk diff # diff synthesized stacks against deployed state
npx cdk deploy --all # deploy every stack
npx cdk deploy <stack-name> # deploy a single stack
The --context <key>=<value> flag overrides cdk.json context at the command
line (e.g. the encryptTrailLogGroup toggle under Monitoring + logging).
Documentation
The canonical map of Sea Haven's AWS infrastructure lives in Confluence. This project's seahaven-account-baseline, seahaven-backup, and seahaven-backup-offsite stacks are represented there as Mermaid subgraphs.
- AWS Architecture Map (Confluence, IT space, page 1540098)
What it deploys
GitHub Actions deploy substrate (per account)
lib/deploy-substrate-stack.ts + lib/deploy-substrate/deploy-substrate.template.yaml
deploy seahaven-deploy-substrate into each member account that hosts SAM
workloads (currently seahaven-prod and seahaven-dev). It contains the shared
account-level deploy plumbing:
- the
seahaven-lambda-execution-boundarypermissions boundary (legacy shared ceiling for roles not yet retargeted) plus per-workload policiesseahaven-lambda-execution-boundary-<workload>(PLAT-52), - the
github-cfn-execution-roleCloudFormation execution role thatcd-samcallers pass ascfn-role-arn, - optionally the GitHub OIDC identity provider (
createOidcProvider: true, only for an account that does not already have one — one provider per URL per account).
The template began as a verbatim extraction of the substrate section of
Sea-Haven-Industries/.github/oidc-deploy-roles.yaml, which remains the source
of truth for mgmt (328440206208) until its stacks migrate out.
The two copies are no longer at parity, and the old "edit both files" rule no
longer applies uniformly. Under INFRA-186, seahaven-lambda-execution-boundary
in this copy was reduced to a fleet-wide floor for prod and dev; later
migrations packed per-workload data plane back into that shared document until
it hit the 6,144-character cap (PLAT-93 / PLAT-100). PLAT-52 adds per-workload
policies seahaven-lambda-execution-boundary-<workload> (floor plus that
stack's data plane) and switches both prod/dev guardrails to a StringEquals
allow-list of the shared ARN plus each per-workload ARN. The shared document
is left unchanged until live roles retarget. Mgmt's copy keeps the
account-wide wildcards and a single-ARN pin pending its own separately
validated rollout across 26 live boundary-carrying roles (PLAT-51). So: the
boundary resource is deliberately divergent, and the guardrail
iam:PermissionsBoundary condition cardinality is also divergent
(enumerated list here, scalar on mgmt). Do not weaken mgmt to ArnLike. Every
other substrate resource (github-cfn-execution-role,
seahaven-cfn-exec-iam-management Sid/Action/Resource sets) is still expected
to change in both files together. The template's provenance header records
which is which — read it before assuming either parity or divergence.
Per-repo githubdeploy-* deploy roles are deliberately NOT part
of the substrate — they are provisioned per repo at migration/onboarding time
so an account never carries trust relationships for repos that do not deploy
to it.
Escalation controls on github-cfn-execution-role. Every iam:CreateRole,
AttachRolePolicy and PutRolePolicy is conditioned on the target carrying
one of the enumerated seahaven-lambda-execution-boundary ARNs (the shared
policy plus each seahaven-lambda-execution-boundary-<workload>). That
condition alone is not sufficient,
so the attached seahaven-cfn-exec-iam-management managed policy also carries
three explicit Deny statements:
DenyBoundaryTampering— no removing a boundary from any role or user. Granting the delete under the sameStringEqualscondition self-defeats the gate, because for a delete the condition key resolves to the boundary already on the target.DenyBoundaryPolicyEdit— no rewriting anyseahaven-*managed policy.DenySelfMutation— the role cannot modify or delete itself or anygithubdeploy-*role. Without it the control is one API call from being undone:IAMRoleReadAndDeletegrantsiam:DetachRolePolicyonResource: "*"unconditioned, so the role could detach the very policy carrying these Denies.
Verify a change to these with aws iam simulate-principal-policy against the
role's own ARN (expect explicitDeny) and against a <stack>-<Function>Role-
name (expect allowed, no regression for normal SAM deploys). Note that
simulation currently does not see this role's inline policies in
seahaven-prod or seahaven-dev — read those back with get-role-policy instead.
Known consequence of DenyBoundaryTampering: a CloudFormation rollback of an
update that adds a boundary to an existing role wedges in
UPDATE_ROLLBACK_FAILED. Recovery is an administrator action, not a pipeline
retry — aws cloudformation continue-update-rollback --stack-name <stack> --resources-to-skip <RoleLogicalId>. Unreachable while every SAM role is
created with the boundary already attached.
Onboarding a future account as a deploy target:
- CDK-bootstrap the account (
npx cdk bootstrap aws://<account>/us-east-1viaOrganizationAccountAccessRole). - Create
githubdeploy-seahaven-org-baselinein the account (same trust and policy as the dev/prod copies) and add the repo secretAWS_DEPLOY_ROLE_ARN_<ACCT>. - Add a
DeploySubstrateStackinstance inbin/app.ts(createOidcProvider: trueif the account has no GitHub OIDC provider) and append its construct id to a new per-account job in.github/workflows/deploy.yaml(explicitstacksselector, one job per account). - Merge; the substrate deploys via CD. Per-repo deploy roles and app stacks follow the cross-account migration playbook from there.
Terraform deploy substrate (per account)
lib/terraform-substrate-stack.ts + lib/terraform-substrate/terraform-substrate.template.yaml
deploy seahaven-terraform-substrate into member accounts that host
Terraform-managed workloads (currently seahaven-prod, seahaven-dev, and
external-dev; never mgmt — mgmt stays SAM until its stacks migrate out).
Durable owner for prod/dev per-workload HCP IAM is app Terraform (PLAT-144). Console / one-shot CLI owns only:
- the existing
app.terraform.ioOIDC provider (alreadyRetain; an account holds one per URL), - one general apply role and one general plan role per prod and dev account
(
hcptf-bootstrap/hcptf-bootstrap-plan, created byscripts/create-hcptf-bootstrap-roles.sh, not a CDK stack).
Do not manage prod/dev workload IAM (hcptf-<stack> pairs, Lambda exec
roles, policy/tf-managed/<stack> ceilings) in the console. Do not append
new prod/dev hcptf-<stack> pairs to this template. New prod/dev HCP stacks
do not need an org-baseline IAM PR.
External-dev IAM stays in this repo (PLAT-148). SHOC backend/frontend HCP
roles, SHOC deploy/runtime boundaries, shoc-frontend-resources.ts, and
stack terraform-substrate-external-dev are not imported into shoc-backend
or shoc-frontend-new. New external-dev IAM still lands here.
Deploy-substrate stays for remaining SAM (PLAT-150).
github-cfn-execution-role, seahaven-cfn-exec-iam-management, and the
enumerated SAM StringEquals allow-list are unchanged. Do not add ArnLike
there (PLAT-52 AC1). Do not add new HCP workloads to those four Sids.
Prod/dev still carry, until PLAT-147 deletes those two stacks:
- the
app.terraform.ioOIDC identity provider (audienceaws.workload.identity; Retain — it is the federation anchor for every futurehcptf-*role), - the
seahaven-hcptf-iam-managementguardrail policy (enumeratedseahaven-lambda-execution-boundary-*allow-list; frozen, do not append), - the eight existing prod
hcptf-<stack>pairs withDeletionPolicy: Retain.
External-dev still carries:
- external-dev-only deploy boundaries
shoc-backend-{dev,staging}-deploy-boundary. Each is the maximum current policy for one exactgithubdeploy-shoc-backend-*role. Dev temporarily retains its live broadelasticbeanstalk-*S3 grants so boundary attachment cannot regress deployment before the separately reviewed policy narrowing, - external-dev-only runtime boundaries
shoc-backend-{dev,staging}-runtime-boundary. These retain only the account-scoped S3, environment health/log, and X-Ray portions ofAWSElasticBeanstalkWebTier, plus each environment's exact secrets/KMS/STS data plane. They deliberately exclude the managed policy's 2026 Bedrock/Marketplace additions.
The retired backend tf-poc deploy/runtime boundaries are no longer declared.
Because their last managed definitions used DeletionPolicy: Retain, the
external-dev stack update only removes CloudFormation ownership. Verify both
policies still have zero attachments, then delete the retained physical
policies in a separately approved post-deploy step.
seahaven-hcptf-iam-management derives from seahaven-cfn-exec-iam-management
but is deliberately stricter — it is not a mirror. The 2026-07-30 security
review confirmed the SAM copy's Resource: "*" role grants as a critical
escalation primitive (iam:UpdateAssumeRolePolicy on * repoints the
AdministratorAccess CDK bootstrap role's trust policy to an external account),
and its justification for the wildcard — SAM auto-generates execution roles at
path / with no settable RolePath — does not transfer, because Terraform's
aws_iam_role supports path. So here:
- every role write (create, delete, detach,
UpdateAssumeRolePolicy, boundary set) andiam:PassRoleis confined to the Terraform-owned pathrole/tf-managed/*; reads stay on*for data sources, - Terraform configs must set
path = "/tf-managed/"on everyaws_iam_rolethey create for workload execution — a role created anywhere else is denied, DenySelfMutationadditionally coverscdk-hnb659fds-*,OrganizationAccountAccessRoleandseahaven-*.
Do not "reconcile" the two files by copying statements between them. The
org-level control for the same class is protect-privileged-roles on prod
and nonprod (PLAT-145), covering hcptf-bootstrap* plus the break-glass /
CDK / githubdeploy-* set already on the security OU.
The eight existing prod/dev per-workspace pairs are imported into the owning
app, not recreated. After import they are Retain-removed from this template
and the prod/dev stacks are deleted. See the first-apply and import runbooks
below. Do not batch those consumer PRs; pilot is afi-backup-monitor.
External-dev SHOC role adoption is a staged CloudFormation import, not a
normal first deploy. Six roles exist today:
hcptf-shoc-backend-{dev,staging} and their -plan partners, plus the
hcptf-shoc-backend-tf-poc pair. Two independent CDK contexts make each
transition explicit:
enableShocBackendPocRoles and enableShocBackendLiveRoles. Both began as
false for the initial rollout and remain version-controlled as true after
their completed ownership transitions.
terraform-substrate-external-dev is deliberately absent from the automatic
external-dev deploy job during this sequence; external-dev-baseline remains
automatic and unchanged.
-
Create the role-free base stack:
npx cdk deploy terraform-substrate-external-dev \ -c enableShocBackendPocRoles=false \ -c enableShocBackendLiveRoles=falseCreateOIDCProvider=falseis fixed inbin/app.ts; the external-dev account therefore creates neither the existing provider, SHOC roles, nor the prod/dev-only shared IAM policy. The base stack does create the four retained backend dev/staging deploy/runtime boundary policies. -
Set
enableShocBackendPocRolestotrueincdk.json, leave the live gatefalse, review the synthesized two-role addition, then run the normal external-dev stack update. This creates only the new tf-poc HCP plan/apply pair. -
Prove both tf-poc HCP assumptions and the retained POC import rehearsal before touching the live-role ownership boundary.
-
In a separately approved administrator/CDK migration, tag the existing HCP apply roles first:
hcptf-shoc-backend-devgetsHcpTerraformWorkspace=shoc-backend-dev, andhcptf-shoc-backend-staginggetsHcpTerraformWorkspace=shoc-backend-staging. Next attachshoc-backend-dev-deploy-boundaryandshoc-backend-staging-deploy-boundaryto the exactgithubdeploy-*roles, and attachshoc-backend-dev-runtime-boundary/shoc-backend-staging-runtime-boundaryto the exact runtime roles. Verify the boundary ceilings before adding the matching manager tag to either targetgithubdeploy-*role. Verify each effective deployment action before continuing. HCP remains blocked while a target tag is missing/different or the target lacks its exact dedicated boundary, so a partial migration cannot authorize policy writes. Complete both runtime/deploy boundary attachments before workload imports. HCP apply roles deliberately have noiam:PutRolePermissionsBoundaryor boundary-policy mutation permissions. -
In the backend bootstrap, add Terraform
removedblocks withdestroy = falsefor only the four dev/staging HCP roles and their inline policies. Apply and verify Terraform state no longer owns them while all four physical roles and ARNs remain unchanged. -
Set both contexts to
true, synthesize withnpx cdk synth terraform-substrate-external-dev, and create a CloudFormation IMPORT change set for the four existingAWS::IAM::Roleresources by exact role name. Do not run a normal CREATE/UPDATE change set for this ownership transition. The POC gate must remain true so the already-managed pair stays in the template. Import records ownership; it does not update existing role properties or inline policies. -
Run a separate, reviewed CloudFormation reconcile update after import and before switching workspace credentials. Each current live role has one inline policy:
shoc-backend-dev-import-plan,shoc-backend-dev-import-apply,shoc-backend-staging-import-plan, orshoc-backend-staging-import-apply. Existing descriptions and tags are inventoried in the backend handoff. Reconcile those explicit differences to the final baseline shape without replacing a role. -
After reconcile and HCP assumption proof, keep both context values committed as
true. Every subsequent normal deployment must synthesize all six roles. Never return either gate to false as a rollback mechanism; Retain protects the physical role but removing it from the stack abandons CloudFormation ownership. -
Only after all imports/reconciliation complete and both context defaults are permanently
true, addterraform-substrate-external-devback to the external-dev workflow stack selector. Until then all substrate operations are deliberate manual deploy/import actions. -
Retire the backend bootstrap only after the POC pair and all four imported live roles are proven under this stack. Role deletion/recreation is never a migration step.
The external-dev apply roles intentionally omit role create/delete,
managed-policy attach/detach, boundary mutation, iam:PassRole, and
secret-value APIs. IAM writes are limited to exact-role inline-policy and
ordinary tag updates plus exact-profile tags; role descriptions remain stable
and HCP receives no UpdateRole or UpdateRoleDescription. Live SHOC backend
apply roles may UpdateAssumeRolePolicy only on the matching
githubdeploy-shoc-backend-{dev,staging} role. Live SHOC frontend apply roles
may UpdateAssumeRolePolicy only on the matching
githubdeploy-shoc-frontend-new-{dev,staging} role and may
ssm:PutParameter (plus tag add/remove) on
/shoc-frontend-new/{dev,staging}/deploy/*. The OU SCP
external-dev-iam-guardrails no longer denies that action; the
account-attached SCP external-dev-shoc-backend-deploy-trust restores the
deny for every other githubdeploy-* and hcptf-* trust update. The SCP
permits only the three enumerated HCP apply roles to mutate a githubdeploy-*
inline policy whose locked HcpTerraformWorkspace resource tag equals the
caller's immutable principal tag. Adding or changing that manager tag remains
administrator/CDK only. POC DNS and certificate access is tag/name constrained because their
physical IDs are allocated by the temporary retained CDK stack before
Terraform imports them. Dev and staging DNS writes are pinned to their existing
hosted-zone IDs and API record names.
Current compact policy-document sizes are 1,387 / 2,055 / 2,030 characters for the POC/dev/staging deploy boundaries and 1,176 / 1,779 / 1,656 for their runtime boundaries, each below IAM's 6,144-character managed-policy limit. The external-dev IAM guardrail SCP is 4,968 compact characters against its 5,120-character Organizations limit. The account-attached SHOC backend deploy trust SCP is 2,189 compact characters. Keep size assertions in every change.
External-dev SHOC frontend adoption uses separate gates and creates its
boundaries first. The two live retained boundaries are
shoc-frontend-new-{dev,staging}-deploy-boundary. Each permits only
bucket location/list/version reads, object get/put/current and version delete,
GetDistribution/GetDistributionConfig, invalidation create/read for
one exact distribution, and GetParameter/GetParameters on
/shoc-frontend-new/<env>/deploy/*. Dev is pinned to E2CWLM1AFB964P; staging is pinned
to E2JDVEZ6EGD49J. SCP ProtectDeploymentPrincipalLifecycle still denies
iam:UpdateRoleDescription on githubdeploy-* for HCP apply roles; the
frontend Terraform role must ignore description drift. The frontend tf-poc rehearsal is retired: its site,
HCP workspace, and GitHub deploy role are gone. enableShocFrontendPocRoles
is false and the five shocFrontendPoc* identifiers are empty, so
ShouldManageShocFrontendPocRoles stays false. After the stack update,
delete the retained orphans hcptf-shoc-frontend-new-tf-poc,
hcptf-shoc-frontend-new-tf-poc-plan, and
shoc-frontend-new-tf-poc-deploy-boundary with
OrganizationAccountAccessRole. A later rehearsal must
inventory new identifiers before that gate is turned on.
The frontend role transition is manual and is not part of the external-dev CD job:
- Deploy
terraform-substrate-external-devwith both frontend role gates false. Verify the three boundary documents before touching any GitHub role. - For dev and staging, an administrator attaches the matching dedicated
boundary to
githubdeploy-shoc-frontend-new-<env>, then addsHcpTerraformWorkspace=shoc-frontend-new-<env>. Verify the current GitHub deployment still uploads to only the exact bucket and invalidates only the exact distribution. The SCP blocks HCP from adding or changing this manager tag itself. - Set
enableShocFrontendLiveRoles=trueand run a reviewed reconcile update. This creates the exact plan/apply pairs with 3,600-second sessions and phase-specific HCPStringEqualstrust. It does not import or replace the existing GitHub roles. - Reconcile the frontend Terraform roots with
removed { destroy = false }ownership handoff for HCP roles/boundaries where applicable. Import the existing site resources only after a no-replacement plan. Apply-role writes are limited to ordinary tags,PutRolePolicyon the exact deploy role,UpdateAssumeRolePolicyon that same role,PutParameter(plus tag add/remove) on/shoc-frontend-new/<env>/deploy/*,PutBucketPolicyon the exact bucket,UpdateDistributionon the exact distribution,CreateInvalidation/GetInvalidationon the exact distribution,GetObject/GetObjectTagging/PutObject/PutObjectTaggingon.release/current,UpdateFunctionandPublishFunctionon the exact CloudFront function, and A/AAAA changes for the exact site name with CREATE/DELETE/UPSERT conditions. - For a future tf-poc, first provision and inventory the site outside these
adoption roles. Set all five
shocFrontendPoc*identifiers from the frontend shared creator outputs while its role gate remains false, deploy and reconcile the boundary, attach it and the matching manager tag to the exact GitHub role, then setenableShocFrontendPocRoles=true. - Keep
terraform-substrate-external-devout of automatic deployment until all enabled frontend roles and target-role guardrails are proven. Do not use a false gate as rollback after CloudFormation owns a role.
The apply roles explicitly deny role lifecycle/boundary/managed-policy
changes, PassRole, secret reads, CloudFront/S3 create and
delete (including OAC mutation), and deletion of inline role or bucket
policies. iam:UpdateRoleDescription stays in that deny and in the OU SCP;
HCP cannot change githubdeploy descriptions. Live apply roles may UpdateAssumeRolePolicy only on the matching
githubdeploy-shoc-frontend-new-<env> role. DenySecretAccess still denies
Secrets Manager and KMS decrypt; SSM GetParameter/GetParameters/GetParametersByPath
are denied except /shoc-frontend-new/<env>/deploy/*, which plan and apply
may read by named GetParameter/GetParameters/ListTagsForResource. DescribeParameters
is a collection API, so it is named on * (not ssm:Get*). UpdateDistribution is allowed on the exact pinned distribution ARN.
CreateInvalidation and GetInvalidation are allowed on that same ARN so the
Terraform invalidation action can run. GetObject/GetObjectTagging/PutObject/PutObjectTagging on
.release/current lets Terraform own the release pointer, including the
tag list aws_s3_object refreshes on every plan. UpdateFunction and PublishFunction are allowed on the exact pinned function
ARN so Phase 2 ownership tags can apply; create, delete, and OAC updates stay
denied. IAM does not expose a condition key for an inline policy name, so
PutRolePolicy is constrained to the exact target-role ARN and requires the
exact dedicated deploy boundary to already be attached. The boundary limits
effective permissions, and the SCP requires the target role's locked
HcpTerraformWorkspace tag to equal the apply role's principal tag. The
Terraform resource must retain the inventoried inline policy name.
HCP Terraform layout (org-level setup, console): one org seahaven
(free tier: 500 managed resources, 1 concurrent run); one HCP project per
AWS account (seahaven-prod, seahaven-dev); one workspace per stack
(<stack>-<env>, one state file = one blast radius). Dedicated iam-bootstrap-<env> workspace in each prod/dev project
(iam-bootstrap-prod, iam-bootstrap-dev; HCP workspace names are
org-unique). Manual apply only.
Default execution mode Remote. Never use HCP's "Quick setup AWS dynamic
credentials" button — it writes the single TFC_AWS_RUN_ROLE_ARN, which
collapses the plan/apply role split. Never project-scoped variable sets.
VCS file triggers (PLAT-183). Keep file-triggers-enabled. If the plan
packages Lambda source or otherwise reads files outside the working
directory, set trigger-prefixes or trigger-patterns to those paths.
trigger-prefixes add to the working directory; trigger-patterns replace
it and must include the working-directory glob. Do not disable file triggers
to make source-only merges queue; docs-only commits must not apply prod.
Record the live prefixes or patterns in the app README next to the workspace
name. Checklist: engineering-handbook
aws-infrastructure.md.
Audit: python3 scripts/check_hcp_workspace_triggers.py (org seahaven,
projects seahaven-mgmt, seahaven-prod, seahaven-dev).
Two-principal model (prod/dev HCP, PLAT-149). Replace enumerated
StringEquals on iam:PermissionsBoundary with a factory vs scoped split.
- General apply role
hcptf-bootstrap: trust pinned toorganization:seahaven:project:seahaven-<env>:workspace:iam-bootstrap-<env>:run_phase:apply(exactStringEqualsonaudandsub; neverStringLikeonrun_phaseor workspace). IAM writes onrole/tf-managed/*andpolicy/tf-managed/*.CreatePolicy/CreatePolicyVersiononly here.CreateRole/PutRolePolicy/AttachRolePolicy/PutRolePermissionsBoundaryontf-managedroles requireiam:PermissionsBoundaryStringLikepolicy/tf-managed/*orpolicy/seahaven-lambda-execution-boundary*.PassRoleontf-managedroles is allowed tolambda.amazonaws.com,ecs-tasks.amazonaws.com, andscheduler.amazonaws.comso a first apply can create Fargate tasks and EventBridge Scheduler targets.Nullfalse is not enough: it would acceptAdministratorAccessas the ceiling. Must not mutategithubdeploy-*,github-cfn-execution-role,cdk-hnb659fds-*,OrganizationAccountAccessRole,seahaven-*. SCPprotect-privileged-rolescovershcptf-bootstrap*in prod and nonprod (OAA and CDK may still update trust). Manual apply. Not in external-dev. - General plan role
hcptf-bootstrap-plan:ViewOnlyAccessplus a refresh sidecar. NeverReadOnlyAccess. Never IAM writes. Trustrun_phase:planon the matchingiam-bootstrap-<env>workspace. - Scoped apply role
hcptf-<stack>: trust pinned to that stack's workspace andrun_phase:apply. Noseahaven-hcptf-iam-management. May managerole/tf-managed/<prefix>-*withiam:PermissionsBoundaryStringLikethis stack'spolicy/tf-managed/<prefix>*, the sharedseahaven-lambda-execution-boundary, orseahaven-lambda-execution-boundary-<stack>. May notCreatePolicy/CreatePolicyVersion. May notUpdateAssumeRolePolicy,DeleteRole, orPutRolePolicyonhcptf-*(including itself). Need more apply-role permission later:--allow-workspace <stack>-<env>on the create script, point that workspace'sTFC_AWS_*athcptf-bootstrap, apply, retarget vars, re-run the script with no extra workspace. Do not leave a stack workspace on bootstrap trust. - Scoped plan role:
ViewOnlyAccessplus a scoped refresh sidecar.
This does not re-open PLAT-52 AC1. On the HCP path, CreatePolicy lives only
on hcptf-bootstrap, which is not githubdeploy-seahaven-org-baseline.
Prefix matching is StringLike on the bootstrap/scoped HCP documents, not
on the SAM guardrail. Do not add ArnLike to deploy-substrate.
First-apply runbook (new prod/dev HCP stack):
- Freeze the app's SAM/CDK CD. Secrets first (ARN references only in state).
- HCP workspace
<stack>-<env>exists. Auto-apply off. Vars still empty. IAM lives in this workspace's state, so this workspace must assumehcptf-bootstrapfor the first apply. Default bootstrap trust does not include it. Never a project-scoped variable set. Enable VCS file triggers before the first merge to the tracked branch. Ifterraform/(or the working directory) packagessrc/,functions/,lambda/, orlambdas/, list those paths intrigger-prefixesortrigger-patternsand write them in the app README next to the workspace name. See engineering-handbook HCP Terraform VCS file triggers. scripts/create-hcptf-bootstrap-roles.sh --account prod|dev --allow-workspace <stack>-<env>(OAA). Trust stays exactStringEqualsoniam-bootstrap-<env>plus this one workspace. NeverStringLike.- Point this workspace's
TFC_AWS_APPLY_ROLE_ARN/TFC_AWS_PLAN_ROLE_ARNathcptf-bootstrap/hcptf-bootstrap-plan. - App PR adds
aws_iam_roleplan/apply (trust exactStringEquals), a boundary atpath = "/tf-managed/", and exec roles with that boundary. Copyexamples/hcptf-workspace-iam/hcp_iam.tf.example. Cross-family review +/sh-security-reviewstill apply to IAM in the app PR. There is no org-baseline IAM PR for the new stack. - One Manual apply. Roles and boundary exist (tolerate partial state on non-IAM resources).
- Switch workspace vars to the new scoped ARNs. Re-run the create script
with no
--allow-workspaceso trust isiam-bootstrap-<env>only again. - Second Manual apply with the scoped role. Prove the stack. Then seal.
- Live-path proof, cutover, docs as before.
Plan role: ViewOnlyAccess (never ReadOnlyAccess) plus a scoped
plan-refresh sidecar. Apply role: scoped IAM statements from
lib/hcptf-bootstrap/scoped-apply-iam.json.tmpl plus prefix-scoped service
wildcards. Do not enumerate provider Get* APIs.
Import runbook (existing eight prod stacks, PLAT-146). Import, do not
recreate. Role names and TFC_AWS_*_ROLE_ARN stay the same. Pilot is
afi-backup-monitor only; do not batch the remaining seven.
Repos that must change: afi-backup-monitor, front-integrations,
paychex-integrations, sh-openswe-traces, procurement-ingest,
seahaven-site, meal-order-manager, seahaven-door-unlock-api.
Repos that do not change: SHOC (shoc-backend, shoc-frontend-new),
remaining SAM / unmigrated stacks.
Each PR adds aws_iam_role / inline policy resources matching live names,
terraform import (see examples/hcptf-workspace-iam/hcp_iam.tf.example),
and drops the attached seahaven-hcptf-iam-management once the scoped
statements live on the role. Role names stay the same. The import apply
cannot run as hcptf-<stack>: live DenySelfMutation denies
DetachRolePolicy / PutRolePolicy on hcptf-*. Use the same
--allow-workspace window as first-apply, point TFC_AWS_* at bootstrap
for that one Manual apply, then retarget the original scoped ARNs and
revoke the extra trust. Lambda permissions_boundary may keep pointing
at seahaven-lambda-execution-boundary-<stack> in deploy-substrate for this
pass.
Prod/dev substrate delete (PLAT-147). After all eight imports:
- Inventory
seahaven-hcptf-iam-managementattachments (scripts/delete-terraform-substrate-prod-dev.sh --account prod --inventory). None may remain. - Remove the eight prod role pairs from the template (they already have
DeletionPolicy: Retain) so CloudFormation forgets them without deleting. - Remove
terraform-substrate-prodandterraform-substrate-devfrombin/app.tsand.github/workflows/deploy.yaml. Keepterraform-substrate-external-dev. - Delete stacks in
011934824531and710827005802only. OIDC is Retain.seahaven-hcptf-iam-managementdeletes with those stacks. Do not deleteterraform-substrate-external-dev. Do not strip SHOC resources from the shared YAML while that stack still synthesizes them.
HCP-side authority is AWS authority. AWS exposes only aud, sub and
amr as trust-policy condition keys for a generic OIDC provider — HCP's
immutable terraform_workspace_id / terraform_project_id claims are not
usable in an IAM condition. The sub pin therefore rests on HCP display
names, so whoever can create, rename, move or delete a workspace in the
seahaven-prod project effectively holds prod deploy authority. Restrict that
HCP team permission to the same people, and when a workspace is retired, delete
its hcptf-* roles in the same change so a reused name cannot inherit them.
Treat "still using the bootstrap ARN" as a defect; check HCP var sets after
each migration.
Terraform state is secret-bearing. HCP-hosted state records sensitive attributes in full and lives outside the AWS accounts, readable by any HCP principal with workspace read. Secrets stay in Secrets Manager / SSM and are referenced by ARN: do not manage secret values in Terraform.
Rollback of terraform-substrate: inventory attachments first. Deleting
prod/dev while any hcptf-* still attaches seahaven-hcptf-iam-management
fails or strips the backstops. Only the provider is Retain. The
guardrail policy is deleted with the stack. After PLAT-147 the standing
control for hcptf-bootstrap* is the prod/nonprod SCP, and scoped apply
roles carry their own DenySelfMutation inline. Do not delete
terraform-substrate-external-dev. Workspaces holding state must be migrated
or destroyed HCP-side first; deleting the OIDC provider strands them mid-run.
First-create rollback trap. The provider is Retain, so if any other
resource in this stack fails on first create, CloudFormation rolls back, the
provider survives untracked, and the stack lands in ROLLBACK_COMPLETE — which
cannot be updated, and cannot be recreated because an account holds exactly one
provider per URL. Recovery: delete the stack, then either remove the orphaned
provider with aws iam delete-open-id-connect-provider before retrying, or
redeploy with createOidcProvider: false. Note cd-cdk's pre-flight and
health check probe only the job's single stack-name input (the account
baseline), so a wedged substrate stack does not show up there — check it
directly.
CloudTrail (audit finding C-1)
| Resource | Logical ID | Notes |
|---|---|---|
| Multi-region trail | Trail (seahaven-org-trail) |
Management events read+write, global service events, log-file validation on, CloudTrail Insights on (ApiCallRate + ApiErrorRate, §37) |
| Log bucket | TrailLogBucket (seahaven-cloudtrail-logs-328440206208) |
Private (Block Public Access all), SSE-KMS, versioned, TLS-only, Object Lock GOVERNANCE 365d, lifecycle (Glacier @90d, expire @365d), server access logging → seahaven-s3-access-logs |
| KMS CMK | TrailKey (alias/cloudtrail-logs) |
Encrypts log files; automatic rotation enabled |
| CloudWatch Logs group | created by the L2 Trail |
365-day retention; this is the group the CIS Section 4 metric filters (H-1) attach to |
Data flow: API activity across all regions → CloudTrail → (a) KMS-encrypted, Object-Locked S3 bucket for durable/tamper-resistant storage and (b) CloudWatch Logs for real-time querying and metric-filter alarms.
Compliance impact: closes CIS 3.1 (multi-region trail), 3.2 (log-file validation), 3.4 (CloudWatch Logs integration), 3.6 (bucket access logging), 3.7 (KMS CMK encryption), and 3.8 (CMK rotation). Unblocks CIS Section 4 / finding H-1 (metric filters + alarms now have a log group to target).
Design decisions
- Management events only. Object-level S3/Lambda data events (CIS 3.10/3.11) are deferred to control cost; revisit with targeted S3 write data events on sensitive buckets (payments / accounting / kb) if needed.
- Object Lock GOVERNANCE, not COMPLIANCE. Tamper-resistant but still
deletable by a principal holding
s3:BypassGovernanceRetention— avoids the irreversibility of COMPLIANCE mode. Revisit if a stricter posture is required. - RETAIN on the bucket and KMS key so a stack teardown never destroys the audit trail.
AWS Backup (audit finding C-7)
Phase 1 ("critical data first") of fixing the account's complete lack of AWS Backup. Protects the data stores with no offsite leg today and copies each recovery point cross-region into a governance-locked vault.
| Resource | Logical ID | Notes |
|---|---|---|
| Primary vault | seahaven-primary (us-east-1) |
KMS-CMK encrypted, unlocked (working copy), RETAIN |
| Offsite vault | seahaven-offsite (us-west-2) |
KMS-CMK encrypted, Vault Lock GOVERNANCE (min-retention 30d, no cooling-off window), RETAIN |
| Backup plan | seahaven-critical-daily |
Daily 06:00 UTC, delete-after 35d, cross-region CopyAction → offsite (retain 90d) |
| Service role | seahaven-backup-service-role |
Backup-only (Backup + S3-Backup managed policies); restore perms intentionally deferred |
Phase-1 scope (selected by explicit ARN, not tags, to avoid drifting other
stacks): RDS proposal-system-db, DynamoDB PaymentsDashboard,
DynamoDB purchase-orders, S3 accounting.seahaven.com,
seahaven-payments-csv-328440206208, google-workspace-seahavenind.com.
(RDS database-1 was originally in this set but was retired 2026-06-03 —
audit H-19, idle 0 conn/60d — and removed from the selection; its final
encrypted recovery point is retained in seahaven-offsite for 7 years.)
Coexists with existing EBS DLM snapshots and DynamoDB PITR — it supplements them with the missing offsite + immutable leg; it does not replace them.
Design decisions:
- Governance lock first, not compliance. Recovery points can't be silently
deleted, but a principal with explicit permission can still intervene while
we validate. Graduate to COMPLIANCE (irreversible) later by adding
changeableForto the offsite vault lock + redeploy. - Backup-only role. Restore policies and
allowRestoresare not granted; restores get a separate audited path once a restore-test process exists.
Pre-deploy gates (must clear before the first scheduled run):
- Enable S3 versioning on
seahaven-payments-csv-328440206208andgoogle-workspace-seahavenind.com(accounting.seahaven.comalready has it, audit C-9), or their jobs fail silently (folds in H-21). database-1is unencrypted (H-19): smoke-test an on-demand backup + copy of it to us-west-2 first; if the copy fails, encrypt it or drop it from the copy.- Enable DynamoDB PITR (H-7) on the two tables for between-window recovery.
AWS Backup phase 2 (audit Day 4)
Expands the same seahaven-critical-daily plan to every remaining data store, so
all of DynamoDB + EBS get the offsite + immutable leg ("offsite for everything").
| Resource | Logical ID | Notes |
|---|---|---|
| Phase-2 selection | Plan/Phase2Resources (phase2-offsite-everything) |
Same plan, same seahaven-backup-service-role, same daily + cross-region copy rule |
Phase-2 scope: the 15 remaining DynamoDB tables (all except the two phase-1 financial tables + the deleted ledgerflow tables) and all 9 in-use EBS volumes, again by explicit ARN — tag-based selection was deliberately avoided because the file-share volumes are standalone-managed and the tables are owned by other stacks, so tagging here would drift them.
No IAM change: AWSBackupServiceRolePolicyForBackup already grants the
DynamoDB/RDS/EBS backup actions, so phase 2 reuses the phase-1 role unchanged
(cross-reviewed, no BLOCK).
Known tradeoff (→ Jira INFRA-31): explicit-ARN EBS entries go stale if a volume is replaced (new volume id), silently dropping it from backup. Migrating the EBS portion to tag-based selection (with the tag codified in each owning stack) is the resilient follow-up; scheduled drift detection is the interim backstop.
Also enabled outside this stack (audit H-7, via CLI — codify per stack →
INFRA-30): PITR + DeletionProtectionEnabled on 12 more DynamoDB tables
(account-wide PITR now 19/21).
Detective controls + budget (audit Day 1)
Account-level detective layer, in lib/detective-controls.ts, plus the cost
budget in lib/governance-toggles.ts. Scope is us-east-1 only (all workloads
live here); multi-region coverage is a follow-up.
| Resource | Logical ID | Finding | Notes |
|---|---|---|---|
| Config delivery bucket | seahaven-config-328440206208 |
H-2 | Private (BPA all), SSE-S3, versioned, TLS-only, 365d lifecycle |
| Config recorder role | seahaven-config-recorder-role |
H-2 | AWS_ConfigRole + scoped S3 delivery; IAM cross-reviewed |
| Config recorder + channel | DetectiveControls/ConfigPutRecorder, ConfigPutChannel, ConfigStartRecorder |
H-2 | AwsCustomResource calls PutConfigurationRecorder → PutDeliveryChannel → StartConfigurationRecorder in sequence; idempotent upsert avoids the L1 CFN deadlock (INFRA-17). Custom-resource role cross-reviewed. |
| GuardDuty detector | DetectiveControls/GuardDutyDetector |
H-3 | Findings every 15 min |
| Security Hub | DetectiveControls/SecurityHub |
H-4 | FSBP v1.0.0 + CIS v3.0.0; controls evaluate once Config is recording |
| Access Analyzer | seahaven-account-analyzer |
M-5 | ACCOUNT external-access analyzer (free) |
| Monthly budget | GovernanceToggles/MonthlyCostBudget (seahaven-monthly-cost) |
M-10 | $1,200/mo, 80%/100% actual + 100% forecast → adam@seahavenind.com |
Config recorder + delivery channel are managed by AwsCustomResource (INFRA-17).
The L1 AWS::Config::ConfigurationRecorder deadlocks the stack (recorder never
reaches CREATE_COMPLETE without a delivery channel; channel can't be created
without a recorder — hit 2026-06-01). The custom resource sidesteps this by
calling the Config SDK directly: Put* is an upsert, so the deploy adopts the
existing CLI-created recorder and channel without destroying them. Active
recording is never interrupted.
CLI-applied governance toggles (no CloudFormation resource)
These account toggles have no native CloudFormation resource, so they are applied via CLI and recorded here — per account (they are account-scoped; a new member account has NONE of them until applied). Applied: 328440206208 (2026-06-01); 001520130573, 710827005802, 011934824531 (2026-07-14 — EBS encryption-by-default + password policy, verified per account; Inspector2 via delegated admin; cost-allocation tags are org-level).
# M-3 EBS encryption-by-default (new volumes; existing 5 plaintext volumes are H-19-adjacent)
aws ec2 enable-ebs-encryption-by-default --region us-east-1
# M-6 Inspector2 (EC2 + Lambda + ECR)
aws inspector2 enable --resource-types EC2 LAMBDA ECR --region us-east-1
# M-7 IAM password policy (CIS 1.8/1.9): >=14 chars, full complexity, no reuse of last 24
aws iam update-account-password-policy \
--minimum-password-length 14 \
--require-symbols --require-numbers \
--require-uppercase-characters --require-lowercase-characters \
--allow-users-to-change-password --password-reuse-prevention 24
# M-11 Activate cost-allocation tags (only activates keys already seen on resources)
aws ce update-cost-allocation-tags-status --cost-allocation-tags-status \
'TagKey=Project,Status=Active' 'TagKey=Owner,Status=Active' 'TagKey=Environment,Status=Active'
Centralized root access management (org-level, no CloudFormation resource)
STATUS: ENABLED 2026-07-14, all member root credentials DELETED (evidence:
~/Documents/repositories/_audits/centralized-root-access-evidence-2026-07-14.md).
Member accounts have NO root credentials; the only root path is a privileged
session from the management account. The management account's own root is NOT
centrally manageable and stays password+MFA hardened.
# Enable (mgmt account). ORDER MATTERS: trusted access must be enabled
# explicitly first — enable-organizations-root-credentials-management does
# NOT auto-enable it (fails ServiceAccessNotEnabledException).
aws organizations enable-aws-service-access --service-principal iam.amazonaws.com
aws iam enable-organizations-root-credentials-management
aws iam enable-organizations-root-sessions
# Periodic verification (add to governance checks): expect BOTH features
aws iam list-organizations-features
Audit / delete member root credentials (task-scoped root sessions, 15-min):
aws sts assume-root --target-principal <acct> \
--task-policy-arn arn=arn:aws:iam::aws:policy/root-task/IAMAuditRootUserCredentials
# then, with the session creds (no --user-name; root has none):
# get-login-profile / list-mfa-devices / list-access-keys / list-signing-certificates
aws sts assume-root --target-principal <acct> \
--task-policy-arn arn=arn:aws:iam::aws:policy/root-task/IAMDeleteRootUserCredentials
# delete-login-profile; deactivate-mfa-device --serial-number <arn>
# GOTCHA: the delete task policy explicitly DENIES iam:DeleteVirtualMFADevice —
# remove the orphaned virtual-device OBJECT via OrganizationAccountAccessRole.
# DONE = four surfaces clear: login profile NoSuchEntity, MFA/keys/certs all empty.
Root recovery runbook (proven by drill on prod 2026-07-14):
deny-root-user(p-2idoxozz) DENIES root sessions in every covered OU (SCPs evaluatests:AssumeRootsessions — the principal is the member root ARN). Recovery therefore starts with a manual, temporary detach (aws organizations detach-policy— NOT a CDK deploy), timeboxed minutes.- GOTCHA (inheritance): p-2idoxozz is attached to
workloadsAND its child OUs — for an account under workloads/, detach from BOTH the child OU and workloads, or the inherited deny still applies. Allow ~10s propagation. - Freeze deploys of
seahaven-org-governancefor the window (a concurrent deploy would re-attach mid-recovery); verify no CD run in flight first. aws sts assume-root --target-principal <acct> --task-policy-arn arn=arn:aws:iam::aws:policy/root-task/IAMCreateRootUserPassword→create-login-profile(no args) restores a login profile.- Do the root-only task, DELETE the credentials again (four-surface verify),
reattach the SCP(s), confirm
list-targets-for-policymatches the pre-detach capture and stack drift is IN_SYNC. - extdev extra:
external-dev-iam-guardrailsalso deniesiam:CreateLoginProfile— recovery there needs that SCP temporarily detached too. The extdev OU sits at the 5-SCP hard quota: any new guardrail for extdev must attach at the ACCOUNT (396287094661) or consolidate into an existing policy.external-dev-shoc-backend-deploy-trustis attached to that account, not the OU.
New-account flow (supersedes root-harden-before-OU-move): create the
account at the org ROOT → it has no root credentials from birth (verify with
the audit session) → bootstrap + deploy role + baseline → move-account into
the target OU → verify SCP inheritance + region-lock canary. No mailbox or
MFA enrollment step. Root-usage monitoring: GuardDuty
Policy:IAMUser/RootCredentialUsage + CIS 4.3 alarm remain active.
Delegated security administration (Phase 3, no CloudFormation resource)
Account seahaven-security (001520130573) is the org's delegated administrator for the detective services. STATUS: APPLIED 2026-07-14, verified (see evidence below). The hard preconditions were enforced before the first delegation call (security review SEC-BASE-B/D — never delegate to an account with unhardened root or before its baseline stack exists):
- Baseline stack
UPDATE_COMPLETE; rootAccountMFAEnabled: 1; account parentou-nbuj-v0s9630uwith SCPs deny-root-user + protect-security-baseline + security-guardrails inherited.
Verification evidence (2026-07-14):
organizations list-delegated-administrators→001520130573(all five service principals registered).- GuardDuty:
AutoEnableOrganizationMembers: ALL; members 328440206208 + 396287094661 bothEnabled. - Security Hub: org auto-enable on; both members
Enabled. - Org Access Analyzer
seahaven-org-analyzercreated; Config org aggregatorseahaven-org-aggregator(AllAwsRegions) on the Config SLR; Inspector2 auto-enable ec2/ecr/lambda + both members associated. - Findings flow verified end-to-end: GuardDuty sample findings created in
member 396287094661 were listed and fully readable from the admin detector
in 001520130573 (
Recon:EC2/PortProbeUnprotectedPort, AccountId 396287094661, sample=true), then archived.
The delegation runbook (all calls idempotent, run from the management account):
# GuardDuty: delegate + auto-enable all org members (adopts existing detectors)
aws guardduty enable-organization-admin-account --admin-account-id 001520130573
# then AS 001520130573: update-organization-configuration --auto-enable-organization-members ALL
# + create-members for pre-existing accounts (mgmt, external-dev)
# Security Hub: delegate + auto-enable new members
aws securityhub enable-organization-admin-account --admin-account-id 001520130573
# then AS 001520130573: update-organization-configuration --auto-enable
# IAM Access Analyzer: delegate + ORGANIZATION-scoped analyzer
aws organizations register-delegated-administrator \
--account-id 001520130573 --service-principal access-analyzer.amazonaws.com
# then AS 001520130573: create-analyzer --type ORGANIZATION
# Config: delegate the aggregator (recorders stay per-account in the baselines;
# the aggregator's recorder-status view is the drift detector)
aws organizations register-delegated-administrator \
--account-id 001520130573 --service-principal config.amazonaws.com
# then AS 001520130573: put-configuration-aggregator --organization-aggregation-source
# Inspector2: delegate + associate members
aws inspector2 enable-delegated-admin-account --delegated-admin-account-id 001520130573
ONLY once delegation is live AND auto-enrollment is verified (a new member shows enrolled in the security account's GuardDuty/Security Hub consoles): new member accounts are then detected/enrolled automatically, and future member baselines can drop per-account GuardDuty/SecurityHub resources. Until then, every member baseline MUST keep them (slimming the existing member stacks is a separate, verification-gated change; note the DA account's own CFN-owned detector/hub become co-managed after delegation — never rename/remove them via CFN while the account is delegated admin).
Accepted read-surface note (SEC-BASE-I): the org Config aggregator + ORGANIZATION Access Analyzer give principals in 001520130573 org-wide READ of resource configurations (including recorded Lambda env vars) and IAM policies. Main-branch write access to this repo therefore implies that read surface — verify no prod Lambda keeps secrets in env vars before creating the aggregator, and keep branch protection tight.
L-8 (billing-metrics preference) is OUTSTANDING — console only. Enabling the
CloudWatch EstimatedCharges metric in us-east-1 requires turning on Receive
Billing Alerts under Billing → Billing preferences; there is no public API/CLI.
The M-10 budget (seahaven-monthly-cost, 80%/100% actual + 100% forecast)
provides cost alerting independent of that metric.
The legacy, manually-created
AWS-MonthlyBillingCloudWatch alarm ($50 threshold onEstimatedCharges, routed tosite-alerts) was deleted 2026-07-07 as unmanaged drift: it was fully redundant with the M-10 budget, sat permanently in ALARM (spend has far exceeded $50/mo), and was never in IaC. Billing alerting is now solely the managed M-10 budget. To restore the old alarm if ever needed:aws cloudwatch put-metric-alarm --alarm-name AWS-MonthlyBilling --namespace AWS/Billing --metric-name EstimatedCharges --dimensions Name=Currency,Value=USD --statistic Maximum --period 86400 --evaluation-periods 1 --threshold 50 --comparison-operator GreaterThanThreshold --alarm-actions arn:aws:sns:us-east-1:328440206208:site-alerts.
Monitoring + logging (audit Day 2)
| Resource | Logical ID | Finding | Notes |
|---|---|---|---|
| CIS metric filters + alarms | CisMonitoring/* |
H-1 | 15 filters (CIS 4.1–4.15) on the CloudTrail log group, each with an alarm → seahaven-cis-alarms. ALARM-only actions (no OK). 4.16 = Security Hub (Day 1) |
| CIS alarm topic | seahaven-cis-alarms |
H-1 | SNS, SSE (alias/aws/sns), email sub to adam@seahavenind.com |
| VPC flow logs | FlowLogs/FlowLog0..4 |
H-14 | ALL traffic on all 5 VPCs → S3 |
| Flow-logs bucket | seahaven-vpc-flow-logs-328440206208 |
H-14 | Private, SSE-S3, TLS-only, Glacier @90d / expire @365d; delivery bucket policy cross-reviewed |
| SES config set | seahaven-email-events |
M-13 | Bounce/complaint/reject → CloudWatch metrics for reputation visibility |
| Sensitive-logs CMK | LogsKey/Key (alias/seahaven-logs) |
M-24 | Encrypts sensitive CloudWatch Logs groups. Key policy grants logs.us-east-1.amazonaws.com Encrypt*/Decrypt*/ReEncrypt*/GenerateDataKey*/DescribeKey scoped by kms:EncryptionContext:aws:logs:arn (required or log delivery breaks). Rotation on, RETAIN. Applied in place to TrailLogGroup via escape hatch (same logical id/name). Cross-reviewed |
M-24 sensitive log groups: alias/seahaven-logs encrypts the CloudTrail CW
log group (codified here) plus the finance/PII Lambda groups owned by other
stacks — exec-aide-*, payments-*, po-email-processor, vendor-reply-processor
— which are associated via aws logs associate-kms-key and tracked as drift to
codify in their owning repos. The CloudTrail group is encrypted in place (escape
hatch on the existing AWS::Logs::LogGroup) so it is additive: same logical id +
physical name, no replacement, CIS Section-4 metric filters keep working. A
context flag encryptTrailLogGroup (default true) allows rolling the CMK out
and smoke-testing it on a low-risk Lambda group before the CloudTrail group:
# Phase 1: deploy CMK only, validate on a low-risk group
cdk deploy account-baseline --context encryptTrailLogGroup=false
# Phase 2: encrypt the CloudTrail group (default)
cdk deploy account-baseline
H-1 log group: the metric filters attach to the existing CloudTrail
CloudWatch Logs group by name (seahaven-account-baseline-TrailLogGroup4CBE3AF5-…),
imported read-only so the live audit trail is never replaced. Stable unless the
Trail is recreated.
H-14 bucket policy note: the flow-logs delivery policy keeps
s3:x-amz-acl=bucket-owner-full-control and the arn:aws:logs:…:* source-ARN
wildcard — both are required by AWS's documented flow-logs-to-S3 policy
(flow-logs-s3-permissions.html). A cross-review suggested dropping them; that
was rejected as it would break delivery. s3:ListBucket was dropped (not needed).
M-13 follow-up: associate seahaven-email-events as the default config set
on the live sending identities to capture events from existing senders:
aws sesv2 put-email-identity-configuration-set-attributes \
--email-identity int.seahaven.com --configuration-set-name seahaven-email-events
Log-group retention + alarm wiring (audit L-4, L-5)
Applied via CLI (auto-created groups spread across stacks; one alarm in another stack). Applied 2026-06-02.
# L-4 90-day retention on the 13 never-expire log groups (CodeBuild + CDK helpers)
for lg in <the 13 groups>; do aws logs put-retention-policy --log-group-name "$lg" --retention-in-days 90; done
# L-5 wire the actionless forgejo backup-verification alarm to site-alerts
aws cloudwatch put-metric-alarm --alarm-name forgejo-backup-verification-errors \
--alarm-actions arn:aws:sns:us-east-1:328440206208:site-alerts # (preserve existing alarm config)
Roadmap (same stack)
Detective layer multi-region expansion (GuardDuty/Config/Security Hub beyond us-east-1). Backup: phase 2 is deployed (see above); remaining is migrating the phase-2 EBS entries to tag-based selection (INFRA-31) and graduating the offsite vault to compliance mode.
Deploy
CI/CD via the org reusable workflows (ci-typescript-cdk.yaml,
cd-cdk.yaml); pushes to main deploy through the OIDC role in
secrets.AWS_DEPLOY_ROLE_ARN. Local: npm ci && npm run build && npx cdk diff.
npx cdk deploy seahaven-account-baseline
Verify
aws cloudtrail get-trail-status --name seahaven-org-trail # IsLogging: true
aws cloudtrail describe-trails --trail-name-list seahaven-org-trail
aws cloudtrail validate-logs --trail-arn <arn> --start-time <t> # digest integrity
AWS Backup (C-7):
aws backup list-backup-vaults # seahaven-primary
aws backup list-backup-vaults --region us-west-2 # seahaven-offsite
aws backup describe-backup-vault --backup-vault-name seahaven-offsite --region us-west-2 # Locked, MinRetentionDays
aws backup get-backup-plan --backup-plan-id <id> # daily rule + CopyAction
# Smoke test: on-demand backup of one resource, then confirm the cross-region copy lands
aws backup start-backup-job --backup-vault-name seahaven-primary \
--resource-arn arn:aws:rds:us-east-1:328440206208:db:proposal-system-db \
--iam-role-arn arn:aws:iam::328440206208:role/seahaven-backup-service-role
aws backup list-copy-jobs --region us-west-2 # copy to offsite present + COMPLETED
# Phase-2 selections live on the plan:
aws backup list-backup-selections --backup-plan-id <id> --query 'BackupSelectionsList[].SelectionName' # critical-data + phase2-offsite-everything
Detective layer + governance (Day 1):
aws configservice describe-configuration-recorder-status # recording: true
aws guardduty list-detectors # one detector id
aws securityhub get-enabled-standards # FSBP + CIS v3.0.0
aws accessanalyzer list-analyzers # seahaven-account-analyzer ACTIVE
aws inspector2 batch-get-account-status --region us-east-1 # ec2/ecr/lambda ENABLED
aws iam get-account-password-policy # length 14, reuse 24
aws ec2 get-ebs-encryption-by-default --region us-east-1 # EbsEncryptionByDefault: true
aws budgets describe-budgets --account-id 328440206208 # seahaven-monthly-cost $1,200
aws ce list-cost-allocation-tags --status Active # Project/Owner/Environment Active