sh-mcp/docs/runbook-auth-deploy.md

# Runbook — deploy `sh-mcp-auth`

First-time deploy of the Cognito auth substrate (Phase 2a). The stack is on `main`
(`infra/`, CDK app `cdk.json` → `tsx infra/bin/app.ts`) but **synth-only / not yet
deployed**. This runbook takes it from "synthesizes in CI" to "live in AWS and
validated."

- **Account / region:** `328440206208` / `us-east-1` (mgmt == prod).
- **Stack name:** `sh-mcp-auth`.
- **Owner of the manual prerequisites:** Adam (Google + IAM steps cannot be done from CDK).
- **Toolchain:** aws-cdk `2.1128.1` (CLI, root devDep), aws-cdk-lib `2.260.0`, constructs `10.6.0`.

> **CD is already wired and currently red.** `.github/workflows/deploy.yaml`
> (manual `workflow_dispatch` → org reusable `cd-cdk.yaml@main`, OIDC) has failed
> on every run so far (~4s) because the `githubdeploy-sh-mcp` deploy role does not
> exist yet (Prereq 4). Once the prerequisites below are in place, CD is triggered
> manually via the Actions "Run workflow" button on `main`; the first deploy is
> done **manually** (Step 2) so a human watches the initial resource creation.

---

## What this deploys

- Cognito user pool `sh-mcp` (FeaturePlan **ESSENTIALS** — required for the V2
  pre-token trigger; Lite silently ignores it).
- Google external OIDC IdP (`client_id`/`secret` resolved from Secrets Manager at
  deploy via a CFN dynamic reference — never inlined).
- Resource servers `sh-mcp-ops` / `sh-mcp-finance` + three app clients whose
  `AllowedOAuthScopes` are the trust-tier ceiling: `sh-agentforce-ops`,
  `-exec` (ops + gmail/calendar, **no finance**), `-finance` (`finance:read`
  only, 15-min access token, 8h refresh).
- Cognito groups `sh-mcp-ops` / `-assistant` / `-finance` / `-admin`.
- DynamoDB `sh-mcp-sync-state` + `sh-mcp-deny-list` (both `RETAIN`, logical IDs
  pinned).
- Lambdas `sh-mcp-pre-token-gen` (suppress-only, V2 trigger) and
  `sh-mcp-group-sync` (5-min EventBridge schedule), arm64, 60-day log groups.
- SNS alarm topic `sh-mcp-alarms` (CMK `alias/seahaven-alarm-topics`) + 3 alarms.

## Out of scope (deferred to Phase 2b / surface wiring)

- **No Cognito hosted-UI domain** and the app-client `callbackUrls` are a
  placeholder. The OAuth *authorization-code* flow (and therefore the Google
  redirect URI `https://<domain>/oauth2/idpresponse`) is **not exercisable until
  the surface is chosen** and a `UserPoolDomain` + real callback URLs are added.
  The substrate deploys and group-sync runs without them.
- API Gateway / WAF / server hosting / real integration clients (`jobs/`, QBO,
  Maps, DynamoDB data clients) — Phase 2b.

---

## Prerequisites (one-time, Adam)

### 1. Confirm CDK bootstrap

The account is used by other stacks, so it is almost certainly bootstrapped with
the modern (`newStyleStackSynthesis`) bootstrap. Verify:

```bash
aws cloudformation describe-stacks --stack-name CDKToolkit --region us-east-1 \
  --query 'Stacks[0].Outputs[?OutputKey==`BootstrapVersion`].OutputValue' --output text
# expect a version >= 6. If the stack is missing:  npx cdk bootstrap aws://328440206208/us-east-1
```

### 2. Google OAuth 2.0 web client → secret `sh-mcp/google-oidc`

In Google Cloud Console (the Workspace's project) → **APIs & Services →
Credentials → Create OAuth client ID → Web application**. The authorized redirect
URI is the Cognito domain callback — **deferred** (added in 2b when the hosted-UI
domain exists: `https://<cognito-domain>/oauth2/idpresponse`). Create the client
now to capture `client_id` / `client_secret`; the redirect URI can be edited later.

Store it (JSON, exact keys — read by the Google IdP `providerDetails`):

```bash
aws secretsmanager create-secret \
  --name sh-mcp/google-oidc --region us-east-1 \
  --description "Google OAuth web client for Cognito federation (sh-mcp-auth)" \
  --secret-string '{"client_id":"<CLIENT_ID>.apps.googleusercontent.com","client_secret":"<CLIENT_SECRET>"}'
```

### 3. Google Workspace service account + domain-wide delegation → secret `sh-mcp/google-directory-sa`

The group-sync Lambda reads Google Group membership via a service account with
**domain-wide delegation** impersonating an admin.

1. Create a service account (GCP) and a **JSON key**.
2. Admin console → **Security → API controls → Domain-wide delegation** → add the
   SA's client ID with scope:
   `https://www.googleapis.com/auth/admin.directory.group.readonly`
3. Pick an admin user for the SA to impersonate (the `subject`).

Store it (JSON — fields consumed by `GoogleServiceAccount` in `auth/group-sync`):

```bash
aws secretsmanager create-secret \
  --name sh-mcp/google-directory-sa --region us-east-1 \
  --description "Google Workspace SA (domain-wide delegation, Directory groups.readonly) for sh-mcp group-sync" \
  --secret-string '{
    "client_email":"sh-mcp-group-sync@<project>.iam.gserviceaccount.com",
    "private_key":"-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n",
    "subject":"admin@seahavenind.com"
  }'
```

> The stack's IAM grant is scoped to `secret:sh-mcp/google-directory-sa-*`, which
> matches the 6-char suffix Secrets Manager appends. Keep the name exactly
> `sh-mcp/google-directory-sa`.

### 4. OIDC deploy role `githubdeploy-sh-mcp`

CD assumes this role via GitHub OIDC. Add the repo to
`Sea-Haven-Industries/.github/oidc-deploy-roles.yaml` (pattern `githubdeploy-<repo>`),
trust limited to `repo:Sea-Haven-Industries/sh-mcp:ref:refs/heads/main`, with
permissions to deploy the stack (CloudFormation + the resource set: Cognito,
Lambda, DynamoDB, IAM PassRole for the Lambda roles, Logs, Events, SNS, and
`secretsmanager:GetSecretValue` on the two `sh-mcp/google-*` secrets for the
dynamic reference). This is an IAM change → run the **mandatory GPT-4.1 IAM
cross-review** on the role JSON before applying.

### 5. Google Groups exist

The four managed groups must exist as Google Groups, or group-sync's first run
errors (and correctly keeps the freshness marker stale → pre-token fails closed):

`sh-mcp-ops@seahavenind.com`, `sh-mcp-assistant@seahavenind.com`,
`sh-mcp-finance@seahavenind.com`, `sh-mcp-admin@seahavenind.com`

---

## Step 1 — synth + diff (no AWS writes)

```bash
cd ~/Documents/repositories/sh-mcp
npm ci
npx cdk synth   sh-mcp-auth
npx cdk diff    sh-mcp-auth     # against an empty account this is the full create set
```

Optional context flags (both safe to omit on the first deploy):

- `-c logsKmsArn=arn:aws:kms:us-east-1:328440206208:key/<seahaven-logs-key-id>` —
  CMK-encrypt the two Lambda log groups (else AWS-managed encryption). The
  group-sync log group records reconciliation counts only, not member emails.
- `-c callbackUrls='https://app.example.com/oauth/callback'` — overrides the
  placeholder; only meaningful once the surface domain exists (2b).

## Step 2 — first deploy (manual, watched)

```bash
npx cdk deploy sh-mcp-auth \
  -c logsKmsArn=arn:aws:kms:us-east-1:328440206208:key/<seahaven-logs-key-id> \
  --require-approval any-change      # review the IAM diff prompt before confirming
```

Expect a CMK-key + IAM-policy approval prompt (the group-sync Cognito/Secrets
grants). Note the stack outputs (exported for 2b cross-stack import):
`sh-mcp-user-pool-id`, `sh-mcp-ops-client-id`, `sh-mcp-exec-client-id`,
`sh-mcp-finance-client-id`.

> **CIS Section-4 alarms** in this account fire on manual IAM changes — expect an
> alarm notification from the role/policy creation; it is benign here.

## Step 3 — hand off to CD

After the manual first deploy succeeds and Prereq 4 is in place, deploys are
triggered manually via `.github/workflows/deploy.yaml` (Actions → deploy → Run
workflow on `main`, i.e. `workflow_dispatch`). Re-run the last failed deploy
to confirm it now goes green:

```bash
gh workflow run deploy.yaml --ref main
gh run watch $(gh run list --workflow=deploy.yaml --limit 1 --json databaseId --jq '.[0].databaseId')
```

---

## Post-deploy validation

```bash
POOL=$(aws cloudformation list-exports --region us-east-1 \
  --query "Exports[?Name=='sh-mcp-user-pool-id'].Value" --output text)

# Pool is ESSENTIALS and the V2 pre-token trigger is attached
aws cognito-idp describe-user-pool --user-pool-id "$POOL" --region us-east-1 \
  --query 'UserPool.{Tier:UserPoolTier,PreToken:LambdaConfig.PreTokenGenerationConfig}'
# expect Tier=ESSENTIALS, PreToken.LambdaVersion=V2_0

# The four managed groups exist
aws cognito-idp list-groups --user-pool-id "$POOL" --region us-east-1 \
  --query 'Groups[].GroupName'

# Finance client ceiling: finance:read ONLY, 15-min access token
aws cognito-idp describe-user-pool-client --user-pool-id "$POOL" --region us-east-1 \
  --client-id <finance-client-id> \
  --query 'UserPoolClient.{Scopes:AllowedOAuthScopes,Access:AccessTokenValidity,Refresh:RefreshTokenValidity,Units:TokenValidityUnits}'

# Group-sync ran and wrote the freshness marker (within the last 5 min)
aws lambda invoke --function-name sh-mcp-group-sync --region us-east-1 /dev/stdout | tail -1
aws dynamodb get-item --table-name sh-mcp-sync-state --region us-east-1 \
  --key '{"pk":{"S":"group-sync"}}' --query 'Item.lastSuccessfulSyncMs'
# Tail the structured logs:
aws logs tail /aws/lambda/sh-mcp-group-sync --region us-east-1 --since 10m
```

**Hard-revocation smoke test** (deny-list overlay) — add a `sub`, confirm the
pre-token Lambda strips all tier scopes on the next mint, then remove it:

```bash
aws dynamodb put-item --table-name sh-mcp-deny-list --region us-east-1 \
  --item '{"sub":{"S":"<test-sub>"},"expiresAt":{"N":"'$(($(date +%s)+600))'"}}'
# (mint a token for that user via the hosted UI once 2b adds the domain; expect base scopes only)
aws dynamodb delete-item --table-name sh-mcp-deny-list --region us-east-1 \
  --key '{"sub":{"S":"<test-sub>"}}'
```

---

## Rollback / teardown

- **Tables `RETAIN` on stack delete** — `cdk destroy sh-mcp-auth` leaves
  `sh-mcp-sync-state` and `sh-mcp-deny-list` (and their logical IDs are pinned, so
  a later redeploy re-adopts them). Delete the tables manually only if you intend
  to lose revocation/freshness state.
- A bad deploy rolls back automatically (CloudFormation). To revert code, revert
  the commit on `main`, then run the deploy workflow manually (Actions → deploy →
  Run workflow) to redeploy the prior template.
- **0a spike teardown is the LAST step, not part of this deploy.** The live 0a
  spike kit (Cognito pool `us-east-1_GsDbGe0pa`, probe Lambda/API, the SF
  `sh_mcp_0a` objects) stays up until `sh-mcp-auth` is deployed **and validated**
  — only then run the spike teardown so the proven chain isn't lost prematurely.

## Owed once deployed (per CLAUDE.md)

- Confluence **"AWS Architecture Map"** (id 1540098) — add the `sh-mcp-auth`
  Mermaid subgraph.
- Update `project_sh_mcp` memory's deployed-resources list (pool id, client ids,
  table/Lambda names) from synth-only to live.