2026-08-07 13:42:35 -04:00
|
|
|
|
# PLAT-11 cutover runbook — WO Dynamo kebab rename
|
|
|
|
|
|
|
|
|
|
|
|
Rename `WorkOrders` / `WorkOrderComments` → `work-orders` / `work-order-comments`
|
|
|
|
|
|
under HCP Terraform. SHOC is still **dev**; webhook quiet ~15–30 min is fine;
|
|
|
|
|
|
backfill after unfreeze.
|
|
|
|
|
|
|
|
|
|
|
|
## Gates
|
|
|
|
|
|
|
|
|
|
|
|
| Gate | Status / action |
|
|
|
|
|
|
|---|---|
|
|
|
|
|
|
| PLAT-86 soak ≥24h from Lambda touch `2026-08-07T15:06:49Z` | Ready at **2026-08-08T15:06:49Z** |
|
|
|
|
|
|
| Emitter failure/rejected SQS empty | Re-check before freeze |
|
|
|
|
|
|
| `shoc-assessment-dynamo-reader` | Torn down 2026-08-07 (prep) |
|
|
|
|
|
|
| SHOC Sync disable | Optional hygiene; not required |
|
|
|
|
|
|
|
|
|
|
|
|
Freeze window: schedule after soak. Buffer ≤60 min.
|
|
|
|
|
|
|
|
|
|
|
|
## Phase 1 — Prep apply (empty kebab tables)
|
|
|
|
|
|
|
|
|
|
|
|
Already in tree: `aws_dynamodb_table.work_orders_kebab` / `work_order_comments_kebab`,
|
|
|
|
|
|
emitter env `WORK_ORDERS_TABLE` / `COMMENTS_TABLE` (still pointing at legacy),
|
|
|
|
|
|
`scripts/migrate_wo_tables.py`.
|
|
|
|
|
|
|
|
|
|
|
|
1. Merge prep PR to `main`.
|
|
|
|
|
|
2. HCP Manual apply on `procurement-ingest-prod` (creates empty kebab tables only).
|
|
|
|
|
|
3. Confirm:
|
|
|
|
|
|
```bash
|
|
|
|
|
|
aws dynamodb describe-table --table-name work-orders --profile seahaven-prod --region us-east-1
|
|
|
|
|
|
aws dynamodb describe-table --table-name work-order-comments --profile seahaven-prod --region us-east-1
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
Writers and ESMs stay on PascalCase.
|
|
|
|
|
|
|
2026-08-07 14:26:44 -04:00
|
|
|
|
## Phase 2 — Freeze cutover (DONE 2026-08-07)
|
|
|
|
|
|
|
|
|
|
|
|
`FREEZE_START=2026-08-07T17:54:21Z`. Copy verified 7767 / 93038. Writers+ESMs on kebab. Replay since freeze: 0 events.
|
2026-08-07 13:42:35 -04:00
|
|
|
|
|
|
|
|
|
|
Record `FREEZE_START` (UTC ISO-8601) before step 1.
|
|
|
|
|
|
|
|
|
|
|
|
1. **Freeze ingest** (pick one):
|
|
|
|
|
|
```bash
|
|
|
|
|
|
aws lambda put-function-concurrency \
|
|
|
|
|
|
--function-name workorder-email-processor \
|
|
|
|
|
|
--reserved-concurrent-executions 0 \
|
|
|
|
|
|
--profile seahaven-prod --region us-east-1
|
|
|
|
|
|
```
|
|
|
|
|
|
2. **Disable ESMs** (note UUIDs from console or CLI list):
|
|
|
|
|
|
```bash
|
|
|
|
|
|
aws lambda list-event-source-mappings \
|
|
|
|
|
|
--function-name workorder-shoc-emitter \
|
|
|
|
|
|
--profile seahaven-prod --region us-east-1
|
|
|
|
|
|
# then for each UUID:
|
|
|
|
|
|
aws lambda update-event-source-mapping --uuid <UUID> --enabled false \
|
|
|
|
|
|
--profile seahaven-prod --region us-east-1
|
|
|
|
|
|
```
|
|
|
|
|
|
3. **Copy**:
|
|
|
|
|
|
```bash
|
|
|
|
|
|
python scripts/migrate_wo_tables.py copy --profile seahaven-prod
|
|
|
|
|
|
python scripts/migrate_wo_tables.py copy --profile seahaven-prod --execute
|
|
|
|
|
|
python scripts/migrate_wo_tables.py verify --profile seahaven-prod
|
|
|
|
|
|
```
|
|
|
|
|
|
4. **TF cutover** (same PR follow-up commit or cutover PR) — retarget:
|
|
|
|
|
|
- `wo_lambda.tf` / `api.tf` env → `aws_dynamodb_table.work_orders_kebab.name` (and comments kebab)
|
|
|
|
|
|
- `wo_shoc.tf` ESM `event_source_arn` → kebab stream ARNs; emitter env table names → kebab
|
|
|
|
|
|
- `iam.tf` DDB ARNs that reference `work_orders` / `work_order_comments` → include kebab resources (or switch)
|
|
|
|
|
|
- `wo_ddb.tf` `wo_ddb_alarm_tables` keys → `work-orders` / `work-order-comments` mapped to kebab resources
|
|
|
|
|
|
5. HCP Manual apply cutover.
|
|
|
|
|
|
6. **Unfreeze**:
|
|
|
|
|
|
```bash
|
|
|
|
|
|
aws lambda delete-function-concurrency \
|
|
|
|
|
|
--function-name workorder-email-processor \
|
|
|
|
|
|
--profile seahaven-prod --region us-east-1
|
|
|
|
|
|
# re-enable ESMs (or let TF `enabled = true` recreate/update)
|
|
|
|
|
|
```
|
|
|
|
|
|
7. Smoke: `scripts/post-deploy-smoke.sh`; SigV4 `GET /work-orders?limit=1`.
|
|
|
|
|
|
8. **Backfill SHOC**:
|
|
|
|
|
|
```bash
|
|
|
|
|
|
python scripts/replay_shoc_webhooks.py \
|
|
|
|
|
|
--url https://api.dev.seahaven.com/api/webhooks/work-orders \
|
|
|
|
|
|
--since "$FREEZE_START" --execute
|
|
|
|
|
|
```
|
|
|
|
|
|
And/or one SHOC reconciliation pass against procurement-api.
|
|
|
|
|
|
|
2026-08-14 11:32:55 -04:00
|
|
|
|
## Phase 3 — Decommission (DONE after ≥24h kebab soak; ready was **2026-08-08T18:16Z**)
|
2026-08-07 13:42:35 -04:00
|
|
|
|
|
2026-08-14 11:32:55 -04:00
|
|
|
|
1. Lift `prevent_destroy` on **legacy** `work_orders` / `work_order_comments` only. Done (#176).
|
2026-08-07 13:42:35 -04:00
|
|
|
|
2. Remove legacy table resources + their import blocks + PascalCase alarm imports.
|
|
|
|
|
|
3. HCP apply (destroys PascalCase tables).
|
|
|
|
|
|
4. Update README + Confluence Architecture Map; note GitHub #24 superseded on PLAT-11.
|
|
|
|
|
|
5. Close PLAT-11 acceptance criteria.
|
|
|
|
|
|
|
2026-08-14 11:32:55 -04:00
|
|
|
|
Rollback of Phase 3 is not available: PascalCase tables are gone. Live traffic is kebab-only.
|
