mirror of
https://github.com/Sea-Haven-Industries/procurement-ingest.git
synced 2026-09-30 06:03:14 +00:00
92 lines
3.9 KiB
Markdown
92 lines
3.9 KiB
Markdown
# PLAT-11 cutover runbook — WO Dynamo kebab rename
|
||
|
||
Rename `WorkOrders` / `WorkOrderComments` → `work-orders` / `work-order-comments`
|
||
under HCP Terraform. SHOC is still **dev**; webhook quiet ~15–30 min is fine;
|
||
backfill after unfreeze.
|
||
|
||
## Gates
|
||
|
||
| Gate | Status / action |
|
||
|---|---|
|
||
| PLAT-86 soak ≥24h from Lambda touch `2026-08-07T15:06:49Z` | Ready at **2026-08-08T15:06:49Z** |
|
||
| Emitter failure/rejected SQS empty | Re-check before freeze |
|
||
| `shoc-assessment-dynamo-reader` | Torn down 2026-08-07 (prep) |
|
||
| SHOC Sync disable | Optional hygiene; not required |
|
||
|
||
Freeze window: schedule after soak. Buffer ≤60 min.
|
||
|
||
## Phase 1 — Prep apply (empty kebab tables)
|
||
|
||
Already in tree: `aws_dynamodb_table.work_orders_kebab` / `work_order_comments_kebab`,
|
||
emitter env `WORK_ORDERS_TABLE` / `COMMENTS_TABLE` (still pointing at legacy),
|
||
`scripts/migrate_wo_tables.py`.
|
||
|
||
1. Merge prep PR to `main`.
|
||
2. HCP Manual apply on `procurement-ingest-prod` (creates empty kebab tables only).
|
||
3. Confirm:
|
||
```bash
|
||
aws dynamodb describe-table --table-name work-orders --profile seahaven-prod --region us-east-1
|
||
aws dynamodb describe-table --table-name work-order-comments --profile seahaven-prod --region us-east-1
|
||
```
|
||
|
||
Writers and ESMs stay on PascalCase.
|
||
|
||
## Phase 2 — Freeze cutover (DONE 2026-08-07)
|
||
|
||
`FREEZE_START=2026-08-07T17:54:21Z`. Copy verified 7767 / 93038. Writers+ESMs on kebab. Replay since freeze: 0 events.
|
||
|
||
Record `FREEZE_START` (UTC ISO-8601) before step 1.
|
||
|
||
1. **Freeze ingest** (pick one):
|
||
```bash
|
||
aws lambda put-function-concurrency \
|
||
--function-name workorder-email-processor \
|
||
--reserved-concurrent-executions 0 \
|
||
--profile seahaven-prod --region us-east-1
|
||
```
|
||
2. **Disable ESMs** (note UUIDs from console or CLI list):
|
||
```bash
|
||
aws lambda list-event-source-mappings \
|
||
--function-name workorder-shoc-emitter \
|
||
--profile seahaven-prod --region us-east-1
|
||
# then for each UUID:
|
||
aws lambda update-event-source-mapping --uuid <UUID> --enabled false \
|
||
--profile seahaven-prod --region us-east-1
|
||
```
|
||
3. **Copy**:
|
||
```bash
|
||
python scripts/migrate_wo_tables.py copy --profile seahaven-prod
|
||
python scripts/migrate_wo_tables.py copy --profile seahaven-prod --execute
|
||
python scripts/migrate_wo_tables.py verify --profile seahaven-prod
|
||
```
|
||
4. **TF cutover** (same PR follow-up commit or cutover PR) — retarget:
|
||
- `wo_lambda.tf` / `api.tf` env → `aws_dynamodb_table.work_orders_kebab.name` (and comments kebab)
|
||
- `wo_shoc.tf` ESM `event_source_arn` → kebab stream ARNs; emitter env table names → kebab
|
||
- `iam.tf` DDB ARNs that reference `work_orders` / `work_order_comments` → include kebab resources (or switch)
|
||
- `wo_ddb.tf` `wo_ddb_alarm_tables` keys → `work-orders` / `work-order-comments` mapped to kebab resources
|
||
5. HCP Manual apply cutover.
|
||
6. **Unfreeze**:
|
||
```bash
|
||
aws lambda delete-function-concurrency \
|
||
--function-name workorder-email-processor \
|
||
--profile seahaven-prod --region us-east-1
|
||
# re-enable ESMs (or let TF `enabled = true` recreate/update)
|
||
```
|
||
7. Smoke: `scripts/post-deploy-smoke.sh`; SigV4 `GET /work-orders?limit=1`.
|
||
8. **Backfill SHOC**:
|
||
```bash
|
||
python scripts/replay_shoc_webhooks.py \
|
||
--url https://api.dev.seahaven.com/api/webhooks/work-orders \
|
||
--since "$FREEZE_START" --execute
|
||
```
|
||
And/or one SHOC reconciliation pass against procurement-api.
|
||
|
||
## Phase 3 — Decommission (DONE after ≥24h kebab soak; ready was **2026-08-08T18:16Z**)
|
||
|
||
1. Lift `prevent_destroy` on **legacy** `work_orders` / `work_order_comments` only. Done (#176).
|
||
2. Remove legacy table resources + their import blocks + PascalCase alarm imports.
|
||
3. HCP apply (destroys PascalCase tables).
|
||
4. Update README + Confluence Architecture Map; note GitHub #24 superseded on PLAT-11.
|
||
5. Close PLAT-11 acceptance criteria.
|
||
|
||
Rollback of Phase 3 is not available: PascalCase tables are gone. Live traffic is kebab-only.
|