procurement-ingest/docs/plat-11/cutover-runbook.md

PLAT-11 cutover runbook — WO Dynamo kebab rename

Rename WorkOrders / WorkOrderComments → work-orders / work-order-comments under HCP Terraform. SHOC is still dev; webhook quiet ~15–30 min is fine; backfill after unfreeze.

Gates

Gate	Status / action
PLAT-86 soak ≥24h from Lambda touch 2026-08-07T15:06:49Z	Ready at 2026-08-08T15:06:49Z
Emitter failure/rejected SQS empty	Re-check before freeze
shoc-assessment-dynamo-reader	Torn down 2026-08-07 (prep)
SHOC Sync disable	Optional hygiene; not required

Freeze window: schedule after soak. Buffer ≤60 min.

Phase 1 — Prep apply (empty kebab tables)

Already in tree: aws_dynamodb_table.work_orders_kebab / work_order_comments_kebab, emitter env WORK_ORDERS_TABLE / COMMENTS_TABLE (still pointing at legacy), scripts/migrate_wo_tables.py.

1. Merge prep PR to main.
2. HCP Manual apply on procurement-ingest-prod (creates empty kebab tables only).
3. Confirm:

   aws dynamodb describe-table --table-name work-orders --profile seahaven-prod --region us-east-1
   aws dynamodb describe-table --table-name work-order-comments --profile seahaven-prod --region us-east-1
   

Writers and ESMs stay on PascalCase.

Phase 2 — Freeze cutover (DONE 2026-08-07)

FREEZE_START=2026-08-07T17:54:21Z. Copy verified 7767 / 93038. Writers+ESMs on kebab. Replay since freeze: 0 events.

Record FREEZE_START (UTC ISO-8601) before step 1.

1. Freeze ingest (pick one):

   aws lambda put-function-concurrency \
     --function-name workorder-email-processor \
     --reserved-concurrent-executions 0 \
     --profile seahaven-prod --region us-east-1
   

2. Disable ESMs (note UUIDs from console or CLI list):

   aws lambda list-event-source-mappings \
     --function-name workorder-shoc-emitter \
     --profile seahaven-prod --region us-east-1
   # then for each UUID:
   aws lambda update-event-source-mapping --uuid <UUID> --enabled false \
     --profile seahaven-prod --region us-east-1
   

3. Copy:

   python scripts/migrate_wo_tables.py copy --profile seahaven-prod
   python scripts/migrate_wo_tables.py copy --profile seahaven-prod --execute
   python scripts/migrate_wo_tables.py verify --profile seahaven-prod
   

4. TF cutover (same PR follow-up commit or cutover PR) — retarget:
   • wo_lambda.tf / api.tf env → aws_dynamodb_table.work_orders_kebab.name (and comments kebab)
   • wo_shoc.tf ESM event_source_arn → kebab stream ARNs; emitter env table names → kebab
   • iam.tf DDB ARNs that reference work_orders / work_order_comments → include kebab resources (or switch)
   • wo_ddb.tf wo_ddb_alarm_tables keys → work-orders / work-order-comments mapped to kebab resources
5. HCP Manual apply cutover.
6. Unfreeze:

   aws lambda delete-function-concurrency \
     --function-name workorder-email-processor \
     --profile seahaven-prod --region us-east-1
   # re-enable ESMs (or let TF `enabled = true` recreate/update)
   

7. Smoke: scripts/post-deploy-smoke.sh; SigV4 GET /work-orders?limit=1.
8. Backfill SHOC:

   python scripts/replay_shoc_webhooks.py \
     --url https://api.dev.seahaven.com/api/webhooks/work-orders \
     --since "$FREEZE_START" --execute
   

   And/or one SHOC reconciliation pass against procurement-api.

Phase 3 — Decommission (DONE after ≥24h kebab soak; ready was 2026-08-08T18:16Z)

1. Lift prevent_destroy on legacy work_orders / work_order_comments only. Done (#176).
2. Remove legacy table resources + their import blocks + PascalCase alarm imports.
3. HCP apply (destroys PascalCase tables).
4. Update README + Confluence Architecture Map; note GitHub #24 superseded on PLAT-11.
5. Close PLAT-11 acceptance criteria.

Rollback of Phase 3 is not available: PascalCase tables are gone. Live traffic is kebab-only.