mirror of
https://github.com/Sea-Haven-Industries/procurement-ingest.git
synced 2026-09-30 04:53:12 +00:00
13efee9682
* feat(infra): add kebab WO tables and PLAT-11 cutover tooling Create empty work-orders/work-order-comments under Terraform while writers stay on PascalCase; make emitter table classification env-driven and add same-account migrate/verify plus cutover runbook. * style(test): ruff-format shoc emitter envelope tests
92 lines
3.7 KiB
Markdown
92 lines
3.7 KiB
Markdown
# PLAT-11 cutover runbook — WO Dynamo kebab rename
|
||
|
||
Rename `WorkOrders` / `WorkOrderComments` → `work-orders` / `work-order-comments`
|
||
under HCP Terraform. SHOC is still **dev**; webhook quiet ~15–30 min is fine;
|
||
backfill after unfreeze.
|
||
|
||
## Gates
|
||
|
||
| Gate | Status / action |
|
||
|---|---|
|
||
| PLAT-86 soak ≥24h from Lambda touch `2026-08-07T15:06:49Z` | Ready at **2026-08-08T15:06:49Z** |
|
||
| Emitter failure/rejected SQS empty | Re-check before freeze |
|
||
| `shoc-assessment-dynamo-reader` | Torn down 2026-08-07 (prep) |
|
||
| SHOC Sync disable | Optional hygiene; not required |
|
||
|
||
Freeze window: schedule after soak. Buffer ≤60 min.
|
||
|
||
## Phase 1 — Prep apply (empty kebab tables)
|
||
|
||
Already in tree: `aws_dynamodb_table.work_orders_kebab` / `work_order_comments_kebab`,
|
||
emitter env `WORK_ORDERS_TABLE` / `COMMENTS_TABLE` (still pointing at legacy),
|
||
`scripts/migrate_wo_tables.py`.
|
||
|
||
1. Merge prep PR to `main`.
|
||
2. HCP Manual apply on `procurement-ingest-prod` (creates empty kebab tables only).
|
||
3. Confirm:
|
||
```bash
|
||
aws dynamodb describe-table --table-name work-orders --profile seahaven-prod --region us-east-1
|
||
aws dynamodb describe-table --table-name work-order-comments --profile seahaven-prod --region us-east-1
|
||
```
|
||
|
||
Writers and ESMs stay on PascalCase.
|
||
|
||
## Phase 2 — Freeze cutover
|
||
|
||
Record `FREEZE_START` (UTC ISO-8601) before step 1.
|
||
|
||
1. **Freeze ingest** (pick one):
|
||
```bash
|
||
aws lambda put-function-concurrency \
|
||
--function-name workorder-email-processor \
|
||
--reserved-concurrent-executions 0 \
|
||
--profile seahaven-prod --region us-east-1
|
||
```
|
||
2. **Disable ESMs** (note UUIDs from console or CLI list):
|
||
```bash
|
||
aws lambda list-event-source-mappings \
|
||
--function-name workorder-shoc-emitter \
|
||
--profile seahaven-prod --region us-east-1
|
||
# then for each UUID:
|
||
aws lambda update-event-source-mapping --uuid <UUID> --enabled false \
|
||
--profile seahaven-prod --region us-east-1
|
||
```
|
||
3. **Copy**:
|
||
```bash
|
||
python scripts/migrate_wo_tables.py copy --profile seahaven-prod
|
||
python scripts/migrate_wo_tables.py copy --profile seahaven-prod --execute
|
||
python scripts/migrate_wo_tables.py verify --profile seahaven-prod
|
||
```
|
||
4. **TF cutover** (same PR follow-up commit or cutover PR) — retarget:
|
||
- `wo_lambda.tf` / `api.tf` env → `aws_dynamodb_table.work_orders_kebab.name` (and comments kebab)
|
||
- `wo_shoc.tf` ESM `event_source_arn` → kebab stream ARNs; emitter env table names → kebab
|
||
- `iam.tf` DDB ARNs that reference `work_orders` / `work_order_comments` → include kebab resources (or switch)
|
||
- `wo_ddb.tf` `wo_ddb_alarm_tables` keys → `work-orders` / `work-order-comments` mapped to kebab resources
|
||
5. HCP Manual apply cutover.
|
||
6. **Unfreeze**:
|
||
```bash
|
||
aws lambda delete-function-concurrency \
|
||
--function-name workorder-email-processor \
|
||
--profile seahaven-prod --region us-east-1
|
||
# re-enable ESMs (or let TF `enabled = true` recreate/update)
|
||
```
|
||
7. Smoke: `scripts/post-deploy-smoke.sh`; SigV4 `GET /work-orders?limit=1`.
|
||
8. **Backfill SHOC**:
|
||
```bash
|
||
python scripts/replay_shoc_webhooks.py \
|
||
--url https://api.dev.seahaven.com/api/webhooks/work-orders \
|
||
--since "$FREEZE_START" --execute
|
||
```
|
||
And/or one SHOC reconciliation pass against procurement-api.
|
||
|
||
## Phase 3 — Decommission (after ≥24h on kebab)
|
||
|
||
1. Lift `prevent_destroy` on **legacy** `work_orders` / `work_order_comments` only.
|
||
2. Remove legacy table resources + their import blocks + PascalCase alarm imports.
|
||
3. HCP apply (destroys PascalCase tables).
|
||
4. Update README + Confluence Architecture Map; note GitHub #24 superseded on PLAT-11.
|
||
5. Close PLAT-11 acceptance criteria.
|
||
|
||
## Rollback (before decommission)
|
||
|
||
Point env + ESMs back at PascalCase tables via TF; HCP apply. Legacy tables still hold data until Phase 3.
|