feat(infra): add kebab WO tables and PLAT-11 cutover tooling (PLAT-11) (#172)

* feat(infra): add kebab WO tables and PLAT-11 cutover tooling

Create empty work-orders/work-order-comments under Terraform while writers
stay on PascalCase; make emitter table classification env-driven and add
same-account migrate/verify plus cutover runbook.

* style(test): ruff-format shoc emitter envelope tests
This commit is contained in:
Adam Moussa 2026-08-07 13:42:35 -04:00 • committed by GitHub
parent 41241eea86
commit 13efee9682
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
9 changed files with 395 additions and 19 deletions

View file

@ -272,6 +272,11 @@ The `purchase-orders` DynamoDB table is **owned by this repo's Terraform config*
### `WorkOrders` and `WorkOrderComments` tables (owned here) ### `WorkOrders` and `WorkOrderComments` tables (owned here)
> **PLAT-11 in progress.** Kebab physical names `work-orders` / `work-order-comments`
> are created empty under Terraform; writers/ESMs stay on PascalCase until the
> freeze cutover in [`docs/plat-11/cutover-runbook.md`](docs/plat-11/cutover-runbook.md).
> After cutover, this section will name the kebab tables.
Both tables are **owned by this repo's Terraform config** (`terraform/wo_ddb.tf`, retain lifecycle): Both tables are **owned by this repo's Terraform config** (`terraform/wo_ddb.tf`, retain lifecycle):
- `WorkOrders` — PK `work_order_id` (S). - `WorkOrders` — PK `work_order_id` (S).
@ -394,6 +399,15 @@ python scripts/backfill_sites.py
python scripts/replay_shoc_webhooks.py --url https://... --work-order-id 11144580730 # targeted, dry-run first python scripts/replay_shoc_webhooks.py --url https://... --work-order-id 11144580730 # targeted, dry-run first
python scripts/replay_shoc_webhooks.py --url https://... --work-order-id 11144580730 --execute # then re-POST python scripts/replay_shoc_webhooks.py --url https://... --work-order-id 11144580730 --execute # then re-POST
python scripts/replay_shoc_webhooks.py --url https://... --since 2026-07-24T02:00:00Z --execute # everything touched since (full Scan) python scripts/replay_shoc_webhooks.py --url https://... --since 2026-07-24T02:00:00Z --execute # everything touched since (full Scan)
**PLAT-11 WO table rename** (PascalCase → kebab, freeze cutover). See
[`docs/plat-11/cutover-runbook.md`](docs/plat-11/cutover-runbook.md). Dry-run copy:
```bash
python scripts/migrate_wo_tables.py copy
python scripts/migrate_wo_tables.py copy --execute
python scripts/migrate_wo_tables.py verify
```
``` ```
## Directory Structure ## Directory Structure

View file

@ -0,0 +1,92 @@
# PLAT-11 cutover runbook — WO Dynamo kebab rename
Rename `WorkOrders` / `WorkOrderComments` → `work-orders` / `work-order-comments`
under HCP Terraform. SHOC is still **dev**; webhook quiet ~15–30 min is fine;
backfill after unfreeze.
## Gates
| Gate | Status / action |
|---|---|
| PLAT-86 soak ≥24h from Lambda touch `2026-08-07T15:06:49Z` | Ready at **2026-08-08T15:06:49Z** |
| Emitter failure/rejected SQS empty | Re-check before freeze |
| `shoc-assessment-dynamo-reader` | Torn down 2026-08-07 (prep) |
| SHOC Sync disable | Optional hygiene; not required |
Freeze window: schedule after soak. Buffer ≤60 min.
## Phase 1 — Prep apply (empty kebab tables)
Already in tree: `aws_dynamodb_table.work_orders_kebab` / `work_order_comments_kebab`,
emitter env `WORK_ORDERS_TABLE` / `COMMENTS_TABLE` (still pointing at legacy),
`scripts/migrate_wo_tables.py`.
1. Merge prep PR to `main`.
2. HCP Manual apply on `procurement-ingest-prod` (creates empty kebab tables only).
3. Confirm:
```bash
aws dynamodb describe-table --table-name work-orders --profile seahaven-prod --region us-east-1
aws dynamodb describe-table --table-name work-order-comments --profile seahaven-prod --region us-east-1
```
Writers and ESMs stay on PascalCase.
## Phase 2 — Freeze cutover
Record `FREEZE_START` (UTC ISO-8601) before step 1.
1. **Freeze ingest** (pick one):
```bash
aws lambda put-function-concurrency \
--function-name workorder-email-processor \
--reserved-concurrent-executions 0 \
--profile seahaven-prod --region us-east-1
```
2. **Disable ESMs** (note UUIDs from console or CLI list):
```bash
aws lambda list-event-source-mappings \
--function-name workorder-shoc-emitter \
--profile seahaven-prod --region us-east-1
# then for each UUID:
aws lambda update-event-source-mapping --uuid <UUID> --enabled false \
--profile seahaven-prod --region us-east-1
```
3. **Copy**:
```bash
python scripts/migrate_wo_tables.py copy --profile seahaven-prod
python scripts/migrate_wo_tables.py copy --profile seahaven-prod --execute
python scripts/migrate_wo_tables.py verify --profile seahaven-prod
```
4. **TF cutover** (same PR follow-up commit or cutover PR) — retarget:
- `wo_lambda.tf` / `api.tf` env → `aws_dynamodb_table.work_orders_kebab.name` (and comments kebab)
- `wo_shoc.tf` ESM `event_source_arn` → kebab stream ARNs; emitter env table names → kebab
- `iam.tf` DDB ARNs that reference `work_orders` / `work_order_comments` → include kebab resources (or switch)
- `wo_ddb.tf` `wo_ddb_alarm_tables` keys → `work-orders` / `work-order-comments` mapped to kebab resources
5. HCP Manual apply cutover.
6. **Unfreeze**:
```bash
aws lambda delete-function-concurrency \
--function-name workorder-email-processor \
--profile seahaven-prod --region us-east-1
# re-enable ESMs (or let TF `enabled = true` recreate/update)
```
7. Smoke: `scripts/post-deploy-smoke.sh`; SigV4 `GET /work-orders?limit=1`.
8. **Backfill SHOC**:
```bash
python scripts/replay_shoc_webhooks.py \
--url https://api.dev.seahaven.com/api/webhooks/work-orders \
--since "$FREEZE_START" --execute
```
And/or one SHOC reconciliation pass against procurement-api.
## Phase 3 — Decommission (after ≥24h on kebab)
1. Lift `prevent_destroy` on **legacy** `work_orders` / `work_order_comments` only.
2. Remove legacy table resources + their import blocks + PascalCase alarm imports.
3. HCP apply (destroys PascalCase tables).
4. Update README + Confluence Architecture Map; note GitHub #24 superseded on PLAT-11.
5. Close PLAT-11 acceptance criteria.
## Rollback (before decommission)
Point env + ESMs back at PascalCase tables via TF; HCP apply. Legacy tables still hold data until Phase 3.

View file

@ -1,4 +1,12 @@
# shoc-assessment-dynamo-reader (TEMPORARY) # shoc-assessment-dynamo-reader (TEMPORARY) — TORN DOWN
**Status (2026-08-07):** Role deleted in seahaven-prod as part of PLAT-11 prep
(`./teardown-assessment-reader.sh`). Do not recreate for the kebab rename.
SHOC steady-state access remains `procurement-api` + webhook only.
---
Original design (historical):
Time-boxed cross-account read role in **seahaven-prod (011934824531)** that lets Time-boxed cross-account read role in **seahaven-prod (011934824531)** that lets
the Luby team assess the procurement-ingest DynamoDB tables directly from the Luby team assess the procurement-ingest DynamoDB tables directly from

View file

@ -1,17 +1,20 @@
"""Stream-record -> SHOC webhook envelope mapping (contract sections 3-4). """Stream-record -> SHOC webhook envelope mapping (contract sections 3-4).
Pure and total: no boto3 clients, no environment reads, no network. Every Pure and total for classification: no boto3 clients, no network. Table names
function either returns a value or returns None (skip) -- a malformed stream come from ``WORK_ORDERS_TABLE`` / ``COMMENTS_TABLE`` env (PLAT-11 kebab rename);
record must never raise here; classification gaps fall through to None so the defaults preserve the pre-rename PascalCase names so unit tests without env
shard is never blocked by an unmappable record. ``build_event`` is the single still pin the legacy stream ARNs. Every function either returns a value or
entry point the handler calls per record. returns None (skip) -- a malformed stream record must never raise here;
classification gaps fall through to None so the shard is never blocked by an
unmappable record. ``build_event`` is the single entry point the handler calls
per record.
Classification (docs/shoc-webhook-contract.md section 4): Classification (docs/shoc-webhook-contract.md section 4):
WorkOrders INSERT -> work_order.created work-orders / WorkOrders INSERT -> work_order.created
WorkOrders MODIFY -> work_order.cancelled iff OLD wo_status was not work-orders / WorkOrders MODIFY -> work_order.cancelled iff OLD
"cancelled" AND NEW wo_status is "cancelled", wo_status was not "cancelled" AND NEW wo_status is "cancelled",
else work_order.updated else work_order.updated
WorkOrderComments INSERT -> work_order.comment_added work-order-comments / WorkOrderComments INSERT -> work_order.comment_added
REMOVE (either table) -> skip (no deletes in this pipeline) REMOVE (either table) -> skip (no deletes in this pipeline)
NEW write_origin == "shoc-write-api" -> skip (echo guard; a no-op branch NEW write_origin == "shoc-write-api" -> skip (echo guard; a no-op branch
today -- nothing writes that attribute yet -- kept so the phase-2 today -- nothing writes that attribute yet -- kept so the phase-2
@ -21,6 +24,7 @@ Classification (docs/shoc-webhook-contract.md section 4):
Anything else (unknown table, comment MODIFYs) -> skip. Anything else (unknown table, comment MODIFYs) -> skip.
""" """
import os
from datetime import datetime, timezone from datetime import datetime, timezone
from decimal import Decimal from decimal import Decimal
@ -30,8 +34,9 @@ SCHEMA_VERSION = 1
SOURCE = "procurement-ingest/workorder-shoc-emitter" SOURCE = "procurement-ingest/workorder-shoc-emitter"
CANCELLED_STATUS = "cancelled" CANCELLED_STATUS = "cancelled"
SHOC_WRITE_ORIGIN = "shoc-write-api" SHOC_WRITE_ORIGIN = "shoc-write-api"
WORK_ORDERS_TABLE = "WorkOrders" # Defaults match pre-PLAT-11 physical names; TF/Lambda env overrides at cutover.
COMMENTS_TABLE = "WorkOrderComments" WORK_ORDERS_TABLE = os.environ.get("WORK_ORDERS_TABLE", "WorkOrders")
COMMENTS_TABLE = os.environ.get("COMMENTS_TABLE", "WorkOrderComments")
# data payload field lists (contract sections 4.1 / 4.2). Absent attributes are # data payload field lists (contract sections 4.1 / 4.2). Absent attributes are
# null-filled; source_email_s3_key is deliberately EXCLUDED (internal key, not # null-filled; source_email_s3_key is deliberately EXCLUDED (internal key, not
@ -100,9 +105,9 @@ def _table_name(event_source_arn: str) -> str | None:
"""Extract the exact table-name segment from a stream eventSourceARN. """Extract the exact table-name segment from a stream eventSourceARN.
ARN format: arn:aws:dynamodb:region:acct:table/NAME/stream/LABEL. Parsing ARN format: arn:aws:dynamodb:region:acct:table/NAME/stream/LABEL. Parsing
the segment exactly (not a loose substring) matters: "WorkOrders" is a the segment exactly (not a loose substring) matters: legacy "WorkOrders"
prefix of nothing, but a substring test for it WOULD match is a leading substring of "WorkOrderComments", and kebab "work-orders" is
"WorkOrderComments"-adjacent names -- segment equality can't. similarly adjacent to "work-order-comments" -- segment equality can't.
""" """
parts = event_source_arn.split("/") parts = event_source_arn.split("/")
if len(parts) >= _MIN_ARN_SLASH_SEGMENTS and parts[0].endswith(":table"): if len(parts) >= _MIN_ARN_SLASH_SEGMENTS and parts[0].endswith(":table"):

View file

@ -0,0 +1,166 @@
#!/usr/bin/env python3
"""PLAT-11 same-account WO table copy: PascalCase -> kebab-case.
Copies ``WorkOrders`` -> ``work-orders`` and ``WorkOrderComments`` ->
``work-order-comments`` inside seahaven-prod. Dry-run is the DEFAULT; nothing
is written unless ``--execute`` is passed.
Usage:
python scripts/migrate_wo_tables.py copy # dry-run both
python scripts/migrate_wo_tables.py copy --execute
python scripts/migrate_wo_tables.py verify
python scripts/migrate_wo_tables.py copy --table work-orders --execute
"""
from __future__ import annotations
import argparse
import sys
import time
import boto3
from boto3.dynamodb.types import TypeDeserializer
EXPECTED_ACCOUNT = "011934824531"
REGION = "us-east-1"
PAIRS = {
"work-orders": {
"source": "WorkOrders",
"dest": "work-orders",
"keys": ["work_order_id"],
},
"work-order-comments": {
"source": "WorkOrderComments",
"dest": "work-order-comments",
"keys": ["work_order_id", "comment_id"],
},
}
_deserializer = TypeDeserializer()
def _client(profile: str):
session = boto3.Session(profile_name=profile, region_name=REGION)
acct = session.client("sts").get_caller_identity()["Account"]
if acct != EXPECTED_ACCOUNT:
sys.exit(
f"ERROR: profile {profile!r} resolves to account {acct}, "
f"expected {EXPECTED_ACCOUNT}. Aborting."
)
return session.client("dynamodb")
def _describe_count(client, table: str) -> int:
return int(client.describe_table(TableName=table)["Table"]["ItemCount"])
def _scan_all(client, table: str):
kwargs = {"TableName": table}
while True:
resp = client.scan(**kwargs)
for item in resp.get("Items", []):
yield item
if "LastEvaluatedKey" not in resp:
break
kwargs["ExclusiveStartKey"] = resp["LastEvaluatedKey"]
def _batch_write(client, table: str, items: list[dict], execute: bool) -> int:
if not execute:
return len(items)
written = 0
for i in range(0, len(items), 25):
chunk = items[i : i + 25]
request_items = {
table: [{"PutRequest": {"Item": item}} for item in chunk],
}
unprocessed = request_items
while unprocessed:
resp = client.batch_write_item(RequestItems=unprocessed)
unprocessed = resp.get("UnprocessedItems") or {}
if unprocessed:
time.sleep(0.2)
written += len(chunk)
return written
def copy_table(client, pair_key: str, execute: bool) -> None:
pair = PAIRS[pair_key]
source, dest = pair["source"], pair["dest"]
print(f"==> copy {source} -> {dest} ({'EXECUTE' if execute else 'dry-run'})")
items = list(_scan_all(client, source))
print(f" scanned {len(items)} items from {source}")
written = _batch_write(client, dest, items, execute=execute)
print(f" {'wrote' if execute else 'would write'} {written} items to {dest}")
def verify_table(client, pair_key: str) -> bool:
pair = PAIRS[pair_key]
source, dest = pair["source"], pair["dest"]
# ItemCount is eventually consistent; prefer live scan counts for gate.
src_items = list(_scan_all(client, source))
dst_items = list(_scan_all(client, dest))
src_n, dst_n = len(src_items), len(dst_items)
print(f"==> verify {source} ({src_n}) vs {dest} ({dst_n})")
if src_n != dst_n:
print(" FAIL count mismatch")
return False
key_names = pair["keys"]
def key_tuple(item):
return tuple(_deserializer.deserialize(item[k]) for k in key_names)
src_by_key = {key_tuple(i): i for i in src_items}
dst_by_key = {key_tuple(i): i for i in dst_items}
missing = [k for k in src_by_key if k not in dst_by_key]
if missing:
print(f" FAIL missing keys on dest (showing up to 5): {missing[:5]}")
return False
# Spot-check up to 25 items for full attribute equality.
checked = 0
for key, src_item in list(src_by_key.items())[:25]:
if src_item != dst_by_key[key]:
print(f" FAIL attribute mismatch for key {key}")
return False
checked += 1
print(f" OK counts equal; spot-checked {checked} items")
return True
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("command", choices=["copy", "verify"])
parser.add_argument(
"--table",
choices=list(PAIRS.keys()),
help="Single pair key (default: both)",
)
parser.add_argument(
"--execute",
action="store_true",
help="Actually write (copy only). Default is dry-run.",
)
parser.add_argument("--profile", default="seahaven-prod")
args = parser.parse_args()
client = _client(args.profile)
keys = [args.table] if args.table else list(PAIRS.keys())
if args.command == "copy":
for key in keys:
copy_table(client, key, execute=args.execute)
if not args.execute:
print("Dry-run complete. Re-run with --execute to write.")
return 0
ok = True
for key in keys:
ok = verify_table(client, key) and ok
return 0 if ok else 1
if __name__ == "__main__":
raise SystemExit(main())

View file

@ -58,14 +58,16 @@ import urllib.request
from datetime import datetime, timedelta, timezone from datetime import datetime, timedelta, timezone
from decimal import Decimal from decimal import Decimal
import os
import boto3 import boto3
from boto3.dynamodb.conditions import Key from boto3.dynamodb.conditions import Key
EXPECTED_ACCOUNT = "011934824531" EXPECTED_ACCOUNT = "011934824531"
REGION = "us-east-1" REGION = "us-east-1"
WO_TABLE = "WorkOrders" WO_TABLE = os.environ.get("WORK_ORDERS_TABLE", "WorkOrders")
COMMENTS_TABLE = "WorkOrderComments" COMMENTS_TABLE = os.environ.get("COMMENTS_TABLE", "WorkOrderComments")
SECRET_NAME = "workorder-ingest/shoc-webhook-hmac" SECRET_NAME = "workorder-ingest/shoc-webhook-hmac"
SCHEMA_VERSION = 1 SCHEMA_VERSION = 1

View file

@ -1,3 +1,7 @@
# PLAT-11: PascalCase tables remain authoritative until the freeze cutover.
# Kebab tables are created empty in the prep apply; writers/ESMs stay on legacy
# until docs/plat-11/cutover-runbook.md Phase 2.
resource "aws_dynamodb_table" "work_orders" { resource "aws_dynamodb_table" "work_orders" {
name = "WorkOrders" name = "WorkOrders"
billing_mode = "PAY_PER_REQUEST" billing_mode = "PAY_PER_REQUEST"
@ -40,7 +44,50 @@ resource "aws_dynamodb_table" "work_order_comments" {
} }
} }
resource "aws_dynamodb_table" "work_orders_kebab" {
name = "work-orders"
billing_mode = "PAY_PER_REQUEST"
hash_key = "work_order_id"
attribute {
name = "work_order_id"
type = "S"
}
stream_enabled = true
stream_view_type = "NEW_AND_OLD_IMAGES"
lifecycle {
prevent_destroy = true
}
}
resource "aws_dynamodb_table" "work_order_comments_kebab" {
name = "work-order-comments"
billing_mode = "PAY_PER_REQUEST"
hash_key = "work_order_id"
range_key = "comment_id"
attribute {
name = "work_order_id"
type = "S"
}
attribute {
name = "comment_id"
type = "S"
}
stream_enabled = true
stream_view_type = "NEW_AND_OLD_IMAGES"
lifecycle {
prevent_destroy = true
}
}
locals { locals {
# Alarms stay on the live writer tables (legacy until cutover).
wo_ddb_alarm_tables = { wo_ddb_alarm_tables = {
"WorkOrders" = aws_dynamodb_table.work_orders.name "WorkOrders" = aws_dynamodb_table.work_orders.name
"WorkOrderComments" = aws_dynamodb_table.work_order_comments.name "WorkOrderComments" = aws_dynamodb_table.work_order_comments.name

View file

@ -292,6 +292,9 @@ resource "aws_lambda_function" "wo_shoc_emitter" {
HMAC_SECRET_ARN = var.shoc_hmac_secret_arn HMAC_SECRET_ARN = var.shoc_hmac_secret_arn
REJECTED_QUEUE_URL = aws_sqs_queue.shoc_emitter_rejected.url REJECTED_QUEUE_URL = aws_sqs_queue.shoc_emitter_rejected.url
SHOC_WEBHOOK_URL = var.shoc_webhook_url SHOC_WEBHOOK_URL = var.shoc_webhook_url
# Stream ARN classification (PLAT-11); must match ESM source table names.
WORK_ORDERS_TABLE = aws_dynamodb_table.work_orders.name
COMMENTS_TABLE = aws_dynamodb_table.work_order_comments.name
} }
} }

View file

@ -12,7 +12,8 @@ contract names AND the published spec's webhooks keys (lambdas/api/
openapi.json), and the classification's status/record_type value space must openapi.json), and the classification's status/record_type value space must
stay anchored to the WO pipeline's template_parser enums. stay anchored to the WO pipeline's template_parser enums.
Offline and pure -- envelope.py has no boto3 clients, no env reads. Offline -- envelope.py has no boto3 clients and no network. Table names
default to PascalCase; set WORK_ORDERS_TABLE / COMMENTS_TABLE for kebab.
""" """
import json import json
@ -231,6 +232,44 @@ def test_table_discrimination_is_exact_segment_not_substring(envelope):
assert envelope.build_event(_record("", "INSERT", _wo_image())) is None assert envelope.build_event(_record("", "INSERT", _wo_image())) is None
def test_kebab_table_names_via_env(monkeypatch):
"""PLAT-11: classification follows WORK_ORDERS_TABLE / COMMENTS_TABLE."""
import sys
monkeypatch.setenv("WORK_ORDERS_TABLE", "work-orders")
monkeypatch.setenv("COMMENTS_TABLE", "work-order-comments")
sys.modules.pop("wo_shoc_emitter_envelope", None)
kebab_envelope = load_lambda_module("wo", "shoc_emitter/envelope")
wo_arn = (
"arn:aws:dynamodb:us-east-1:011934824531:table/work-orders"
"/stream/2026-08-07T00:00:00.000"
)
comments_arn = (
"arn:aws:dynamodb:us-east-1:011934824531:table/work-order-comments"
"/stream/2026-08-07T00:00:00.000"
)
assert (
kebab_envelope.build_event(_record(wo_arn, "INSERT", _wo_image()))["event_type"]
== "work_order.created"
)
assert (
kebab_envelope.build_event(_record(comments_arn, "INSERT", _comment_image()))[
"event_type"
]
== "work_order.comment_added"
)
# Legacy PascalCase ARNs must not match kebab env names.
assert (
kebab_envelope.build_event(_record(WO_STREAM_ARN, "INSERT", _wo_image()))
is None
)
# Restore cached module for later tests that expect PascalCase defaults.
sys.modules.pop("wo_shoc_emitter_envelope", None)
monkeypatch.delenv("WORK_ORDERS_TABLE", raising=False)
monkeypatch.delenv("COMMENTS_TABLE", raising=False)
load_lambda_module("wo", "shoc_emitter/envelope")
# --- Envelope fields --------------------------------------------------------- # --- Envelope fields ---------------------------------------------------------