From 7d0f2729013c07fd2eeffddff046a6eb1368b17e Mon Sep 17 00:00:00 2001 From: Adam Moussa <166072409+amoussa1229@users.noreply.github.com> Date: Fri, 7 Aug 2026 14:26:44 -0400 Subject: [PATCH] chore(infra): lift prevent_destroy on legacy WO tables for PLAT-11 (#176) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Writers already use kebab tables. Allow HCP to destroy PascalCase WorkOrders / WorkOrderComments after the ≥24h kebab soak. Docs scrub to kebab as the live physical names; GitHub #24 noted superseded. --- README.md | 27 +++++++++++++++------------ docs/plat-11/cutover-runbook.md | 6 ++++-- terraform/wo_ddb.tf | 9 +++++---- 3 files changed, 24 insertions(+), 18 deletions(-) diff --git a/README.md b/README.md index ed71c20..2946af5 100644 --- a/README.md +++ b/README.md @@ -61,7 +61,7 @@ Or aggregate overall agreement per field: `| filter ispresent(DerivedFieldAgreem ### Work Orders (`WorkorderIngestStack` stack) -Amazon APM work order emails (from Hexagon EAM / HxGN SmartCloud) are received at `apm@int.seahaven.com`, parsed **deterministic-template-first with a Claude-on-Bedrock fallback**, and written to the `WorkOrders` DynamoDB table. +Amazon APM work order emails (from Hexagon EAM / HxGN SmartCloud) are received at `apm@int.seahaven.com`, parsed **deterministic-template-first with a Claude-on-Bedrock fallback**, and written to the `work-orders` DynamoDB table. **Flow:** 1. Hexagon EAM sends email notifications (new assignments, comments, updates, cancellations) to `amazon@seahavenind.com`. @@ -70,7 +70,7 @@ Amazon APM work order emails (from Hexagon EAM / HxGN SmartCloud) are received a 4. S3 triggers the `workorder-email-processor` Lambda. 5. Fail-closed sender authentication (INFRA-107): the SES-stamped `Authentication-Results` header must show `dkim=pass` for the domain that re-signs the forward (currently allowlisted as `seahaven.com` — see the validation caveat under [Sender authentication](#sender-authentication-infra-107)); otherwise the email is logged and dropped. 6. **Parse:** a pure, offline template parser (`template_parser.py`) tries the two known Hexagon templates first, behind a strict fail-closed validation gate. Only on a miss/invalid result does the Lambda fall back to the Claude-on-Bedrock AI extractor. Both paths emit the identical structured-JSON contract (work order ID, site code, severity, priority, dates, assigned technician). -7. Work order upserted to `WorkOrders`, event/comment appended to `WorkOrderComments`. +7. Work order upserted to `work-orders`, event/comment appended to `work-order-comments`. **Deterministic template parser.** ~93.6% of WO traffic is the plain-text "AMAZON UPDATE WO DETAILS \" comment template (T1) and ~6.4% is the HTML "AMAZON assign Work Order \ on building \" assignment template (T2). `template_parser.try_deterministic_parse()` classifies by subject, extracts the contract fields, and returns a parsed result **only if** it passes a strict validation gate (exact contract-key set; `work_order_id` matches the subject and is all-digits; the T1 `Work Order: ` double-space is literally present; `email_type` matches the template; `site_code` shape; per-type required fields; and a label-bleed guard so a value that over-ran into another field fails). Anything that fails — the rare update/cancellation shapes, Hexagon template drift, or an extractor exception — falls back to the AI extractor. The AI path is gated too: the untrusted email reaches Bedrock inside a neutralized `` data block (tag lookalikes in the body are defanged), and the raw model output must pass the fail-closed `validate_ai_fallback()` schema/enum/date gate before any DynamoDB write — output that fails is dropped and paged (see the `ai-fallback-rejected` alarm below), a prompt-injection defence for DKIM-passing but attacker-influenced mail. **Data is never corrupted; only the fallback rate rises.** Every record emits one CloudWatch EMF metric (see below). @@ -83,12 +83,12 @@ Amazon APM work order emails (from Hexagon EAM / HxGN SmartCloud) are received a | `workorder-shoc-hmac-rotator` | Secrets Manager rotation schedule (30 days) | Rotates the webhook HMAC signing keys (dual-key overlap) | **Tables:** -- `WorkOrders` (PK: `work_order_id`, Streams: NEW_AND_OLD_IMAGES) — `site-code-index` and `status-index` GSIs removed 2026-06-03 (audit M-20) -- `WorkOrderComments` (PK: `work_order_id`, SK: `comment_id`, Streams: NEW_AND_OLD_IMAGES) — see the `comment_id` format note below +- `work-orders` (PK: `work_order_id`, Streams: NEW_AND_OLD_IMAGES) — `site-code-index` and `status-index` GSIs removed 2026-06-03 (audit M-20) +- `work-order-comments` (PK: `work_order_id`, SK: `comment_id`, Streams: NEW_AND_OLD_IMAGES) — see the `comment_id` format note below ### SHOC webhook feed (`workorder-shoc-emitter`) -**Flow:** `WorkOrders` / `WorkOrderComments` DynamoDB Streams (`NEW_AND_OLD_IMAGES` — the OLD image is what lets the emitter detect the `wo_status → cancelled` transition) → `workorder-shoc-emitter` → HMAC-signed HTTPS POST → SHOC backend. Every WO mutation becomes one webhook event (`work_order.created` / `.updated` / `.cancelled` / `.comment_added`) within seconds of the DynamoDB commit; DynamoDB stays the source of truth. +**Flow:** `work-orders` / `work-order-comments` DynamoDB Streams (`NEW_AND_OLD_IMAGES` — the OLD image is what lets the emitter detect the `wo_status → cancelled` transition) → `workorder-shoc-emitter` → HMAC-signed HTTPS POST → SHOC backend. Every WO mutation becomes one webhook event (`work_order.created` / `.updated` / `.cancelled` / `.comment_added`) within seconds of the DynamoDB commit; DynamoDB stays the source of truth. - **Contract:** `docs/shoc-webhook-contract.md` (Rev 2026-07-23) is the producer/consumer contract SHOC builds its receiver against, and `docs/shoc-webhook-test-vectors.json` is the shared receiver-verification vector set — both sides pin their HMAC implementation against the same vectors (producer-side via the golden-vector tests over `delivery.sign_body`). - **ACTIVE since 2026-07-30.** Both event-source mappings run with `enabled=True`, flipped after the SHOC receiver on `api.dev.seahaven.com` passed the shared test vectors live (valid current-kid signature accepted, duplicate `delivery_id` deduplicated, tampered/stale/unknown-kid all rejected 401). The stack originally shipped dark (`enabled=False`) so it could deploy and be tested with zero deliveries while SHOC had no receiver. The ESMs start at `LATEST` — no historical flood; SHOC backfills history through the `procurement-api` read API, not the stream. @@ -191,7 +191,7 @@ The alarm `po-email-processor-template-fallback-rate` is **deliberately retuned A second alarm, `po-email-processor-ai-fallback-rejected`, monitors the rejected series on its own — **retuned for ~57 emails/day, not WO's 5-minute sparse idiom** (which needs two rejections inside one 30-minute window and would be structurally dead at PO volume). It uses the same 6h/`IF`-floor/eval-4/datapoints-2 idiom as the fallback-rate alarm above, but as a plain count-floor on the rejected series itself (`IF(FILL(rej,0)>=1, …)`, no denominator so no divide guard is needed): threshold ≥1, over **6-hour periods**, **eval 4 / datapoints 2** — a lone stray rejection self-clears, while ≥2 rejections landing in ≥2 distinct 6h windows within 24h (sustained prompt-injection probing, or template drift whose AI output also fails the gate) pages within ~12–24h. ALARM-only `SnsAction` to `site-alerts`, `NOT_BREACHING`. Accepted residual: a single isolated rejected email never pages this alarm by itself — it is still visible as an `ai_fallback_rejected` datapoint and in the `ReasonCode` log line, and it has already raised the fallback-rate numerator above via its pre-call `ai_fallback` emit. -**DynamoDB alarms** (`AWS/DynamoDB`): each owned table gets `-throttles` (`ThrottledRequests`) and `
-system-errors` (`SystemErrors`). These metrics emit only at the `TableName` + `Operation` dimension set, so each alarm is a `Sum` math expression across the operations the table uses (Get/BatchGet/Query/Scan/Put/Update/Delete/BatchWrite). Tables covered: `purchase-orders`, `verified-sites`, `pending-site-review` (po-ingest); `WorkOrders`, `WorkOrderComments` (workorder-ingest). +**DynamoDB alarms** (`AWS/DynamoDB`): each owned table gets `
-throttles` (`ThrottledRequests`) and `
-system-errors` (`SystemErrors`). These metrics emit only at the `TableName` + `Operation` dimension set, so each alarm is a `Sum` math expression across the operations the table uses (Get/BatchGet/Query/Scan/Put/Update/Delete/BatchWrite). Tables covered: `purchase-orders`, `verified-sites`, `pending-site-review` (po-ingest); `work-orders`, `work-order-comments` (workorder-ingest). ## Deploy-Pipeline Guards (Phase 0) @@ -270,19 +270,22 @@ The `purchase-orders` DynamoDB table is **owned by this repo's Terraform config* **Known exception (INFRA-51):** `amazon-po-parser` currently writes directly to `purchase-orders` outside this stack (backfill/enrichment scripts). This second writer is being folded into the `po-ingest` pipeline so this stack is the sole writer; until INFRA-51 closes, coordinate any schema change with `amazon-po-parser` as well. -### `WorkOrders` and `WorkOrderComments` tables (owned here) +### `work-orders` and `work-order-comments` tables (owned here) + +> **PLAT-11 cutover complete 2026-08-07.** Live physical names are kebab-case. +> PascalCase `WorkOrders` / `WorkOrderComments` retained until ≥24h soak ends +> (**ready 2026-08-08T18:16Z**), then destroyed under HCP. GitHub #24 superseded +> (issues disabled on this repo); track [PLAT-11](https://seahaven.atlassian.net/browse/PLAT-11). -> **PLAT-11 in progress.** Kebab physical names `work-orders` / `work-order-comments` -> are created empty under Terraform; writers/ESMs stay on PascalCase until the > freeze cutover in [`docs/plat-11/cutover-runbook.md`](docs/plat-11/cutover-runbook.md). > After cutover, this section will name the kebab tables. Both tables are **owned by this repo's Terraform config** (`terraform/wo_ddb.tf`, retain lifecycle): -- `WorkOrders` — PK `work_order_id` (S). -- `WorkOrderComments` — PK `work_order_id` (S), SK `comment_id` (S). +- `work-orders` — PK `work_order_id` (S). +- `work-order-comments` — PK `work_order_id` (S), SK `comment_id` (S). -> **`comment_id` format change (issue #23).** The `WorkOrderComments` range key is now +> **`comment_id` format change (issue #23).** The `work-order-comments` range key is now > `work_order_id##` > (e.g. `11144580730#2026-04-27T23:51:48#a1b2c3d4e5f6`, or `…#nocomment#…` when the source > email carries no comment time). Previously it was `work_order_id#`, where two diff --git a/docs/plat-11/cutover-runbook.md b/docs/plat-11/cutover-runbook.md index 8547829..104760d 100644 --- a/docs/plat-11/cutover-runbook.md +++ b/docs/plat-11/cutover-runbook.md @@ -31,7 +31,9 @@ emitter env `WORK_ORDERS_TABLE` / `COMMENTS_TABLE` (still pointing at legacy), Writers and ESMs stay on PascalCase. -## Phase 2 — Freeze cutover +## Phase 2 — Freeze cutover (DONE 2026-08-07) + +`FREEZE_START=2026-08-07T17:54:21Z`. Copy verified 7767 / 93038. Writers+ESMs on kebab. Replay since freeze: 0 events. Record `FREEZE_START` (UTC ISO-8601) before step 1. @@ -79,7 +81,7 @@ Record `FREEZE_START` (UTC ISO-8601) before step 1. ``` And/or one SHOC reconciliation pass against procurement-api. -## Phase 3 — Decommission (after ≥24h on kebab) +## Phase 3 — Decommission (after ≥24h on kebab; ready **2026-08-08T18:16Z**) 1. Lift `prevent_destroy` on **legacy** `work_orders` / `work_order_comments` only. 2. Remove legacy table resources + their import blocks + PascalCase alarm imports. diff --git a/terraform/wo_ddb.tf b/terraform/wo_ddb.tf index d447e79..49f0ed9 100644 --- a/terraform/wo_ddb.tf +++ b/terraform/wo_ddb.tf @@ -1,5 +1,6 @@ -# PLAT-11: PascalCase tables retained until decommission soak. -# Writers/ESMs/alarms point at kebab tables after the freeze cutover. +# PLAT-11: live WO tables are kebab-case. PascalCase WorkOrders / +# WorkOrderComments retained until ≥24h soak; prevent_destroy lifted for +# the follow-up destroy apply (ready 2026-08-08T18:16Z). resource "aws_dynamodb_table" "work_orders" { name = "WorkOrders" @@ -15,7 +16,7 @@ resource "aws_dynamodb_table" "work_orders" { stream_view_type = "NEW_AND_OLD_IMAGES" lifecycle { - prevent_destroy = true + prevent_destroy = false } } @@ -39,7 +40,7 @@ resource "aws_dynamodb_table" "work_order_comments" { stream_view_type = "NEW_AND_OLD_IMAGES" lifecycle { - prevent_destroy = true + prevent_destroy = false } }