mirror of
https://github.com/Sea-Haven-Industries/procurement-ingest.git
synced 2026-09-30 11:53:13 +00:00
chore(infra): lift prevent_destroy on legacy WO tables for PLAT-11
Writers already use kebab tables. Allow HCP to destroy PascalCase WorkOrders / WorkOrderComments after the ≥24h kebab soak. Docs scrub to kebab as the live physical names; GitHub #24 noted superseded.
This commit is contained in:
parent
ebd5c40a34
commit
8a58584c12
3 changed files with 24 additions and 18 deletions
27
README.md
27
README.md
|
|
@ -61,7 +61,7 @@ Or aggregate overall agreement per field: `| filter ispresent(DerivedFieldAgreem
|
|||
|
||||
### Work Orders (`WorkorderIngestStack` stack)
|
||||
|
||||
Amazon APM work order emails (from Hexagon EAM / HxGN SmartCloud) are received at `apm@int.seahaven.com`, parsed **deterministic-template-first with a Claude-on-Bedrock fallback**, and written to the `WorkOrders` DynamoDB table.
|
||||
Amazon APM work order emails (from Hexagon EAM / HxGN SmartCloud) are received at `apm@int.seahaven.com`, parsed **deterministic-template-first with a Claude-on-Bedrock fallback**, and written to the `work-orders` DynamoDB table.
|
||||
|
||||
**Flow:**
|
||||
1. Hexagon EAM sends email notifications (new assignments, comments, updates, cancellations) to `amazon@seahavenind.com`.
|
||||
|
|
@ -70,7 +70,7 @@ Amazon APM work order emails (from Hexagon EAM / HxGN SmartCloud) are received a
|
|||
4. S3 triggers the `workorder-email-processor` Lambda.
|
||||
5. Fail-closed sender authentication (INFRA-107): the SES-stamped `Authentication-Results` header must show `dkim=pass` for the domain that re-signs the forward (currently allowlisted as `seahaven.com` — see the validation caveat under [Sender authentication](#sender-authentication-infra-107)); otherwise the email is logged and dropped.
|
||||
6. **Parse:** a pure, offline template parser (`template_parser.py`) tries the two known Hexagon templates first, behind a strict fail-closed validation gate. Only on a miss/invalid result does the Lambda fall back to the Claude-on-Bedrock AI extractor. Both paths emit the identical structured-JSON contract (work order ID, site code, severity, priority, dates, assigned technician).
|
||||
7. Work order upserted to `WorkOrders`, event/comment appended to `WorkOrderComments`.
|
||||
7. Work order upserted to `work-orders`, event/comment appended to `work-order-comments`.
|
||||
|
||||
**Deterministic template parser.** ~93.6% of WO traffic is the plain-text "AMAZON UPDATE WO DETAILS \<id\>" comment template (T1) and ~6.4% is the HTML "AMAZON assign Work Order \<id\> on building \<SITE\>" assignment template (T2). `template_parser.try_deterministic_parse()` classifies by subject, extracts the contract fields, and returns a parsed result **only if** it passes a strict validation gate (exact contract-key set; `work_order_id` matches the subject and is all-digits; the T1 `Work Order: <id>` double-space is literally present; `email_type` matches the template; `site_code` shape; per-type required fields; and a label-bleed guard so a value that over-ran into another field fails). Anything that fails — the rare update/cancellation shapes, Hexagon template drift, or an extractor exception — falls back to the AI extractor. The AI path is gated too: the untrusted email reaches Bedrock inside a neutralized `<email>` data block (tag lookalikes in the body are defanged), and the raw model output must pass the fail-closed `validate_ai_fallback()` schema/enum/date gate before any DynamoDB write — output that fails is dropped and paged (see the `ai-fallback-rejected` alarm below), a prompt-injection defence for DKIM-passing but attacker-influenced mail. **Data is never corrupted; only the fallback rate rises.** Every record emits one CloudWatch EMF metric (see below).
|
||||
|
||||
|
|
@ -83,12 +83,12 @@ Amazon APM work order emails (from Hexagon EAM / HxGN SmartCloud) are received a
|
|||
| `workorder-shoc-hmac-rotator` | Secrets Manager rotation schedule (30 days) | Rotates the webhook HMAC signing keys (dual-key overlap) |
|
||||
|
||||
**Tables:**
|
||||
- `WorkOrders` (PK: `work_order_id`, Streams: NEW_AND_OLD_IMAGES) — `site-code-index` and `status-index` GSIs removed 2026-06-03 (audit M-20)
|
||||
- `WorkOrderComments` (PK: `work_order_id`, SK: `comment_id`, Streams: NEW_AND_OLD_IMAGES) — see the `comment_id` format note below
|
||||
- `work-orders` (PK: `work_order_id`, Streams: NEW_AND_OLD_IMAGES) — `site-code-index` and `status-index` GSIs removed 2026-06-03 (audit M-20)
|
||||
- `work-order-comments` (PK: `work_order_id`, SK: `comment_id`, Streams: NEW_AND_OLD_IMAGES) — see the `comment_id` format note below
|
||||
|
||||
### SHOC webhook feed (`workorder-shoc-emitter`)
|
||||
|
||||
**Flow:** `WorkOrders` / `WorkOrderComments` DynamoDB Streams (`NEW_AND_OLD_IMAGES` — the OLD image is what lets the emitter detect the `wo_status → cancelled` transition) → `workorder-shoc-emitter` → HMAC-signed HTTPS POST → SHOC backend. Every WO mutation becomes one webhook event (`work_order.created` / `.updated` / `.cancelled` / `.comment_added`) within seconds of the DynamoDB commit; DynamoDB stays the source of truth.
|
||||
**Flow:** `work-orders` / `work-order-comments` DynamoDB Streams (`NEW_AND_OLD_IMAGES` — the OLD image is what lets the emitter detect the `wo_status → cancelled` transition) → `workorder-shoc-emitter` → HMAC-signed HTTPS POST → SHOC backend. Every WO mutation becomes one webhook event (`work_order.created` / `.updated` / `.cancelled` / `.comment_added`) within seconds of the DynamoDB commit; DynamoDB stays the source of truth.
|
||||
|
||||
- **Contract:** `docs/shoc-webhook-contract.md` (Rev 2026-07-23) is the producer/consumer contract SHOC builds its receiver against, and `docs/shoc-webhook-test-vectors.json` is the shared receiver-verification vector set — both sides pin their HMAC implementation against the same vectors (producer-side via the golden-vector tests over `delivery.sign_body`).
|
||||
- **ACTIVE since 2026-07-30.** Both event-source mappings run with `enabled=True`, flipped after the SHOC receiver on `api.dev.seahaven.com` passed the shared test vectors live (valid current-kid signature accepted, duplicate `delivery_id` deduplicated, tampered/stale/unknown-kid all rejected 401). The stack originally shipped dark (`enabled=False`) so it could deploy and be tested with zero deliveries while SHOC had no receiver. The ESMs start at `LATEST` — no historical flood; SHOC backfills history through the `procurement-api` read API, not the stream.
|
||||
|
|
@ -191,7 +191,7 @@ The alarm `po-email-processor-template-fallback-rate` is **deliberately retuned
|
|||
|
||||
A second alarm, `po-email-processor-ai-fallback-rejected`, monitors the rejected series on its own — **retuned for ~57 emails/day, not WO's 5-minute sparse idiom** (which needs two rejections inside one 30-minute window and would be structurally dead at PO volume). It uses the same 6h/`IF`-floor/eval-4/datapoints-2 idiom as the fallback-rate alarm above, but as a plain count-floor on the rejected series itself (`IF(FILL(rej,0)>=1, …)`, no denominator so no divide guard is needed): threshold ≥1, over **6-hour periods**, **eval 4 / datapoints 2** — a lone stray rejection self-clears, while ≥2 rejections landing in ≥2 distinct 6h windows within 24h (sustained prompt-injection probing, or template drift whose AI output also fails the gate) pages within ~12–24h. ALARM-only `SnsAction` to `site-alerts`, `NOT_BREACHING`. Accepted residual: a single isolated rejected email never pages this alarm by itself — it is still visible as an `ai_fallback_rejected` datapoint and in the `ReasonCode` log line, and it has already raised the fallback-rate numerator above via its pre-call `ai_fallback` emit.
|
||||
|
||||
**DynamoDB alarms** (`AWS/DynamoDB`): each owned table gets `<table>-throttles` (`ThrottledRequests`) and `<table>-system-errors` (`SystemErrors`). These metrics emit only at the `TableName` + `Operation` dimension set, so each alarm is a `Sum` math expression across the operations the table uses (Get/BatchGet/Query/Scan/Put/Update/Delete/BatchWrite). Tables covered: `purchase-orders`, `verified-sites`, `pending-site-review` (po-ingest); `WorkOrders`, `WorkOrderComments` (workorder-ingest).
|
||||
**DynamoDB alarms** (`AWS/DynamoDB`): each owned table gets `<table>-throttles` (`ThrottledRequests`) and `<table>-system-errors` (`SystemErrors`). These metrics emit only at the `TableName` + `Operation` dimension set, so each alarm is a `Sum` math expression across the operations the table uses (Get/BatchGet/Query/Scan/Put/Update/Delete/BatchWrite). Tables covered: `purchase-orders`, `verified-sites`, `pending-site-review` (po-ingest); `work-orders`, `work-order-comments` (workorder-ingest).
|
||||
|
||||
## Deploy-Pipeline Guards (Phase 0)
|
||||
|
||||
|
|
@ -270,19 +270,22 @@ The `purchase-orders` DynamoDB table is **owned by this repo's Terraform config*
|
|||
|
||||
**Known exception (INFRA-51):** `amazon-po-parser` currently writes directly to `purchase-orders` outside this stack (backfill/enrichment scripts). This second writer is being folded into the `po-ingest` pipeline so this stack is the sole writer; until INFRA-51 closes, coordinate any schema change with `amazon-po-parser` as well.
|
||||
|
||||
### `WorkOrders` and `WorkOrderComments` tables (owned here)
|
||||
### `work-orders` and `work-order-comments` tables (owned here)
|
||||
|
||||
> **PLAT-11 cutover complete 2026-08-07.** Live physical names are kebab-case.
|
||||
> PascalCase `WorkOrders` / `WorkOrderComments` retained until ≥24h soak ends
|
||||
> (**ready 2026-08-08T18:16Z**), then destroyed under HCP. GitHub #24 superseded
|
||||
> (issues disabled on this repo); track [PLAT-11](https://seahaven.atlassian.net/browse/PLAT-11).
|
||||
|
||||
> **PLAT-11 in progress.** Kebab physical names `work-orders` / `work-order-comments`
|
||||
> are created empty under Terraform; writers/ESMs stay on PascalCase until the
|
||||
> freeze cutover in [`docs/plat-11/cutover-runbook.md`](docs/plat-11/cutover-runbook.md).
|
||||
> After cutover, this section will name the kebab tables.
|
||||
|
||||
Both tables are **owned by this repo's Terraform config** (`terraform/wo_ddb.tf`, retain lifecycle):
|
||||
|
||||
- `WorkOrders` — PK `work_order_id` (S).
|
||||
- `WorkOrderComments` — PK `work_order_id` (S), SK `comment_id` (S).
|
||||
- `work-orders` — PK `work_order_id` (S).
|
||||
- `work-order-comments` — PK `work_order_id` (S), SK `comment_id` (S).
|
||||
|
||||
> **`comment_id` format change (issue #23).** The `WorkOrderComments` range key is now
|
||||
> **`comment_id` format change (issue #23).** The `work-order-comments` range key is now
|
||||
> `work_order_id#<comment_time|nocomment>#<sha256(s3_object_key)[:12]>`
|
||||
> (e.g. `11144580730#2026-04-27T23:51:48#a1b2c3d4e5f6`, or `…#nocomment#…` when the source
|
||||
> email carries no comment time). Previously it was `work_order_id#<timestamp>`, where two
|
||||
|
|
|
|||
|
|
@ -31,7 +31,9 @@ emitter env `WORK_ORDERS_TABLE` / `COMMENTS_TABLE` (still pointing at legacy),
|
|||
|
||||
Writers and ESMs stay on PascalCase.
|
||||
|
||||
## Phase 2 — Freeze cutover
|
||||
## Phase 2 — Freeze cutover (DONE 2026-08-07)
|
||||
|
||||
`FREEZE_START=2026-08-07T17:54:21Z`. Copy verified 7767 / 93038. Writers+ESMs on kebab. Replay since freeze: 0 events.
|
||||
|
||||
Record `FREEZE_START` (UTC ISO-8601) before step 1.
|
||||
|
||||
|
|
@ -79,7 +81,7 @@ Record `FREEZE_START` (UTC ISO-8601) before step 1.
|
|||
```
|
||||
And/or one SHOC reconciliation pass against procurement-api.
|
||||
|
||||
## Phase 3 — Decommission (after ≥24h on kebab)
|
||||
## Phase 3 — Decommission (after ≥24h on kebab; ready **2026-08-08T18:16Z**)
|
||||
|
||||
1. Lift `prevent_destroy` on **legacy** `work_orders` / `work_order_comments` only.
|
||||
2. Remove legacy table resources + their import blocks + PascalCase alarm imports.
|
||||
|
|
|
|||
|
|
@ -1,5 +1,6 @@
|
|||
# PLAT-11: PascalCase tables retained until decommission soak.
|
||||
# Writers/ESMs/alarms point at kebab tables after the freeze cutover.
|
||||
# PLAT-11: live WO tables are kebab-case. PascalCase WorkOrders /
|
||||
# WorkOrderComments retained until ≥24h soak; prevent_destroy lifted for
|
||||
# the follow-up destroy apply (ready 2026-08-08T18:16Z).
|
||||
|
||||
resource "aws_dynamodb_table" "work_orders" {
|
||||
name = "WorkOrders"
|
||||
|
|
@ -15,7 +16,7 @@ resource "aws_dynamodb_table" "work_orders" {
|
|||
stream_view_type = "NEW_AND_OLD_IMAGES"
|
||||
|
||||
lifecycle {
|
||||
prevent_destroy = true
|
||||
prevent_destroy = false
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -39,7 +40,7 @@ resource "aws_dynamodb_table" "work_order_comments" {
|
|||
stream_view_type = "NEW_AND_OLD_IMAGES"
|
||||
|
||||
lifecycle {
|
||||
prevent_destroy = true
|
||||
prevent_destroy = false
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue