Compare commits

...

4 commits

Author SHA1 Message Date
Adam Moussa
5464d66d19
chore(infra): tag legacy WO tables pending PLAT-11 destroy (#177)
Force an in-place DynamoDB update so HCP state records
prevent_destroy=false before the post-soak resource removal.
2026-08-07 14:44:16 -04:00
Adam Moussa
7d0f272901
chore(infra): lift prevent_destroy on legacy WO tables for PLAT-11 (#176)
Writers already use kebab tables. Allow HCP to destroy PascalCase
WorkOrders / WorkOrderComments after the ≥24h kebab soak. Docs scrub
to kebab as the live physical names; GitHub #24 noted superseded.
2026-08-07 14:26:44 -04:00
Adam Moussa
ebd5c40a34
fix(infra): import kebab-stream SHOC emitter ESMs after cutover recovery (#175)
Cutover apply updated Lambda env/IAM but ESM recreate failed until the
lambda execution boundary included kebab table ARNs. Import the
out-of-band Enabled mappings so HCP state matches live.
2026-08-07 14:20:01 -04:00
Adam Moussa
419293b8ed
feat(infra): retarget WO writers and ESMs to kebab tables (PLAT-11) (#174)
* feat(infra): retarget WO writers and ESMs to kebab Dynamo tables

Point Lambda env, IAM, emitter streams, and DDB alarms at work-orders /
work-order-comments for the PLAT-11 freeze cutover. Legacy PascalCase
tables remain in state until the decommission soak.

* style(infra): terraform fmt wo_ddb alarm map alignment
2026-08-07 14:07:57 -04:00
8 changed files with 66 additions and 69 deletions

View file

@ -61,7 +61,7 @@ Or aggregate overall agreement per field: `| filter ispresent(DerivedFieldAgreem
### Work Orders (`WorkorderIngestStack` stack)
Amazon APM work order emails (from Hexagon EAM / HxGN SmartCloud) are received at `apm@int.seahaven.com`, parsed **deterministic-template-first with a Claude-on-Bedrock fallback**, and written to the `WorkOrders` DynamoDB table.
Amazon APM work order emails (from Hexagon EAM / HxGN SmartCloud) are received at `apm@int.seahaven.com`, parsed **deterministic-template-first with a Claude-on-Bedrock fallback**, and written to the `work-orders` DynamoDB table.
**Flow:**
1. Hexagon EAM sends email notifications (new assignments, comments, updates, cancellations) to `amazon@seahavenind.com`.
@ -70,7 +70,7 @@ Amazon APM work order emails (from Hexagon EAM / HxGN SmartCloud) are received a
4. S3 triggers the `workorder-email-processor` Lambda.
5. Fail-closed sender authentication (INFRA-107): the SES-stamped `Authentication-Results` header must show `dkim=pass` for the domain that re-signs the forward (currently allowlisted as `seahaven.com` — see the validation caveat under [Sender authentication](#sender-authentication-infra-107)); otherwise the email is logged and dropped.
6. **Parse:** a pure, offline template parser (`template_parser.py`) tries the two known Hexagon templates first, behind a strict fail-closed validation gate. Only on a miss/invalid result does the Lambda fall back to the Claude-on-Bedrock AI extractor. Both paths emit the identical structured-JSON contract (work order ID, site code, severity, priority, dates, assigned technician).
7. Work order upserted to `WorkOrders`, event/comment appended to `WorkOrderComments`.
7. Work order upserted to `work-orders`, event/comment appended to `work-order-comments`.
**Deterministic template parser.** ~93.6% of WO traffic is the plain-text "AMAZON UPDATE WO DETAILS \<id\>" comment template (T1) and ~6.4% is the HTML "AMAZON assign Work Order \<id\> on building \<SITE\>" assignment template (T2). `template_parser.try_deterministic_parse()` classifies by subject, extracts the contract fields, and returns a parsed result **only if** it passes a strict validation gate (exact contract-key set; `work_order_id` matches the subject and is all-digits; the T1 `Work Order: <id>` double-space is literally present; `email_type` matches the template; `site_code` shape; per-type required fields; and a label-bleed guard so a value that over-ran into another field fails). Anything that fails — the rare update/cancellation shapes, Hexagon template drift, or an extractor exception — falls back to the AI extractor. The AI path is gated too: the untrusted email reaches Bedrock inside a neutralized `<email>` data block (tag lookalikes in the body are defanged), and the raw model output must pass the fail-closed `validate_ai_fallback()` schema/enum/date gate before any DynamoDB write — output that fails is dropped and paged (see the `ai-fallback-rejected` alarm below), a prompt-injection defence for DKIM-passing but attacker-influenced mail. **Data is never corrupted; only the fallback rate rises.** Every record emits one CloudWatch EMF metric (see below).
@ -83,12 +83,12 @@ Amazon APM work order emails (from Hexagon EAM / HxGN SmartCloud) are received a
| `workorder-shoc-hmac-rotator` | Secrets Manager rotation schedule (30 days) | Rotates the webhook HMAC signing keys (dual-key overlap) |
**Tables:**
- `WorkOrders` (PK: `work_order_id`, Streams: NEW_AND_OLD_IMAGES) — `site-code-index` and `status-index` GSIs removed 2026-06-03 (audit M-20)
- `WorkOrderComments` (PK: `work_order_id`, SK: `comment_id`, Streams: NEW_AND_OLD_IMAGES) — see the `comment_id` format note below
- `work-orders` (PK: `work_order_id`, Streams: NEW_AND_OLD_IMAGES) — `site-code-index` and `status-index` GSIs removed 2026-06-03 (audit M-20)
- `work-order-comments` (PK: `work_order_id`, SK: `comment_id`, Streams: NEW_AND_OLD_IMAGES) — see the `comment_id` format note below
### SHOC webhook feed (`workorder-shoc-emitter`)
**Flow:** `WorkOrders` / `WorkOrderComments` DynamoDB Streams (`NEW_AND_OLD_IMAGES` — the OLD image is what lets the emitter detect the `wo_status → cancelled` transition) → `workorder-shoc-emitter` → HMAC-signed HTTPS POST → SHOC backend. Every WO mutation becomes one webhook event (`work_order.created` / `.updated` / `.cancelled` / `.comment_added`) within seconds of the DynamoDB commit; DynamoDB stays the source of truth.
**Flow:** `work-orders` / `work-order-comments` DynamoDB Streams (`NEW_AND_OLD_IMAGES` — the OLD image is what lets the emitter detect the `wo_status → cancelled` transition) → `workorder-shoc-emitter` → HMAC-signed HTTPS POST → SHOC backend. Every WO mutation becomes one webhook event (`work_order.created` / `.updated` / `.cancelled` / `.comment_added`) within seconds of the DynamoDB commit; DynamoDB stays the source of truth.
- **Contract:** `docs/shoc-webhook-contract.md` (Rev 2026-07-23) is the producer/consumer contract SHOC builds its receiver against, and `docs/shoc-webhook-test-vectors.json` is the shared receiver-verification vector set — both sides pin their HMAC implementation against the same vectors (producer-side via the golden-vector tests over `delivery.sign_body`).
- **ACTIVE since 2026-07-30.** Both event-source mappings run with `enabled=True`, flipped after the SHOC receiver on `api.dev.seahaven.com` passed the shared test vectors live (valid current-kid signature accepted, duplicate `delivery_id` deduplicated, tampered/stale/unknown-kid all rejected 401). The stack originally shipped dark (`enabled=False`) so it could deploy and be tested with zero deliveries while SHOC had no receiver. The ESMs start at `LATEST` — no historical flood; SHOC backfills history through the `procurement-api` read API, not the stream.
@ -191,7 +191,7 @@ The alarm `po-email-processor-template-fallback-rate` is **deliberately retuned
A second alarm, `po-email-processor-ai-fallback-rejected`, monitors the rejected series on its own — **retuned for ~57 emails/day, not WO's 5-minute sparse idiom** (which needs two rejections inside one 30-minute window and would be structurally dead at PO volume). It uses the same 6h/`IF`-floor/eval-4/datapoints-2 idiom as the fallback-rate alarm above, but as a plain count-floor on the rejected series itself (`IF(FILL(rej,0)>=1, …)`, no denominator so no divide guard is needed): threshold ≥1, over **6-hour periods**, **eval 4 / datapoints 2** — a lone stray rejection self-clears, while ≥2 rejections landing in ≥2 distinct 6h windows within 24h (sustained prompt-injection probing, or template drift whose AI output also fails the gate) pages within ~12–24h. ALARM-only `SnsAction` to `site-alerts`, `NOT_BREACHING`. Accepted residual: a single isolated rejected email never pages this alarm by itself — it is still visible as an `ai_fallback_rejected` datapoint and in the `ReasonCode` log line, and it has already raised the fallback-rate numerator above via its pre-call `ai_fallback` emit.
**DynamoDB alarms** (`AWS/DynamoDB`): each owned table gets `<table>-throttles` (`ThrottledRequests`) and `<table>-system-errors` (`SystemErrors`). These metrics emit only at the `TableName` + `Operation` dimension set, so each alarm is a `Sum` math expression across the operations the table uses (Get/BatchGet/Query/Scan/Put/Update/Delete/BatchWrite). Tables covered: `purchase-orders`, `verified-sites`, `pending-site-review` (po-ingest); `WorkOrders`, `WorkOrderComments` (workorder-ingest).
**DynamoDB alarms** (`AWS/DynamoDB`): each owned table gets `<table>-throttles` (`ThrottledRequests`) and `<table>-system-errors` (`SystemErrors`). These metrics emit only at the `TableName` + `Operation` dimension set, so each alarm is a `Sum` math expression across the operations the table uses (Get/BatchGet/Query/Scan/Put/Update/Delete/BatchWrite). Tables covered: `purchase-orders`, `verified-sites`, `pending-site-review` (po-ingest); `work-orders`, `work-order-comments` (workorder-ingest).
## Deploy-Pipeline Guards (Phase 0)
@ -270,19 +270,22 @@ The `purchase-orders` DynamoDB table is **owned by this repo's Terraform config*
**Known exception (INFRA-51):** `amazon-po-parser` currently writes directly to `purchase-orders` outside this stack (backfill/enrichment scripts). This second writer is being folded into the `po-ingest` pipeline so this stack is the sole writer; until INFRA-51 closes, coordinate any schema change with `amazon-po-parser` as well.
### `WorkOrders` and `WorkOrderComments` tables (owned here)
### `work-orders` and `work-order-comments` tables (owned here)
> **PLAT-11 cutover complete 2026-08-07.** Live physical names are kebab-case.
> PascalCase `WorkOrders` / `WorkOrderComments` retained until ≥24h soak ends
> (**ready 2026-08-08T18:16Z**), then destroyed under HCP. GitHub #24 superseded
> (issues disabled on this repo); track [PLAT-11](https://seahaven.atlassian.net/browse/PLAT-11).
> **PLAT-11 in progress.** Kebab physical names `work-orders` / `work-order-comments`
> are created empty under Terraform; writers/ESMs stay on PascalCase until the
> freeze cutover in [`docs/plat-11/cutover-runbook.md`](docs/plat-11/cutover-runbook.md).
> After cutover, this section will name the kebab tables.
Both tables are **owned by this repo's Terraform config** (`terraform/wo_ddb.tf`, retain lifecycle):
- `WorkOrders` — PK `work_order_id` (S).
- `WorkOrderComments` — PK `work_order_id` (S), SK `comment_id` (S).
- `work-orders` — PK `work_order_id` (S).
- `work-order-comments` — PK `work_order_id` (S), SK `comment_id` (S).
> **`comment_id` format change (issue #23).** The `WorkOrderComments` range key is now
> **`comment_id` format change (issue #23).** The `work-order-comments` range key is now
> `work_order_id#<comment_time|nocomment>#<sha256(s3_object_key)[:12]>`
> (e.g. `11144580730#2026-04-27T23:51:48#a1b2c3d4e5f6`, or `…#nocomment#…` when the source
> email carries no comment time). Previously it was `work_order_id#<timestamp>`, where two

View file

@ -31,7 +31,9 @@ emitter env `WORK_ORDERS_TABLE` / `COMMENTS_TABLE` (still pointing at legacy),
Writers and ESMs stay on PascalCase.
## Phase 2 — Freeze cutover
## Phase 2 — Freeze cutover (DONE 2026-08-07)
`FREEZE_START=2026-08-07T17:54:21Z`. Copy verified 7767 / 93038. Writers+ESMs on kebab. Replay since freeze: 0 events.
Record `FREEZE_START` (UTC ISO-8601) before step 1.
@ -79,7 +81,7 @@ Record `FREEZE_START` (UTC ISO-8601) before step 1.
```
And/or one SHOC reconciliation pass against procurement-api.
## Phase 3 — Decommission (after ≥24h on kebab)
## Phase 3 — Decommission (after ≥24h on kebab; ready **2026-08-08T18:16Z**)
1. Lift `prevent_destroy` on **legacy** `work_orders` / `work_order_comments` only.
2. Remove legacy table resources + their import blocks + PascalCase alarm imports.

View file

@ -123,8 +123,8 @@ resource "aws_lambda_function" "procurement_api" {
environment {
variables = {
VERIFIED_SITES_TABLE = aws_dynamodb_table.verified_sites.name
WORK_ORDERS_TABLE = aws_dynamodb_table.work_orders.name
COMMENTS_TABLE = aws_dynamodb_table.work_order_comments.name
WORK_ORDERS_TABLE = aws_dynamodb_table.work_orders_kebab.name
COMMENTS_TABLE = aws_dynamodb_table.work_order_comments_kebab.name
PO_TABLE = aws_dynamodb_table.purchase_orders.name
WEB_UI_AUTH_TOKEN_SECRET_ARN = var.web_ui_auth_token_secret_arn
}

View file

@ -357,10 +357,10 @@ resource "aws_iam_role_policy" "wo_email_processor_ddb" {
"dynamodb:UpdateItem",
]
Resource = [
aws_dynamodb_table.work_orders.arn,
"${aws_dynamodb_table.work_orders.arn}/index/*",
aws_dynamodb_table.work_order_comments.arn,
"${aws_dynamodb_table.work_order_comments.arn}/index/*",
aws_dynamodb_table.work_orders_kebab.arn,
"${aws_dynamodb_table.work_orders_kebab.arn}/index/*",
aws_dynamodb_table.work_order_comments_kebab.arn,
"${aws_dynamodb_table.work_order_comments_kebab.arn}/index/*",
]
}
]
@ -421,10 +421,10 @@ resource "aws_iam_role_policy" "wo_web_ui_ddb" {
"dynamodb:Scan",
]
Resource = [
aws_dynamodb_table.work_orders.arn,
"${aws_dynamodb_table.work_orders.arn}/index/*",
aws_dynamodb_table.work_order_comments.arn,
"${aws_dynamodb_table.work_order_comments.arn}/index/*",
aws_dynamodb_table.work_orders_kebab.arn,
"${aws_dynamodb_table.work_orders_kebab.arn}/index/*",
aws_dynamodb_table.work_order_comments_kebab.arn,
"${aws_dynamodb_table.work_order_comments_kebab.arn}/index/*",
]
}
]
@ -468,16 +468,16 @@ resource "aws_iam_role_policy" "wo_shoc_emitter_streams" {
"dynamodb:ListStreams",
]
Resource = [
aws_dynamodb_table.work_orders.stream_arn,
aws_dynamodb_table.work_order_comments.stream_arn,
aws_dynamodb_table.work_orders_kebab.stream_arn,
aws_dynamodb_table.work_order_comments_kebab.stream_arn,
]
},
{
Effect = "Allow"
Action = ["dynamodb:DescribeTable"]
Resource = [
aws_dynamodb_table.work_orders.arn,
aws_dynamodb_table.work_order_comments.arn,
aws_dynamodb_table.work_orders_kebab.arn,
aws_dynamodb_table.work_order_comments_kebab.arn,
]
}
]
@ -635,10 +635,10 @@ resource "aws_iam_role_policy" "procurement_api_ddb" {
"${aws_dynamodb_table.purchase_orders.arn}/index/*",
aws_dynamodb_table.verified_sites.arn,
"${aws_dynamodb_table.verified_sites.arn}/index/*",
aws_dynamodb_table.work_orders.arn,
"${aws_dynamodb_table.work_orders.arn}/index/*",
aws_dynamodb_table.work_order_comments.arn,
"${aws_dynamodb_table.work_order_comments.arn}/index/*",
aws_dynamodb_table.work_orders_kebab.arn,
"${aws_dynamodb_table.work_orders_kebab.arn}/index/*",
aws_dynamodb_table.work_order_comments_kebab.arn,
"${aws_dynamodb_table.work_order_comments_kebab.arn}/index/*",
]
}
]

View file

@ -120,12 +120,12 @@ import {
import {
to = aws_lambda_event_source_mapping.wo_shoc_emitter_work_orders
id = "cc149f4d-5375-4820-8fb8-c514accd3a85"
id = "9cde1138-f108-4e9d-8696-cc49f9ee7b6b"
}
import {
to = aws_lambda_event_source_mapping.wo_shoc_emitter_comments
id = "da37d4a0-3086-4929-9ace-6eca5c20cd07"
id = "afe9b559-9f5d-4b1c-875b-09348a3bab67"
}
# --- Log groups ---
@ -570,25 +570,9 @@ import {
id = "workorder-shoc-emitter-rejected-messages"
}
import {
to = aws_cloudwatch_metric_alarm.wo_ddb_throttles["WorkOrders"]
id = "WorkOrders-throttles"
}
import {
to = aws_cloudwatch_metric_alarm.wo_ddb_system_errors["WorkOrders"]
id = "WorkOrders-system-errors"
}
import {
to = aws_cloudwatch_metric_alarm.wo_ddb_throttles["WorkOrderComments"]
id = "WorkOrderComments-throttles"
}
import {
to = aws_cloudwatch_metric_alarm.wo_ddb_system_errors["WorkOrderComments"]
id = "WorkOrderComments-system-errors"
}
# PLAT-11 cutover: PascalCase WO DDB alarm imports removed when alarm map
# keys moved to work-orders / work-order-comments. Legacy alarm resources are
# destroyed by the cutover apply; kebab alarms are created fresh.
# --- CloudWatch alarms (API) ---
import {

View file

@ -1,6 +1,6 @@
# PLAT-11: PascalCase tables remain authoritative until the freeze cutover.
# Kebab tables are created empty in the prep apply; writers/ESMs stay on legacy
# until docs/plat-11/cutover-runbook.md Phase 2.
# PLAT-11: live WO tables are kebab-case. PascalCase WorkOrders /
# WorkOrderComments retained until ≥24h soak; prevent_destroy lifted for
# the follow-up destroy apply (ready 2026-08-08T18:16Z).
resource "aws_dynamodb_table" "work_orders" {
name = "WorkOrders"
@ -15,8 +15,12 @@ resource "aws_dynamodb_table" "work_orders" {
stream_enabled = true
stream_view_type = "NEW_AND_OLD_IMAGES"
tags = {
Plat11 = "legacy-pending-destroy"
}
lifecycle {
prevent_destroy = true
prevent_destroy = false
}
}
@ -39,8 +43,12 @@ resource "aws_dynamodb_table" "work_order_comments" {
stream_enabled = true
stream_view_type = "NEW_AND_OLD_IMAGES"
tags = {
Plat11 = "legacy-pending-destroy"
}
lifecycle {
prevent_destroy = true
prevent_destroy = false
}
}
@ -87,10 +95,10 @@ resource "aws_dynamodb_table" "work_order_comments_kebab" {
}
locals {
# Alarms stay on the live writer tables (legacy until cutover).
# Alarms follow live writer tables after PLAT-11 cutover.
wo_ddb_alarm_tables = {
"WorkOrders" = aws_dynamodb_table.work_orders.name
"WorkOrderComments" = aws_dynamodb_table.work_order_comments.name
"work-orders" = aws_dynamodb_table.work_orders_kebab.name
"work-order-comments" = aws_dynamodb_table.work_order_comments_kebab.name
}
}

View file

@ -35,8 +35,8 @@ resource "aws_lambda_function" "wo_email_processor" {
environment {
variables = {
WORK_ORDERS_TABLE = aws_dynamodb_table.work_orders.name
COMMENTS_TABLE = aws_dynamodb_table.work_order_comments.name
WORK_ORDERS_TABLE = aws_dynamodb_table.work_orders_kebab.name
COMMENTS_TABLE = aws_dynamodb_table.work_order_comments_kebab.name
ALLOWED_DKIM_DOMAINS = "seahaven.com"
BEDROCK_MODEL_ID = local.bedrock_model_id
}
@ -68,8 +68,8 @@ resource "aws_lambda_function" "wo_web_ui" {
environment {
variables = {
WORK_ORDERS_TABLE = aws_dynamodb_table.work_orders.name
COMMENTS_TABLE = aws_dynamodb_table.work_order_comments.name
WORK_ORDERS_TABLE = aws_dynamodb_table.work_orders_kebab.name
COMMENTS_TABLE = aws_dynamodb_table.work_order_comments_kebab.name
WEB_UI_AUTH_TOKEN_SECRET_ARN = var.web_ui_auth_token_secret_arn
}
}

View file

@ -293,8 +293,8 @@ resource "aws_lambda_function" "wo_shoc_emitter" {
REJECTED_QUEUE_URL = aws_sqs_queue.shoc_emitter_rejected.url
SHOC_WEBHOOK_URL = var.shoc_webhook_url
# Stream ARN classification (PLAT-11); must match ESM source table names.
WORK_ORDERS_TABLE = aws_dynamodb_table.work_orders.name
COMMENTS_TABLE = aws_dynamodb_table.work_order_comments.name
WORK_ORDERS_TABLE = aws_dynamodb_table.work_orders_kebab.name
COMMENTS_TABLE = aws_dynamodb_table.work_order_comments_kebab.name
}
}
@ -310,7 +310,7 @@ resource "aws_lambda_function" "wo_shoc_emitter" {
}
resource "aws_lambda_event_source_mapping" "wo_shoc_emitter_work_orders" {
event_source_arn = aws_dynamodb_table.work_orders.stream_arn
event_source_arn = aws_dynamodb_table.work_orders_kebab.stream_arn
function_name = aws_lambda_function.wo_shoc_emitter.arn
starting_position = "LATEST"
batch_size = 10
@ -329,7 +329,7 @@ resource "aws_lambda_event_source_mapping" "wo_shoc_emitter_work_orders" {
}
resource "aws_lambda_event_source_mapping" "wo_shoc_emitter_comments" {
event_source_arn = aws_dynamodb_table.work_order_comments.stream_arn
event_source_arn = aws_dynamodb_table.work_order_comments_kebab.stream_arn
function_name = aws_lambda_function.wo_shoc_emitter.arn
starting_position = "LATEST"
batch_size = 10