diff --git a/README.md b/README.md index 84a04e3..5af2777 100644 --- a/README.md +++ b/README.md @@ -10,13 +10,13 @@ Unified email ingestion pipelines for Amazon procurement data. Two independent p ### Purchase Orders (`po-ingest` stack) -Coupa PO emails are received at `amazon_po@int.seahaven.com`, parsed by Claude Haiku 4.5, and written to the `purchase-orders` DynamoDB table. +Coupa PO emails are received at `amazon_po@int.seahaven.com`, parsed by Claude Haiku 4.5 **on Amazon Bedrock**, and written to the `purchase-orders` DynamoDB table. PO parsing is fully AI (no deterministic parser). **Flow:** 1. Coupa sends a PO email (new, revision, or cancellation). 2. SES (`INBOUND_MAIL` rule set) drops the raw MIME into `s3://po-ingest-emails-{AccountId}/inbound/`. 3. S3 `ObjectCreated` triggers the `po-email-processor` Lambda. -4. Claude extracts structured JSON (PO number, status, supplier, site code, trade classification, line items, fiscal year). +4. Claude (via Bedrock `InvokeModel`) extracts structured JSON (PO number, status, supplier, site code, trade classification, line items, fiscal year). Numeric amounts are decoded with `parse_float=Decimal` (DynamoDB rejects floats). 5. Conditional write to DynamoDB: - `new_po` — idempotent insert (no-op if PO exists) - `revision` — unconditional overwrite @@ -39,16 +39,18 @@ Coupa PO emails are received at `amazon_po@int.seahaven.com`, parsed by Claude H ### Work Orders (`WorkorderIngestStack` stack) -Amazon APM work order emails (from Hexagon EAM / HxGN SmartCloud) are received at `apm@int.seahaven.com`, parsed by Claude Haiku 4.5, and written to the `WorkOrders` DynamoDB table. +Amazon APM work order emails (from Hexagon EAM / HxGN SmartCloud) are received at `apm@int.seahaven.com`, parsed **deterministic-template-first with a Claude-on-Bedrock fallback**, and written to the `WorkOrders` DynamoDB table. **Flow:** 1. Hexagon EAM sends email notifications (new assignments, comments, updates, cancellations) to `amazon@seahavenind.com`. 2. Gmail filter forwards APM emails to `apm@int.seahaven.com` (SES). 3. SES drops the raw MIME into `s3://workorder-ingest-emails-{AccountId}/inbound/`. 4. S3 triggers the `workorder-email-processor` Lambda. -5. Claude extracts structured JSON (work order ID, site code, severity, priority, dates, assigned technician). +5. **Parse:** a pure, offline template parser (`template_parser.py`) tries the two known Hexagon templates first, behind a strict fail-closed validation gate. Only on a miss/invalid result does the Lambda fall back to the Claude-on-Bedrock AI extractor. Both paths emit the identical structured-JSON contract (work order ID, site code, severity, priority, dates, assigned technician). 6. Work order upserted to `WorkOrders`, event/comment appended to `WorkOrderComments`. +**Deterministic template parser.** ~93.6% of WO traffic is the plain-text "AMAZON UPDATE WO DETAILS \" comment template (T1) and ~6.4% is the HTML "AMAZON assign Work Order \ on building \" assignment template (T2). `template_parser.try_deterministic_parse()` classifies by subject, extracts the contract fields, and returns a parsed result **only if** it passes a strict validation gate (exact contract-key set; `work_order_id` matches the subject and is all-digits; the T1 `Work Order: ` double-space is literally present; `email_type` matches the template; `site_code` shape; per-type required fields; and a label-bleed guard so a value that over-ran into another field fails). Anything that fails — the rare update/cancellation shapes, Hexagon template drift, or an extractor exception — falls back to the AI extractor. **Data is never corrupted; only the fallback rate rises.** Every record emits one CloudWatch EMF metric (see below). + **Lambdas** (`lambdas/wo/`): | Function | Trigger | Purpose | |---|---|---| @@ -57,7 +59,7 @@ Amazon APM work order emails (from Hexagon EAM / HxGN SmartCloud) are received a **Tables:** - `WorkOrders` (PK: `work_order_id`) — `site-code-index` and `status-index` GSIs removed 2026-06-03 (audit M-20) -- `WorkOrderComments` (PK: `work_order_id`, SK: `comment_id`) +- `WorkOrderComments` (PK: `work_order_id`, SK: `comment_id`) — see the `comment_id` format note below ## Architecture @@ -65,9 +67,9 @@ Amazon APM work order emails (from Hexagon EAM / HxGN SmartCloud) are received a All Lambdas: Python 3.12, ARM64, 60-day log retention. -**Secrets:** -- `po-ingest/anthropic-api-key` — Anthropic API key for PO parsing -- `workorder-ingest/anthropic-api-key` — Anthropic API key for WO parsing +**LLM provider — Amazon Bedrock.** Both email processors call Claude Haiku 4.5 through the Bedrock inference profile `us.anthropic.claude-haiku-4-5-20251001-v1:0` (`bedrock-runtime` `InvokeModel`), configured via the `BEDROCK_MODEL_ID` env var. There is **no Anthropic API key and no Secrets Manager secret** any more. Each processor role is granted `bedrock:InvokeModel` + `bedrock:InvokeModelWithResponseStream` on **both** the inference-profile ARN **and** the per-region foundation-model ARNs for `us-east-1` / `us-east-2` / `us-west-2` (empty-account foundation-model ARNs) — the `us.*` profile can route cross-region, so a profile-only grant would `AccessDenied` at runtime under load. + +> **Retired secrets (manual cleanup outstanding):** the former secrets `po-ingest/anthropic-api-key` and `workorder-ingest/anthropic-api-key` had `RemovalPolicy.RETAIN`, so removing them from the CDK stacks **orphans** them rather than deleting them. Delete both by hand post-deploy and revoke the stored keys at the provider. **SES:** Both stacks add rules to the shared `INBOUND_MAIL` receipt rule set on `int.seahaven.com`. @@ -89,6 +91,8 @@ The `-duration` and `-throttles` alarms for `po-email-processor` and `wo **DLQ alarms** (`AWS/SQS`): `po-email-processor-dlq-messages` and `workorder-email-processor-dlq-messages` fire when any message is visible on an email-processor DLQ (`ApproximateNumberOfMessagesVisible` Maximum, 5 min, `> 0`, eval 1) — a message there means an email was dropped after Lambda exhausted its async retries. +**Parse-outcome metric + fallback-rate alarm (workorder-ingest):** the WO processor writes one CloudWatch **EMF** line per email to namespace `Seahaven/WorkorderIngest`, metric `ParseOutcome` (Unit Count, value 1), dimensioned by `ParseMethod` (`template` | `ai_fallback`) and `TemplateId` (`update_plaintext` | `assign_html` | `unknown`). Non-dimension EMF properties `ReasonCode` and `work_order_id` are queryable in Logs Insights but not promoted to metrics (kept low-cardinality). EMF is used instead of `PutMetricData` so there is no extra sync call / latency / IAM grant on the async hot path (the role already has `logs:PutLogEvents`). The alarm `workorder-email-processor-template-fallback-rate` fires when the AI-fallback share of parses exceeds **15%** sustained (a `MathExpression` with `FILL(...,0)` and a ≥10-sample volume floor over 15-minute periods, eval 3 / datapoints 2) — catching Hexagon template-drift coverage collapse while the volume floor + `FILL` prevent low-volume false pages / `INSUFFICIENT_DATA`. ALARM-only `SnsAction` to `site-alerts`, no OK action, `NOT_BREACHING`. The 15-minute period is a deliberate deviation from the 5-minute house style to accumulate a stable denominator at the low ~760/day volume. + **DynamoDB alarms** (`AWS/DynamoDB`): each owned table gets `-throttles` (`ThrottledRequests`) and `
-system-errors` (`SystemErrors`). These metrics emit only at the `TableName` + `Operation` dimension set, so each alarm is a `Sum` math expression across the operations the table uses (Get/BatchGet/Query/Scan/Put/Update/Delete/BatchWrite). Tables covered: `purchase-orders`, `verified-sites`, `pending-site-review` (po-ingest); `WorkOrders`, `WorkOrderComments` (workorder-ingest). ## Shared Resources @@ -116,6 +120,23 @@ Both tables are **owned by this repo's `WorkorderIngestStack`** (`cdk/wo_stack.p - `WorkOrders` — PK `work_order_id` (S). - `WorkOrderComments` — PK `work_order_id` (S), SK `comment_id` (S). +> **`comment_id` format change (issue #23).** The `WorkOrderComments` range key is now +> `work_order_id##` +> (e.g. `11144580730#2026-04-27T23:51:48#a1b2c3d4e5f6`, or `…#nocomment#…` when the source +> email carries no comment time). Previously it was `work_order_id#`, where two +> emails on the same WO with an identical/absent comment time collided and overwrote each other. +> The 12-hex suffix is derived from the **S3 object key alone** — deterministic, so a Lambda +> async **retry** of the same object produces a byte-identical key (idempotent, no duplicate row), +> while two distinct emails on the same WO get distinct keys. Wall-clock `now()` is kept **out** of +> the key. Consumers that split on `#` and read index `[0]`/`[1]` are unaffected; anything that +> treated "everything after the first `#`" as a bare timestamp now also captures the hash segment. + +> **Timestamp format shift.** All stored ISO timestamps (`created_at`, `updated_at`, `ingested_at` +> on WO; `processed_at`, `cancelled_at` on PO) moved from naive `datetime.utcnow().isoformat()` +> to timezone-aware `datetime.now(timezone.utc).isoformat()`, so they now carry a `+00:00` suffix +> (e.g. `2026-07-15T12:00:00+00:00`). Downstream parsers that assumed a naive/no-offset string +> must accept the offset. + Both currently use default DynamoDB encryption — they are **not** yet on the shared customer-managed CMK (`alias/seahaven-dynamodb`, INFRA-95 / M-3); that migration is tracked in INFRA-6. The `workorder-email-processor` role already holds a pre-emptive encrypt/decrypt grant on the CMK so the migration won't break it. **Consumer (read-only) — data contract:** `seahaven-slack-bot` imports both tables via `Table.fromTableName(...)` (`grantReadData`) and reads them from two Lambdas: `workorder-sync` (daily full-table scan into the Bedrock knowledge base) and `wo-po-lookup` (the Bedrock agent's direct WO lookup action group). The bot depends on the PK/SK schema above and these attributes: on `WorkOrders` — `description`, `wo_status`, `customer`, `site_code`, `building`, `address`, `severity`, `priority`, `assigned_to`, `date_reported`, `scheduled_start`, `due_date`, `updated_at`; on `WorkOrderComments` — `created_at` (used to sort comments), `commenter`, `text`. Any change to table name, key schema, these attribute names, or the encryption configuration (e.g. the INFRA-6 CMK migration) must be coordinated with `seahaven-slack-bot` before it ships, or the Bedrock agent breaks at runtime (not at deploy — the tables are imported by name, so there is no compile-time link). @@ -146,18 +167,34 @@ Branch protection on `main` — all changes through PR. ## Setup 1. Bootstrap CDK: `cdk bootstrap aws://{AccountId}/us-east-1` -2. Deploy both stacks (this also creates the two Secrets Manager secrets — they are CDK-managed, so don't `create-secret` them by hand): +2. Ensure the Bedrock inference profile `us.anthropic.claude-haiku-4-5-20251001-v1:0` is enabled in `us-east-1` (it is; the CDK grants cover cross-region routing to `us-east-2`/`us-west-2`). No API key or secret to set — the processors authenticate to Bedrock via their IAM roles. +3. Deploy both stacks: ```bash cd cdk pip install -r requirements.txt cdk deploy --all ``` -3. Set the Anthropic API key values: +4. **Post-deploy cleanup (one-time):** the retired `RETAIN`-policy secrets `po-ingest/anthropic-api-key` and `workorder-ingest/anthropic-api-key` are orphaned by this deploy, not deleted. Remove them and revoke the keys at the provider: ```bash - aws secretsmanager put-secret-value --secret-id po-ingest/anthropic-api-key --secret-string "sk-ant-..." - aws secretsmanager put-secret-value --secret-id workorder-ingest/anthropic-api-key --secret-string "sk-ant-..." + aws secretsmanager delete-secret --secret-id po-ingest/anthropic-api-key --force-delete-without-recovery + aws secretsmanager delete-secret --secret-id workorder-ingest/anthropic-api-key --force-delete-without-recovery ``` -4. Dashboards: `po-web-ui` and `workorder-web-ui` have no public endpoint (the Function URLs were removed 2026-06-08, INFRA-74). Invoke them manually and render the returned HTML, e.g. `aws lambda invoke --function-name po-web-ui /tmp/out.json`. +5. Dashboards: `po-web-ui` and `workorder-web-ui` have no public endpoint (the Function URLs were removed 2026-06-08, INFRA-74). Invoke them manually and render the returned HTML, e.g. `aws lambda invoke --function-name po-web-ui /tmp/out.json`. + +## Tests + +Offline unit tests (no AWS, no network) run via pytest from the repo root: + +```bash +pip install pytest +pytest +``` + +Coverage: +- `tests/` — shared handler tests (`parse_raw_email`, `pad_zip`). +- `lambdas/wo/email_processor/tests/` — the deterministic WO parser suite: golden-file tests over 55 real scrubbed `.eml` fixtures (`test_parser.py`), fail-closed validation-gate rules and adversarial/injection cases (`test_validation_gate.py`), the issue #23 `comment_id` idempotency invariants (`test_comment_id.py`), and the Bedrock-fallback dispatch/EMF-metric behavior with a mocked `invoke_model` (`test_bedrock_fallback.py`). Golden JSON lives under `tests/fixtures/expected/`. + +Both roots are discovered by `pytest.ini` (`testpaths`). ## Scripts @@ -181,14 +218,18 @@ cdk/ wo_stack.py # Work order pipeline resources lambdas/ po/ # PO pipeline Lambdas - email_processor/ + email_processor/ # handler.py (Bedrock, fully AI) site_extractor/ web_ui/ wo/ # WO pipeline Lambdas email_processor/ + handler.py # template-first + Bedrock fallback, EMF metric, #23 comment_id + template_parser.py # pure deterministic parser + fail-closed validation gate + tests/ # golden-file + validation-gate + comment_id + fallback tests web_ui/ scripts/ reprocess.py backfill_sites.py -test_local.py # Parse sample emails through Claude locally (no AWS) +tests/ # shared handler tests (discovered with the WO suite) +test_local.py # Parse sample emails through Bedrock locally (no AWS mutation) ```