diff --git a/README.md b/README.md index 3b7fb55..84a04e3 100644 --- a/README.md +++ b/README.md @@ -34,7 +34,7 @@ Coupa PO emails are received at `amazon_po@int.seahaven.com`, parsed by Claude H **Tables:** - `purchase-orders` (PK: `po_number`, Streams: NEW_IMAGE) — shared with seahaven-slack-bot (read-only; see Shared Resources) -- `verified-sites` (PK: `siteCode`, GSI: `by-state`) — ~1,100 unique Amazon facility sites +- `verified-sites` (PK: `siteCode`) — ~1,100 unique Amazon facility sites (`by-state` GSI removed 2026-06-03, audit M-20) - `pending-site-review` (PK: `po_number`) — unresolvable POs for manual Payee Central verification ### Work Orders (`WorkorderIngestStack` stack) @@ -56,7 +56,7 @@ Amazon APM work order emails (from Hexagon EAM / HxGN SmartCloud) are received a | `workorder-web-ui` | Manual invoke | HTML dashboard (public Function URL removed 2026-06-08, INFRA-74) | **Tables:** -- `WorkOrders` (PK: `work_order_id`, GSIs: `site-code-index`, `status-index`) +- `WorkOrders` (PK: `work_order_id`) — `site-code-index` and `status-index` GSIs removed 2026-06-03 (audit M-20) - `WorkOrderComments` (PK: `work_order_id`, SK: `comment_id`) ## Architecture @@ -87,13 +87,15 @@ Every alarm is **ALARM-only** (no OK action), sends to the shared `site-alerts` The `-duration` and `-throttles` alarms for `po-email-processor` and `workorder-email-processor` supersede the orphaned, CLI-created `Lambda-Duration-*` / `Lambda-Throttles-*` alarms (deleted post-deploy). +**DLQ alarms** (`AWS/SQS`): `po-email-processor-dlq-messages` and `workorder-email-processor-dlq-messages` fire when any message is visible on an email-processor DLQ (`ApproximateNumberOfMessagesVisible` Maximum, 5 min, `> 0`, eval 1) — a message there means an email was dropped after Lambda exhausted its async retries. + **DynamoDB alarms** (`AWS/DynamoDB`): each owned table gets `-throttles` (`ThrottledRequests`) and `
-system-errors` (`SystemErrors`). These metrics emit only at the `TableName` + `Operation` dimension set, so each alarm is a `Sum` math expression across the operations the table uses (Get/BatchGet/Query/Scan/Put/Update/Delete/BatchWrite). Tables covered: `purchase-orders`, `verified-sites`, `pending-site-review` (po-ingest); `WorkOrders`, `WorkOrderComments` (workorder-ingest). ## Shared Resources ### `purchase-orders` table (owned here) -The `purchase-orders` DynamoDB table is **owned by this repo's `po-ingest` stack** (defined in `cdk/po_stack.py` with `RemovalPolicy.RETAIN` and `StreamViewType.NEW_IMAGE`). The `po-email-processor` Lambda is the authoritative writer — it performs the conditional inserts, revision overwrites, and cancellation updates described above. +The `purchase-orders` DynamoDB table is **owned by this repo's `po-ingest` stack** (defined in `cdk/po_stack.py` with `RemovalPolicy.RETAIN`, `StreamViewType.NEW_IMAGE`, and SSE-KMS encryption with the shared customer-managed CMK `alias/seahaven-dynamodb`, INFRA-95 / M-3). The `po-email-processor` Lambda is the authoritative writer — it performs the conditional inserts, revision overwrites, and cancellation updates described above. **Consumers (read-only):** @@ -109,16 +111,18 @@ The consumer imports the table via `Table.fromTableName(...)` and is granted rea ### `WorkOrders` and `WorkOrderComments` tables (owned here) -Both tables are **owned by this repo's `WorkorderIngestStack`** (`cdk/wo_stack.py`, `RemovalPolicy.RETAIN`, shared customer-managed CMK per INFRA-95 / M-3): +Both tables are **owned by this repo's `WorkorderIngestStack`** (`cdk/wo_stack.py`, `RemovalPolicy.RETAIN`): - `WorkOrders` — PK `work_order_id` (S). - `WorkOrderComments` — PK `work_order_id` (S), SK `comment_id` (S). -**Consumer (read-only) — data contract:** `seahaven-slack-bot` imports both tables via `Table.fromTableName(...)` (`grantReadData` plus an explicit `kms:Decrypt` grant on the shared CMK) and reads them from two Lambdas: `workorder-sync` (daily full-table scan into the Bedrock knowledge base) and `wo-po-lookup` (the Bedrock agent's direct WO lookup action group). The bot depends on the PK/SK schema above, the CMK encryption, and these attributes: on `WorkOrders` — `description`, `wo_status`, `customer`, `site_code`, `building`, `address`, `severity`, `priority`, `assigned_to`, `date_reported`, `scheduled_start`, `due_date`, `updated_at`; on `WorkOrderComments` — `created_at` (used to sort comments), `commenter`, `text`. Any change to table name, key schema, these attribute names, or the encryption key must be coordinated with `seahaven-slack-bot` before it ships, or the Bedrock agent breaks at runtime (not at deploy — the tables are imported by name, so there is no compile-time link). +Both currently use default DynamoDB encryption — they are **not** yet on the shared customer-managed CMK (`alias/seahaven-dynamodb`, INFRA-95 / M-3); that migration is tracked in INFRA-6. The `workorder-email-processor` role already holds a pre-emptive encrypt/decrypt grant on the CMK so the migration won't break it. + +**Consumer (read-only) — data contract:** `seahaven-slack-bot` imports both tables via `Table.fromTableName(...)` (`grantReadData`) and reads them from two Lambdas: `workorder-sync` (daily full-table scan into the Bedrock knowledge base) and `wo-po-lookup` (the Bedrock agent's direct WO lookup action group). The bot depends on the PK/SK schema above and these attributes: on `WorkOrders` — `description`, `wo_status`, `customer`, `site_code`, `building`, `address`, `severity`, `priority`, `assigned_to`, `date_reported`, `scheduled_start`, `due_date`, `updated_at`; on `WorkOrderComments` — `created_at` (used to sort comments), `commenter`, `text`. Any change to table name, key schema, these attribute names, or the encryption configuration (e.g. the INFRA-6 CMK migration) must be coordinated with `seahaven-slack-bot` before it ships, or the Bedrock agent breaks at runtime (not at deploy — the tables are imported by name, so there is no compile-time link). ### `verified-sites` table (owned here) -Owned by this repo's `po-ingest` stack (`cdk/po_stack.py`). PK `siteCode` (S); AWS-managed encryption (NOT the shared CMK). +Owned by this repo's `po-ingest` stack (`cdk/po_stack.py`). PK `siteCode` (S); default DynamoDB encryption (NOT the shared CMK). **Consumer (read-only) — data contract:** `seahaven-slack-bot`'s `wo-po-lookup` Lambda imports this table via `Table.fromTableName(...)` for the Bedrock agent's `lookup_site` action. It does point lookups by `siteCode` and reads `address`, `fullAddress`, `city`, `state`, `zip`, `latitude`, `longitude`, `notes`. Coordinate any change to the table name, key schema, or these attribute names with `seahaven-slack-bot`. @@ -126,33 +130,34 @@ Owned by this repo's `po-ingest` stack (`cdk/po_stack.py`). PK `siteCode` (S); A ## Documentation -The canonical map of Sea Haven's AWS infrastructure lives in Confluence. This project's `po-ingest` and `workorder-ingest` stacks are represented there as Mermaid subgraphs. +The canonical map of Sea Haven's AWS infrastructure lives in Confluence. This project's `po-ingest` and `WorkorderIngestStack` stacks are represented there as Mermaid subgraphs. - **[AWS Architecture Map](https://seahaven.atlassian.net/wiki/spaces/IT/pages/1540098)** (Confluence, IT space, page 1540098) ## CI/CD -GitHub Actions with reusable workflows from `Sea-Haven-Industries/.github`: -- **CI** (PR to `main`): linting + `cdk synth` via `ci-python-sam.yaml@main` -- **CD** (push to `main`): `cdk deploy --all` via `cd-cdk.yaml@main` (OIDC auth) +GitHub Actions with reusable workflows from `Sea-Haven-Industries/.github` (all pinned to a commit SHA of `main`): +- **CI** (`ci.yaml`, PR to `main`): linting + `cdk synth` via `ci-python-sam.yaml` +- **CD** (`deploy.yaml`, push to `main`): CDK deploy via `cd-cdk.yaml` (OIDC auth) +- Plus dependency review and PR labeler workflows on every PR Branch protection on `main` — all changes through PR. ## Setup 1. Bootstrap CDK: `cdk bootstrap aws://{AccountId}/us-east-1` -2. Store Anthropic API keys: - ```bash - aws secretsmanager create-secret --name po-ingest/anthropic-api-key --secret-string "sk-ant-..." - aws secretsmanager create-secret --name workorder-ingest/anthropic-api-key --secret-string "sk-ant-..." - ``` -3. Deploy both stacks: +2. Deploy both stacks (this also creates the two Secrets Manager secrets — they are CDK-managed, so don't `create-secret` them by hand): ```bash cd cdk pip install -r requirements.txt cdk deploy --all ``` -4. CloudFormation outputs include `WebUIUrl` for each stack's dashboard. +3. Set the Anthropic API key values: + ```bash + aws secretsmanager put-secret-value --secret-id po-ingest/anthropic-api-key --secret-string "sk-ant-..." + aws secretsmanager put-secret-value --secret-id workorder-ingest/anthropic-api-key --secret-string "sk-ant-..." + ``` +4. Dashboards: `po-web-ui` and `workorder-web-ui` have no public endpoint (the Function URLs were removed 2026-06-08, INFRA-74). Invoke them manually and render the returned HTML, e.g. `aws lambda invoke --function-name po-web-ui /tmp/out.json`. ## Scripts @@ -171,7 +176,7 @@ python scripts/backfill_sites.py ``` cdk/ - app.py # Two stacks: po-ingest + WorkorderIngestStack + app.py # Two stacks: po-ingest + WorkorderIngestStack po_stack.py # Purchase order pipeline resources wo_stack.py # Work order pipeline resources lambdas/ @@ -182,9 +187,8 @@ lambdas/ wo/ # WO pipeline Lambdas email_processor/ web_ui/ -shared/ - models.py # Work order dataclasses/enums scripts/ reprocess.py backfill_sites.py +test_local.py # Parse sample emails through Claude locally (no AWS) ```