adam/procurement-ingest
1
0
Fork 0
mirror of https://github.com/Sea-Haven-Industries/procurement-ingest.git synced 2026-09-30 10:43:14 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Correct stale architecture facts in README (#96)

Browse source 
The README had drifted from the deployed stacks: it still listed
the removed shared/ directory, the by-state / site-code-index /
status-index GSIs deleted in the 2026-06-03 audit (M-20), the
WebUIUrl CloudFormation output retired with the public Function
URLs (INFRA-74), and it attributed the shared DynamoDB CMK to the
work-order tables when only purchase-orders is migrated (the WO
migration is still tracked in INFRA-6). Setup steps also told the
reader to create the CDK-managed secrets by hand, which would
collide on deploy. Align all of it with cdk/po_stack.py,
cdk/wo_stack.py, and the actual repo layout, and document the DLQ
messages alarms that were missing from the alarms section.

Refs: INFRA-74
This commit is contained in:
Adam Moussa 2026-07-15 19:00:43 -04:00 • committed by GitHub
parent 00517cdb9d
commit 7b90f54815
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 24 additions and 20 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

44
README.md
View file

@ -34,7 +34,7 @@ Coupa PO emails are received at `amazon_po@int.seahaven.com`, parsed by Claude H
**Tables:**
- `purchase-orders` (PK: `po_number`, Streams: NEW_IMAGE) — shared with seahaven-slack-bot (read-only; see Shared Resources)
- `verified-sites` (PK: `siteCode`, GSI: `by-state`) — ~1,100 unique Amazon facility sites
- `verified-sites` (PK: `siteCode`) — ~1,100 unique Amazon facility sites (`by-state` GSI removed 2026-06-03, audit M-20)
- `pending-site-review` (PK: `po_number`) — unresolvable POs for manual Payee Central verification
### Work Orders (`WorkorderIngestStack` stack)
@ -56,7 +56,7 @@ Amazon APM work order emails (from Hexagon EAM / HxGN SmartCloud) are received a
| `workorder-web-ui` | Manual invoke | HTML dashboard (public Function URL removed 2026-06-08, INFRA-74) |
**Tables:**
- `WorkOrders` (PK: `work_order_id`, GSIs: `site-code-index`, `status-index`)
- `WorkOrders` (PK: `work_order_id`) — `site-code-index` and `status-index` GSIs removed 2026-06-03 (audit M-20)
- `WorkOrderComments` (PK: `work_order_id`, SK: `comment_id`)
## Architecture
@ -87,13 +87,15 @@ Every alarm is **ALARM-only** (no OK action), sends to the shared `site-alerts`
The `<fn>-duration` and `<fn>-throttles` alarms for `po-email-processor` and `workorder-email-processor` supersede the orphaned, CLI-created `Lambda-Duration-*` / `Lambda-Throttles-*` alarms (deleted post-deploy).
**DLQ alarms** (`AWS/SQS`): `po-email-processor-dlq-messages` and `workorder-email-processor-dlq-messages` fire when any message is visible on an email-processor DLQ (`ApproximateNumberOfMessagesVisible` Maximum, 5 min, `> 0`, eval 1) — a message there means an email was dropped after Lambda exhausted its async retries.
**DynamoDB alarms** (`AWS/DynamoDB`): each owned table gets `<table>-throttles` (`ThrottledRequests`) and `<table>-system-errors` (`SystemErrors`). These metrics emit only at the `TableName` + `Operation` dimension set, so each alarm is a `Sum` math expression across the operations the table uses (Get/BatchGet/Query/Scan/Put/Update/Delete/BatchWrite). Tables covered: `purchase-orders`, `verified-sites`, `pending-site-review` (po-ingest); `WorkOrders`, `WorkOrderComments` (workorder-ingest).
## Shared Resources
### `purchase-orders` table (owned here)
The `purchase-orders` DynamoDB table is **owned by this repo's `po-ingest` stack** (defined in `cdk/po_stack.py` with `RemovalPolicy.RETAIN` and `StreamViewType.NEW_IMAGE`). The `po-email-processor` Lambda is the authoritative writer — it performs the conditional inserts, revision overwrites, and cancellation updates described above.
The `purchase-orders` DynamoDB table is **owned by this repo's `po-ingest` stack** (defined in `cdk/po_stack.py` with `RemovalPolicy.RETAIN`, `StreamViewType.NEW_IMAGE`, and SSE-KMS encryption with the shared customer-managed CMK `alias/seahaven-dynamodb`, INFRA-95 / M-3). The `po-email-processor` Lambda is the authoritative writer — it performs the conditional inserts, revision overwrites, and cancellation updates described above.
**Consumers (read-only):**
@ -109,16 +111,18 @@ The consumer imports the table via `Table.fromTableName(...)` and is granted rea
### `WorkOrders` and `WorkOrderComments` tables (owned here)
Both tables are **owned by this repo's `WorkorderIngestStack`** (`cdk/wo_stack.py`, `RemovalPolicy.RETAIN`, shared customer-managed CMK per INFRA-95 / M-3):
Both tables are **owned by this repo's `WorkorderIngestStack`** (`cdk/wo_stack.py`, `RemovalPolicy.RETAIN`):
- `WorkOrders` — PK `work_order_id` (S).
- `WorkOrderComments` — PK `work_order_id` (S), SK `comment_id` (S).
**Consumer (read-only) — data contract:** `seahaven-slack-bot` imports both tables via `Table.fromTableName(...)` (`grantReadData` plus an explicit `kms:Decrypt` grant on the shared CMK) and reads them from two Lambdas: `workorder-sync` (daily full-table scan into the Bedrock knowledge base) and `wo-po-lookup` (the Bedrock agent's direct WO lookup action group). The bot depends on the PK/SK schema above, the CMK encryption, and these attributes: on `WorkOrders` — `description`, `wo_status`, `customer`, `site_code`, `building`, `address`, `severity`, `priority`, `assigned_to`, `date_reported`, `scheduled_start`, `due_date`, `updated_at`; on `WorkOrderComments` — `created_at` (used to sort comments), `commenter`, `text`. Any change to table name, key schema, these attribute names, or the encryption key must be coordinated with `seahaven-slack-bot` before it ships, or the Bedrock agent breaks at runtime (not at deploy — the tables are imported by name, so there is no compile-time link).
Both currently use default DynamoDB encryption — they are **not** yet on the shared customer-managed CMK (`alias/seahaven-dynamodb`, INFRA-95 / M-3); that migration is tracked in INFRA-6. The `workorder-email-processor` role already holds a pre-emptive encrypt/decrypt grant on the CMK so the migration won't break it.
**Consumer (read-only) — data contract:** `seahaven-slack-bot` imports both tables via `Table.fromTableName(...)` (`grantReadData`) and reads them from two Lambdas: `workorder-sync` (daily full-table scan into the Bedrock knowledge base) and `wo-po-lookup` (the Bedrock agent's direct WO lookup action group). The bot depends on the PK/SK schema above and these attributes: on `WorkOrders` — `description`, `wo_status`, `customer`, `site_code`, `building`, `address`, `severity`, `priority`, `assigned_to`, `date_reported`, `scheduled_start`, `due_date`, `updated_at`; on `WorkOrderComments` — `created_at` (used to sort comments), `commenter`, `text`. Any change to table name, key schema, these attribute names, or the encryption configuration (e.g. the INFRA-6 CMK migration) must be coordinated with `seahaven-slack-bot` before it ships, or the Bedrock agent breaks at runtime (not at deploy — the tables are imported by name, so there is no compile-time link).
### `verified-sites` table (owned here)
Owned by this repo's `po-ingest` stack (`cdk/po_stack.py`). PK `siteCode` (S); AWS-managed encryption (NOT the shared CMK).
Owned by this repo's `po-ingest` stack (`cdk/po_stack.py`). PK `siteCode` (S); default DynamoDB encryption (NOT the shared CMK).
**Consumer (read-only) — data contract:** `seahaven-slack-bot`'s `wo-po-lookup` Lambda imports this table via `Table.fromTableName(...)` for the Bedrock agent's `lookup_site` action. It does point lookups by `siteCode` and reads `address`, `fullAddress`, `city`, `state`, `zip`, `latitude`, `longitude`, `notes`. Coordinate any change to the table name, key schema, or these attribute names with `seahaven-slack-bot`.
@ -126,33 +130,34 @@ Owned by this repo's `po-ingest` stack (`cdk/po_stack.py`). PK `siteCode` (S); A
## Documentation
The canonical map of Sea Haven's AWS infrastructure lives in Confluence. This project's `po-ingest` and `workorder-ingest` stacks are represented there as Mermaid subgraphs.
The canonical map of Sea Haven's AWS infrastructure lives in Confluence. This project's `po-ingest` and `WorkorderIngestStack` stacks are represented there as Mermaid subgraphs.
- **[AWS Architecture Map](https://seahaven.atlassian.net/wiki/spaces/IT/pages/1540098)** (Confluence, IT space, page 1540098)
## CI/CD
GitHub Actions with reusable workflows from `Sea-Haven-Industries/.github`:
- **CI** (PR to `main`): linting + `cdk synth` via `ci-python-sam.yaml@main`
- **CD** (push to `main`): `cdk deploy --all` via `cd-cdk.yaml@main` (OIDC auth)
GitHub Actions with reusable workflows from `Sea-Haven-Industries/.github` (all pinned to a commit SHA of `main`):
- **CI** (`ci.yaml`, PR to `main`): linting + `cdk synth` via `ci-python-sam.yaml`
- **CD** (`deploy.yaml`, push to `main`): CDK deploy via `cd-cdk.yaml` (OIDC auth)
- Plus dependency review and PR labeler workflows on every PR
Branch protection on `main` — all changes through PR.
## Setup
1. Bootstrap CDK: `cdk bootstrap aws://{AccountId}/us-east-1`
2. Store Anthropic API keys:
```bash
aws secretsmanager create-secret --name po-ingest/anthropic-api-key --secret-string "sk-ant-..."
aws secretsmanager create-secret --name workorder-ingest/anthropic-api-key --secret-string "sk-ant-..."
```
3. Deploy both stacks:
2. Deploy both stacks (this also creates the two Secrets Manager secrets — they are CDK-managed, so don't `create-secret` them by hand):
```bash
cd cdk
pip install -r requirements.txt
cdk deploy --all
```
4. CloudFormation outputs include `WebUIUrl` for each stack's dashboard.
3. Set the Anthropic API key values:
```bash
aws secretsmanager put-secret-value --secret-id po-ingest/anthropic-api-key --secret-string "sk-ant-..."
aws secretsmanager put-secret-value --secret-id workorder-ingest/anthropic-api-key --secret-string "sk-ant-..."
```
4. Dashboards: `po-web-ui` and `workorder-web-ui` have no public endpoint (the Function URLs were removed 2026-06-08, INFRA-74). Invoke them manually and render the returned HTML, e.g. `aws lambda invoke --function-name po-web-ui /tmp/out.json`.
## Scripts
@ -171,7 +176,7 @@ python scripts/backfill_sites.py
```
cdk/
app.py # Two stacks: po-ingest + WorkorderIngestStack
app.py # Two stacks: po-ingest + WorkorderIngestStack
po_stack.py # Purchase order pipeline resources
wo_stack.py # Work order pipeline resources
lambdas/
@ -182,9 +187,8 @@ lambdas/
wo/ # WO pipeline Lambdas
email_processor/
web_ui/
shared/
models.py # Work order dataclasses/enums
scripts/
reprocess.py
backfill_sites.py
test_local.py # Parse sample emails through Claude locally (no AWS)
```