mirror of
https://github.com/Sea-Haven-Industries/procurement-ingest.git
synced 2026-09-30 11:53:13 +00:00
85 lines
3.8 KiB
Markdown
85 lines
3.8 KiB
Markdown
# PO Ingest
|
|
|
|
Coupa purchase-order email ingestion pipeline. SES receives Amazon PO emails, Claude extracts structured data, and the result lands in the shared `purchase-orders` DynamoDB table.
|
|
|
|
## Flow
|
|
|
|
1. Coupa sends a PO email to `amazon_po@int.seahaven.com`.
|
|
2. SES (using the shared `INBOUND_MAIL` rule set) drops the raw MIME into `s3://po-ingest-emails-{AccountId}/inbound/`.
|
|
3. S3 `ObjectCreated` fires the `po-email-processor` Lambda.
|
|
4. The Lambda parses the email, sends it to Claude Haiku 4.5 for structured JSON extraction, and writes to DynamoDB.
|
|
- `email_type: new_po` — conditional `PutItem` on `purchase-orders` (idempotent on `po_number`).
|
|
- `email_type: cancellation` — `UpdateItem` marking the existing row `Cancelled`.
|
|
5. DynamoDB Streams (NEW_AND_OLD_IMAGES) on `purchase-orders` feeds two downstream consumers:
|
|
- **LedgerFlow** (`seahaven-slack-bot/po-sync`) — daily KB sync.
|
|
- **Verified-sites pipeline** (`po-ingest-site-extractor`) — real-time site address extraction (see below).
|
|
|
|
A separate `po-web-ui` Lambda (Function URL, unauthenticated) renders a simple HTML dashboard scanning the table.
|
|
|
|
### Verified-sites pipeline
|
|
|
|
The `po-ingest-site-extractor` Lambda is triggered by the DynamoDB Stream on every PO INSERT/MODIFY. It:
|
|
|
|
1. Extracts an Amazon facility site code from `ship_to.name` using a regex cascade (parentheses, `LLC - CODE`, `Station CODE`, `DS - CODE`) with a fallback to the first `line_items` description.
|
|
2. Parses `ship_to.address` into structured fields (street, city, state, zip).
|
|
3. Upserts to the `verified-sites` DynamoDB table — atomically increments `poCount` and appends the PO number to `sourcePOs`.
|
|
|
|
POs with no extractable site code are logged and skipped.
|
|
|
|
Backfill stats (initial run): 14,825 POs scanned → 9,900 with extractable site codes → 1,100 unique sites.
|
|
|
|
## Architecture
|
|
|
|
- **IaC:** AWS CDK (Python), stack name `PoIngestStack`, region `us-east-1`.
|
|
- **Lambdas** (all Python 3.12, arm64, 60-day log retention):
|
|
- `po-email-processor` — S3-triggered, parses PO emails via Claude Haiku.
|
|
- `po-web-ui` — Function URL, HTML dashboard.
|
|
- `po-ingest-site-extractor` — DynamoDB Streams-triggered, extracts site addresses.
|
|
- **Storage:**
|
|
- S3 `po-ingest-emails-{AccountId}` — 90-day lifecycle expiry.
|
|
- DynamoDB `purchase-orders` — owned by this stack, Streams enabled (NEW_AND_OLD_IMAGES).
|
|
- DynamoDB `verified-sites` — PK `siteCode`, GSI `by-state` on `state`.
|
|
- **Secrets:** Anthropic API key in Secrets Manager at `po-ingest/anthropic-api-key`.
|
|
- **SES:** adds the `PoEmailRule` to the existing `INBOUND_MAIL` receipt rule set (shared with `workorder-ingest`).
|
|
|
|
## Setup
|
|
|
|
1. Bootstrap CDK in the account if you haven't already: `cdk bootstrap aws://{AccountId}/us-east-1`.
|
|
2. Store the Anthropic API key:
|
|
```bash
|
|
aws secretsmanager create-secret \
|
|
--name po-ingest/anthropic-api-key \
|
|
--secret-string "sk-ant-..."
|
|
```
|
|
3. Install Lambda dependencies into the deployable package directory (gitignored):
|
|
```bash
|
|
pip install -r lambdas/email_processor/requirements.txt -t lambdas/email_processor/package/
|
|
```
|
|
4. Deploy:
|
|
```bash
|
|
cd cdk
|
|
pip install -r requirements.txt
|
|
cdk deploy
|
|
```
|
|
5. The `WebUIUrl` CloudFormation output is the dashboard URL.
|
|
|
|
## Reprocessing
|
|
|
|
To re-run the processor against every email still sitting in `inbound/` (useful after a parser change):
|
|
|
|
```bash
|
|
python scripts/reprocess.py # dry-run — lists keys
|
|
python scripts/reprocess.py --execute # invokes po-email-processor for each
|
|
```
|
|
|
|
Inserts are conditional on `po_number`, so re-processing existing POs is a no-op.
|
|
|
|
## Backfilling verified sites
|
|
|
|
The stream Lambda handles all future POs automatically. To backfill from historical PO data (one-time):
|
|
|
|
```bash
|
|
python scripts/backfill_sites.py
|
|
```
|
|
|
|
Uses the same extraction logic as the Lambda. Idempotent — safe to re-run.
|