mirror of
https://github.com/Sea-Haven-Industries/procurement-ingest.git
synced 2026-09-30 14:13:13 +00:00
36f49ae259
* Add CI/CD pipeline and fix stack name to kebab-case CodePipeline V2 (po-ingest-pipeline) triggers CodeBuild on push to main, running cdk deploy via buildspec.yml. Stack name changed from PoIngestStack to po-ingest to match naming conventions. * Add RETAIN policy to Secrets Manager secret Prevents the Anthropic API key from being deleted if the stack is ever removed. Matches the RETAIN policy on all other stateful resources (DynamoDB tables, S3 bucket).
93 lines
4.6 KiB
Markdown
93 lines
4.6 KiB
Markdown
# PO Ingest
|
|
|
|
Coupa purchase-order email ingestion pipeline. SES receives Amazon PO emails, Claude extracts structured data, and the result lands in the shared `purchase-orders` DynamoDB table.
|
|
|
|
## Flow
|
|
|
|
1. Coupa sends a PO email to `amazon_po@int.seahaven.com`.
|
|
2. SES (using the shared `INBOUND_MAIL` rule set) drops the raw MIME into `s3://po-ingest-emails-{AccountId}/inbound/`.
|
|
3. S3 `ObjectCreated` fires the `po-email-processor` Lambda.
|
|
4. The Lambda parses the email, sends it to Claude Haiku 4.5 for structured JSON extraction, and writes to DynamoDB.
|
|
- `email_type: new_po` — conditional `PutItem` on `purchase-orders` (idempotent on `po_number`).
|
|
- `email_type: cancellation` — `UpdateItem` marking the existing row `Cancelled`.
|
|
5. DynamoDB Streams (NEW_AND_OLD_IMAGES) on `purchase-orders` feeds two downstream consumers:
|
|
- **LedgerFlow** (`seahaven-slack-bot/po-sync`) — daily KB sync.
|
|
- **Verified-sites pipeline** (`po-ingest-site-extractor`) — real-time site address extraction (see below).
|
|
|
|
A separate `po-web-ui` Lambda (Function URL, unauthenticated) renders a simple HTML dashboard scanning the table.
|
|
|
|
### Verified-sites pipeline
|
|
|
|
The `po-ingest-site-extractor` Lambda is triggered by the DynamoDB Stream on every PO INSERT/MODIFY. It:
|
|
|
|
1. Extracts an Amazon facility site code from `ship_to.name` using a regex cascade (parentheses, `LLC - CODE`, `Station CODE`, `DS - CODE`) with a fallback to the first `line_items` description.
|
|
2. Parses `ship_to.address` into structured fields (street, city, state, zip).
|
|
3. Upserts to the `verified-sites` DynamoDB table — atomically increments `poCount` and appends the PO number to `sourcePOs`.
|
|
|
|
POs with no extractable site code fall through to an address reverse-lookup against the verified-sites cache (normalized street + zip). If still unresolved, the PO is written to the `pending-site-review` table for manual verification against Payee Central.
|
|
|
|
Backfill stats (initial run): 14,825 POs scanned → 9,900 with extractable site codes → 1,100 unique sites.
|
|
|
|
## Architecture
|
|
|
|
- **IaC:** AWS CDK (Python), stack name `po-ingest`, region `us-east-1`.
|
|
- **Lambdas** (all Python 3.12, arm64, 60-day log retention):
|
|
- `po-email-processor` — S3-triggered, parses PO emails via Claude Haiku.
|
|
- `po-web-ui` — Function URL, HTML dashboard.
|
|
- `po-ingest-site-extractor` — DynamoDB Streams-triggered, extracts site addresses.
|
|
- **Storage:**
|
|
- S3 `po-ingest-emails-{AccountId}` — 90-day lifecycle expiry.
|
|
- DynamoDB `purchase-orders` — owned by this stack, Streams enabled (NEW_AND_OLD_IMAGES).
|
|
- DynamoDB `verified-sites` — PK `siteCode`, GSI `by-state` on `state`.
|
|
- DynamoDB `pending-site-review` — PK `po_number`. POs with no extractable site code and no address match, awaiting manual Payee Central verification.
|
|
- **Secrets:** Anthropic API key in Secrets Manager at `po-ingest/anthropic-api-key`.
|
|
- **SES:** adds the `PoEmailRule` to the existing `INBOUND_MAIL` receipt rule set (shared with `workorder-ingest`).
|
|
- **CI/CD:** CodePipeline V2 (`po-ingest-pipeline`) → CodeBuild (`po-ingest-build`). Pushes to `main` auto-deploy via `buildspec.yml`.
|
|
|
|
## CI/CD
|
|
|
|
Merges to `main` trigger the `po-ingest-pipeline` (CodePipeline V2) which runs CodeBuild to `cdk deploy`. The pipeline uses the existing CodeStar connection to the Sea-Haven-Industries GitHub org.
|
|
|
|
**Branch protection:** `main` requires a PR (no direct push), no deletion, no force push.
|
|
|
|
## Setup
|
|
|
|
1. Bootstrap CDK in the account if you haven't already: `cdk bootstrap aws://{AccountId}/us-east-1`.
|
|
2. Store the Anthropic API key:
|
|
```bash
|
|
aws secretsmanager create-secret \
|
|
--name po-ingest/anthropic-api-key \
|
|
--secret-string "sk-ant-..."
|
|
```
|
|
3. Install Lambda dependencies into the deployable package directory (gitignored):
|
|
```bash
|
|
pip install -r lambdas/email_processor/requirements.txt -t lambdas/email_processor/package/
|
|
```
|
|
4. Deploy:
|
|
```bash
|
|
cd cdk
|
|
pip install -r requirements.txt
|
|
cdk deploy
|
|
```
|
|
5. The `WebUIUrl` CloudFormation output is the dashboard URL.
|
|
|
|
## Reprocessing
|
|
|
|
To re-run the processor against every email still sitting in `inbound/` (useful after a parser change):
|
|
|
|
```bash
|
|
python scripts/reprocess.py # dry-run — lists keys
|
|
python scripts/reprocess.py --execute # invokes po-email-processor for each
|
|
```
|
|
|
|
Inserts are conditional on `po_number`, so re-processing existing POs is a no-op.
|
|
|
|
## Backfilling verified sites
|
|
|
|
The stream Lambda handles all future POs automatically. To backfill from historical PO data (one-time):
|
|
|
|
```bash
|
|
python scripts/backfill_sites.py
|
|
```
|
|
|
|
Uses the same extraction logic as the Lambda. Idempotent — safe to re-run.
|