procurement-ingest/README.md

Procurement Ingest

Unified email ingestion pipelines for Amazon procurement data. Two independent pipelines — purchase orders (Coupa) and work orders (APM/Hexagon EAM) — share a single repo and CDK app but deploy as separate CloudFormation stacks.

Pipelines

Purchase Orders (po-ingest stack)

Coupa PO emails are received at amazon_po@int.seahaven.com, parsed by Claude Haiku 4.5, and written to the purchase-orders DynamoDB table.

Flow:

1. Coupa sends a PO email (new, revision, or cancellation).
2. SES (INBOUND_MAIL rule set) drops the raw MIME into s3://po-ingest-emails-{AccountId}/inbound/.
3. S3 ObjectCreated triggers the po-email-processor Lambda.
4. The processor rejects the email unless the verified sender domain is on the allowlist (see Security) — anyone can email the public address, but only legitimate Coupa/Amazon senders may write data.
5. Claude extracts structured JSON (PO number, status, supplier, site code, trade classification, line items, fiscal year).
6. Merge write to DynamoDB:
   • new_po — merge insert. Creates the PO, or backfills data into a pre-existing Cancelled skeleton left by an out-of-order cancellation (preserving the Cancelled status). No longer silently dropped when a record already exists.
   • revision — field-level merge (update_item SETs only the fields present in the revision). A revision that omits line_items/supplier no longer deletes them. Will not un-cancel a Cancelled PO.
   • cancellation — marks the row Cancelled (creating a minimal skeleton if the cancellation arrives before the new_po).
7. DynamoDB Streams feeds downstream consumers:
   • LedgerFlow (seahaven-slack-bot/po-sync) — daily KB sync
   • Site extractor (po-ingest-site-extractor) — real-time site address extraction into verified-sites table

Lambdas (lambdas/po/):

Function	Trigger	Purpose
po-email-processor	S3 ObjectCreated	Claude extraction + DynamoDB write
po-ingest-site-extractor	DynamoDB Streams	Site code/address extraction -> verified-sites
po-web-ui	Manual invoke	HTML dashboard (public Function URL removed 2026-06-08, INFRA-74)

Tables:

• purchase-orders (PK: po_number, Streams: NEW_IMAGE) — shared with seahaven-slack-bot (read-only; see Shared Resources)
• verified-sites (PK: siteCode, GSI: by-state) — ~1,100 unique Amazon facility sites
• pending-site-review (PK: po_number) — unresolvable POs for manual Payee Central verification

Work Orders (WorkorderIngestStack stack)

Amazon APM work order emails (from Hexagon EAM / HxGN SmartCloud) are received at apm@int.seahaven.com, parsed by Claude Haiku 4.5, and written to the WorkOrders DynamoDB table.

Flow:

1. Hexagon EAM sends email notifications (new assignments, comments, updates, cancellations) to amazon@seahavenind.com.
2. Gmail filter forwards APM emails to apm@int.seahaven.com (SES).
3. SES drops the raw MIME into s3://workorder-ingest-emails-{AccountId}/inbound/.
4. S3 triggers the workorder-email-processor Lambda.
5. Claude extracts structured JSON (work order ID, site code, severity, priority, dates, assigned technician).
6. Work order upserted to WorkOrders, event/comment appended to WorkOrderComments.

Lambdas (lambdas/wo/):

Function	Trigger	Purpose
workorder-email-processor	S3 ObjectCreated	Claude extraction + DynamoDB write
workorder-web-ui	Manual invoke	HTML dashboard (public Function URL removed 2026-06-08, INFRA-74)

Tables:

• WorkOrders (PK: work_order_id, GSIs: site-code-index, status-index)
• WorkOrderComments (PK: work_order_id, SK: comment_id)

Architecture

IaC: AWS CDK (Python), two stacks in one app, region us-east-1.

All Lambdas: Python 3.12, ARM64, 60-day log retention.

Secrets:

• po-ingest/anthropic-api-key — Anthropic API key for PO parsing
• workorder-ingest/anthropic-api-key — Anthropic API key for WO parsing

The email processors require ANTHROPIC_API_KEY_SECRET_ARN to be set and resolve the key from Secrets Manager at runtime. There is no plaintext ANTHROPIC_API_KEY env fallback — a deploy missing the ARN fails loudly instead of silently running on an unmanaged key.

SES: Both stacks add rules to the shared INBOUND_MAIL receipt rule set on int.seahaven.com.

Security

Sender authorization (email processors). The public receipt addresses (amazon_po@, apm@) accept mail from anyone, so each processor validates the verified sender domain against an allowlist before any DynamoDB write. The allowlist is configurable per function via the ALLOWED_SENDER_DOMAINS env var (comma-separated; a domain matches itself or any subdomain). Defaults: PO = coupahost.com,amazon.com; WO = amazon.com,hxgnsmartcloud.com,hexagon.com. When SES receipt-rule scanning is enabled, the processors additionally drop any email whose stored object shows an explicit SPF/DKIM/spam/virus fail.

Web UI auth (defense-in-depth). The po-web-ui / workorder-web-ui handlers refuse unauthenticated requests even though their public Function URLs were removed (INFRA-74). Each requires a shared secret in the X-Auth-Token header (or Authorization: Bearer <token>), compared in constant time against the WEB_UI_AUTH_TOKEN env var. The handler fails closed if the token is unset (denies all). The CDK wires WEB_UI_AUTH_TOKEN from the SSM String parameter /procurement-ingest/web-ui-auth-token — create it before deploy. (A plaintext String, not a SecureString, is used because value_for_string_parameter only resolves String params into Lambda env vars at deploy; this is a defense-in-depth floor for a detached URL, not primary auth.)

Output escaping. All caller-influenced values (including prompt-injectable strings Claude may return for total_amount/line-item amounts) are HTML-escaped before interpolation to prevent stored XSS.

Failure handling (INFRA-41): Each email-processor is async-invoked (S3 → Lambda). Both have a CDK-managed SQS dead-letter queue (dead_letter_queue=, 14-day retention, SSL-enforced) so a failed parse is captured rather than silently dropped after Lambda's retries, plus an ALARM-only CloudWatch Errors alarm (Sum, threshold > 0) wired to the shared site-alerts SNS topic.

Shared Resources

purchase-orders table (owned here)

The purchase-orders DynamoDB table is owned by this repo's po-ingest stack (defined in cdk/po_stack.py with RemovalPolicy.RETAIN and StreamViewType.NEW_IMAGE). The po-email-processor Lambda is the authoritative writer — it performs the conditional inserts, revision overwrites, and cancellation updates described above.

Consumers (read-only):

Repo	How it reads	Purpose
seahaven-slack-bot	po-sync (DynamoDB Streams + daily scan) and wo-po-lookup	Daily KB sync + Bedrock agent PO lookups

The consumer imports the table via Table.fromTableName(...) and is granted read-only access (grantReadData); it does not own or define it.

Schema-coordination rule: Any change to the purchase-orders schema (partition key, item shape, attribute names, streams view type) must be coordinated with seahaven-slack-bot. The owner here ships the change; the consumer must be updated in lockstep so its readers do not break. Treat schema changes as a cross-repo migration, not a local edit.

Known exception (INFRA-51): amazon-po-parser currently writes directly to purchase-orders outside this stack (backfill/enrichment scripts). This second writer is being folded into the po-ingest pipeline so this stack is the sole writer; until INFRA-51 closes, coordinate any schema change with amazon-po-parser as well.

CI/CD

GitHub Actions with reusable workflows from Sea-Haven-Industries/.github:

• CI (PR to main): linting + cdk synth via ci-python-sam.yaml@main
• CD (push to main): cdk deploy --all via cd-cdk.yaml@main (OIDC auth)

Branch protection on main — all changes through PR.

Setup

1. Bootstrap CDK: cdk bootstrap aws://{AccountId}/us-east-1
2. Store Anthropic API keys:

   aws secretsmanager create-secret --name po-ingest/anthropic-api-key --secret-string "sk-ant-..."
   aws secretsmanager create-secret --name workorder-ingest/anthropic-api-key --secret-string "sk-ant-..."
   

3. Store the web UI shared secret (the web-ui Lambdas fail closed without it):

   aws ssm put-parameter --name /procurement-ingest/web-ui-auth-token --type String --value "$(openssl rand -hex 32)"
   

4. Deploy both stacks:

   cd cdk
   pip install -r requirements.txt
   cdk deploy --all
   

The web UI Lambdas have no public Function URL (INFRA-74); invoke them only through an authenticated path that forwards the X-Auth-Token header.

Scripts

Reprocess PO emails (re-run parser against all emails still in S3):

python scripts/reprocess.py            # dry-run
python scripts/reprocess.py --execute  # invoke po-email-processor for each

Backfill verified sites (one-time scan of historical POs):

python scripts/backfill_sites.py

Directory Structure

cdk/
  app.py              # Two stacks: po-ingest + WorkorderIngestStack
  po_stack.py          # Purchase order pipeline resources
  wo_stack.py          # Work order pipeline resources
lambdas/
  po/                  # PO pipeline Lambdas
    email_processor/
    site_extractor/
    web_ui/
  wo/                  # WO pipeline Lambdas
    email_processor/
    web_ui/
shared/
  models.py            # Work order dataclasses/enums
scripts/
  reprocess.py
  backfill_sites.py