mirror of https://github.com/Sea-Haven-Industries/procurement-ingest.git synced 2026-09-30 13:03:14 +00:00

Merge workorder-ingest into unified procurement repo (#22 )

* Merge workorder-ingest pipeline into unified repo

Move PO lambdas under lambdas/po/, add WO pipeline under lambdas/wo/.
Two independent CloudFormation stacks in one CDK app. Fix WO stack
compliance: ARM64 architecture, 60-day log retention, aarch64 bundling,
RETAIN on Anthropic secret. Remove stale CodePipeline buildspec.

* Fix test_local.py import path and remove dead shared/models.py

test_local.py referenced the old lambdas/email_processor path. Updated
to lambdas/wo/email_processor. Removed shared/ directory entirely as
nothing imports from it.

* Escape HTML in both web UI dashboards to prevent XSS

Both Function URLs are public (auth_type=NONE) and render
email-derived content via f-strings. Attacker-crafted emails
could inject scripts. Added html.escape() on all interpolated
values in both PO and WO dashboards.

* Add pagination to WO web UI scan

get_work_orders() only fetched the first 1MB page from DynamoDB.
Loop on LastEvaluatedKey to match the PO web UI pattern.

* Fix esc(None) TypeError and javascript: scheme in PO web UI

Coerce supplier name through `or ""` before escaping to handle
nested None from DynamoDB. Add scheme allowlist on view_order_url
to block javascript:/data: hrefs from LLM-extracted URLs.

* Fix WO render_badge None guard, updated_at slice, and backfill path

Add null guard to WO render_badge matching the PO version. Use
`or ""` before slicing updated_at to handle explicit None values.
Fix backfill_sites.py sys.path to use new lambdas/po/site_extractor.

* Harden WO web UI and fix JS-context XSS in both dashboards

- Use json.dumps for onclick URLs to prevent JS string breakout
- Add .lower() to WO render_badge color lookup matching PO pattern
- Add pagination to get_comments query
- Cap get_work_orders to 500 results matching PO pattern

* Apply ruff formatting to web UI handlers

2026-05-12 15:21:06 -04:00

5 KiB

Raw Blame History

Procurement Ingest

Unified email ingestion pipelines for Amazon procurement data. Two independent pipelines — purchase orders (Coupa) and work orders (APM/Hexagon EAM) — share a single repo and CDK app but deploy as separate CloudFormation stacks.

Pipelines

Purchase Orders (`po-ingest` stack)

Coupa PO emails are received at amazon_po@int.seahaven.com, parsed by Claude Haiku 4.5, and written to the purchase-orders DynamoDB table.

Flow:

Coupa sends a PO email (new, revision, or cancellation).
SES (INBOUND_MAIL rule set) drops the raw MIME into s3://po-ingest-emails-{AccountId}/inbound/.
S3 ObjectCreated triggers the po-email-processor Lambda.
Claude extracts structured JSON (PO number, status, supplier, site code, trade classification, line items, fiscal year).
Conditional write to DynamoDB:
- new_po — idempotent insert (no-op if PO exists)
- revision — unconditional overwrite
- cancellation — marks existing row Cancelled
DynamoDB Streams feeds downstream consumers:
- LedgerFlow (seahaven-slack-bot/po-sync) — daily KB sync
- Site extractor (po-ingest-site-extractor) — real-time site address extraction into verified-sites table

Lambdas (lambdas/po/):

Function	Trigger	Purpose
`po-email-processor`	S3 ObjectCreated	Claude extraction + DynamoDB write
`po-ingest-site-extractor`	DynamoDB Streams	Site code/address extraction -> `verified-sites`
`po-web-ui`	Function URL	HTML dashboard

Tables:

purchase-orders (PK: po_number, Streams: NEW_IMAGE) — shared with payments-dashboard and seahaven-slack-bot
verified-sites (PK: siteCode, GSI: by-state) — ~1,100 unique Amazon facility sites
pending-site-review (PK: po_number) — unresolvable POs for manual Payee Central verification

Work Orders (`WorkorderIngestStack` stack)

Amazon APM work order emails (from Hexagon EAM / HxGN SmartCloud) are received at apm@int.seahaven.com, parsed by Claude Haiku 4.5, and written to the WorkOrders DynamoDB table.

Flow:

Hexagon EAM sends email notifications (new assignments, comments, updates, cancellations) to amazon@seahavenind.com.
Gmail filter forwards APM emails to apm@int.seahaven.com (SES).
SES drops the raw MIME into s3://workorder-ingest-emails-{AccountId}/inbound/.
S3 triggers the workorder-email-processor Lambda.
Claude extracts structured JSON (work order ID, site code, severity, priority, dates, assigned technician).
Work order upserted to WorkOrders, event/comment appended to WorkOrderComments.

Lambdas (lambdas/wo/):

Function	Trigger	Purpose
`workorder-email-processor`	S3 ObjectCreated	Claude extraction + DynamoDB write
`workorder-web-ui`	Function URL	HTML dashboard

Tables:

WorkOrders (PK: work_order_id, GSIs: site-code-index, status-index)
WorkOrderComments (PK: work_order_id, SK: comment_id)

Architecture

IaC: AWS CDK (Python), two stacks in one app, region us-east-1.

All Lambdas: Python 3.12, ARM64, 60-day log retention.

Secrets:

po-ingest/anthropic-api-key — Anthropic API key for PO parsing
workorder-ingest/anthropic-api-key — Anthropic API key for WO parsing

SES: Both stacks add rules to the shared INBOUND_MAIL receipt rule set on int.seahaven.com.

CI/CD

GitHub Actions with reusable workflows from Sea-Haven-Industries/.github:

CI (PR to main): linting + cdk synth via ci-python-sam.yaml@main
CD (push to main): cdk deploy --all via cd-cdk.yaml@main (OIDC auth)

Branch protection on main — all changes through PR.

Setup

Bootstrap CDK: cdk bootstrap aws://{AccountId}/us-east-1

Store Anthropic API keys:

aws secretsmanager create-secret --name po-ingest/anthropic-api-key --secret-string "sk-ant-..."
aws secretsmanager create-secret --name workorder-ingest/anthropic-api-key --secret-string "sk-ant-..."

Deploy both stacks:

cd cdk
pip install -r requirements.txt
cdk deploy --all

CloudFormation outputs include WebUIUrl for each stack's dashboard.

Scripts

Reprocess PO emails (re-run parser against all emails still in S3):

python scripts/reprocess.py            # dry-run
python scripts/reprocess.py --execute  # invoke po-email-processor for each

Backfill verified sites (one-time scan of historical POs):

python scripts/backfill_sites.py

Directory Structure

cdk/
  app.py              # Two stacks: po-ingest + WorkorderIngestStack
  po_stack.py          # Purchase order pipeline resources
  wo_stack.py          # Work order pipeline resources
lambdas/
  po/                  # PO pipeline Lambdas
    email_processor/
    site_extractor/
    web_ui/
  wo/                  # WO pipeline Lambdas
    email_processor/
    web_ui/
shared/
  models.py            # Work order dataclasses/enums
scripts/
  reprocess.py
  backfill_sites.py

5 KiB Raw Blame History