procurement-ingest/README.md

# Procurement Ingest

![Python](https://img.shields.io/badge/Python-3776AB?logo=python&logoColor=white)
![AWS CDK](https://img.shields.io/badge/AWS-CDK-FF9900?logo=amazonaws&logoColor=white)
![CI](https://github.com/Sea-Haven-Industries/procurement-ingest/actions/workflows/ci.yaml/badge.svg)

Unified email ingestion pipelines for Amazon procurement data. Two independent pipelines — purchase orders (Coupa) and work orders (APM/Hexagon EAM) — share a single repo and CDK app but deploy as separate CloudFormation stacks.

## Pipelines

### Purchase Orders (`po-ingest` stack)

Coupa PO emails are received at `amazon_po@int.seahaven.com`, parsed **deterministic-template-first with a Claude-Haiku-4.5-on-Bedrock fallback**, and written to the `purchase-orders` DynamoDB table.

**Flow:**
1. Coupa sends a PO email (new, revision, or cancellation).
2. SES (`INBOUND_MAIL` rule set) drops the raw MIME into `s3://po-ingest-emails-{AccountId}/inbound/`.
3. S3 `ObjectCreated` triggers the `po-email-processor` Lambda.
4. Fail-closed sender authentication (INFRA-107): the SES-stamped `Authentication-Results` header must show `dkim=pass` for `amazon.coupahost.com` (see [Sender authentication](#sender-authentication-infra-107)); otherwise the email is logged and dropped.
5. **Parse:** a pure, offline template parser (`template_parser.py`) tries the two known Coupa templates first — `coupa_new_po` (95.5% of traffic) and `coupa_cancellation` (2.9%) — behind a strict fail-closed validation gate. Only on a miss/invalid result does the Lambda fall back to the Claude-on-Bedrock AI extractor (`InvokeModel`), which extracts the identical structured-JSON contract (PO number, status, supplier, nested ship-to, line items). Numeric amounts are `Decimal` on both paths (the AI decode uses `parse_float=Decimal`; DynamoDB rejects floats). The derived classification fields (`site_code`, `trade`, `fiscal_year`) are computed by a deterministic Python classifier in the shared `enrich_parsed()` post-stage (zip padding, `ship_to_raw`/`state` promotion, metadata) which runs identically on both paths — Python-authoritative on the template path and an LLM-authoritative-with-Python-shadow bake on the AI-fallback path (see the **Derived fields** note below).
6. Merge write to DynamoDB:
   - `new_po` — merge insert. Creates the PO, or backfills data into a pre-existing `Cancelled` skeleton left by an out-of-order cancellation (preserving the `Cancelled` status). No longer silently dropped when a record already exists.
   - `revision` — field-level merge (`update_item` SETs only the fields present in the revision). A revision that omits `line_items`/`supplier` no longer deletes them. Will not un-cancel a `Cancelled` PO.
   - `cancellation` — marks the row `Cancelled` (creating a minimal skeleton if the cancellation arrives before the `new_po`).
7. DynamoDB Streams feeds downstream consumers:
   - **LedgerFlow** (`seahaven-slack-bot/po-sync`) — daily KB sync
   - **Site extractor** (`po-ingest-site-extractor`) — real-time site address extraction into `verified-sites` table

**Deterministic template parser.** `template_parser.try_deterministic_parse()` classifies by exact subject regex, extracts the nested contract (`supplier{}`, `ship_to{}` — 8 keys, `line_items[]` — 10 keys per item), and returns a parsed result **only if** it passes a fail-closed validation gate: recursive exact key-set at every nesting level; `email_type` emitted **only** on the exact new-PO subject *and* a confirmed-safe `Status` (never defaulted — a cancellation misrouted as `new_po` would defeat the sticky-`Cancelled` guard); `po_number` shape + byte-equality with the subject, the body `PO ID`, the `Amazon Purchase Order #` heading, and the `orders/<id>` URL; duplicate-label anchor integrity (`Supplier`/`Shipping`/`Total` each appear twice, the first `Shipping` must be the literal `None` placeholder); money fidelity (every `Decimal` re-serializes byte-identically to its source token with a digit/comma border check — the thousands-separator-truncation kill switch — plus `sum(line amounts) == total`, proven against **both** `Total` blocks); bullet-metadata label discipline (closed label set, assigned by leading label, never ordinal — immune to the optional `Part Number` segment); USPS address shape on the raw pre-enrichment value; and rejection of any unparseable sentinel or residual `\r`/`\xa0` artifact. Multi-line-item (0.18%) and non-USD (0 observed) new-POs, comment emails, and anything else falls back to the AI extractor. The AI path is gated too: the untrusted email reaches Bedrock inside a neutralized `<email>` data block (forged tag lookalikes in the body are defanged) with `temperature=0`, and the raw model output must pass the fail-closed `validate_ai_fallback()` gate — the PO-specific nested contract (exact key-set at every level, with missing keys normalized rather than rejected), a `po_number` shape check hardened against fullwidth-digit and trailing-artifact injection (the same regex family protecting the DynamoDB partition key the handler builds from it), an `email_type` allow-list enforced *before* dispatch so a miss can never fall into the `new_po` default branch, and `Decimal`/`int`/`None` money typing (PO decodes with `parse_float=Decimal`) — before any DynamoDB write. **The template path is gate-enforced end-to-end; the AI-fallback path is validated and fail-closed — output that fails the gate is skipped, never written (see `ai_fallback_rejected` below), so a malformed or injected email raises the fallback rate rather than corrupting a record.** Every record emits one CloudWatch EMF metric (see below).

**Derived fields (`site_code`, `trade`, `fiscal_year`).** These three are **classified**, not verbatim-extracted, so neither parse path emits them directly — instead a deterministic Python classifier (`derived_fields.derive_all()`) computes them inside the shared `enrich_parsed()` post-stage, which runs identically on both paths. `trade` is a closed label set (the same one the AI prompt enumerates): `Plumbing - PM`, `Plumbing - Reactive`, `Electrical`, `HVAC`, `Dock Doors`, `Doors`, `Signage`, `Carpentry`, `Fencing/Gates`, `Conveyance/MHE`, `Painting`, `Flooring`, `Janitorial`, `Fire/Life Safety`, `Landscaping/Yard`, `Roofing`, `Security/Locksmith`, `Snow Removal`, `PO Uplift`, `General Building - Emergency`, `General Building - Handyman`, `General Building - Project`, `General Building`. Authority differs by path:

- **Template path** — the parser leaves all three `null`, so Python is authoritative: it fills them.
- **AI-fallback path** — the LLM value stays **authoritative during the bake** (Python fills only a gap the LLM left `null`, never overwrites), and Python additionally runs in **shadow mode**: `enrich_parsed()` emits one `DerivedFieldAgreement` EMF record per field comparing the two.

**`DerivedFieldAgreement` metric** — namespace `Seahaven/PoIngest`, metric name `DerivedFieldAgreement` (Unit Count, value 1), dimensioned **only** by `Field` × `Agreement` (cardinality fixed at 3 × 4). `Agreement` ∈ {`agree` (both non-null, equal after str-strip), `disagree` (both non-null, different), `llm_null_python_filled` (LLM null, Python supplied a value), `python_null` (LLM non-null, Python null)}; the record is skipped entirely when both are null. `po_number`, `PythonValue`, and `LlmValue` ride along as non-dimensioned Logs-Insights properties so a disagreement can be reviewed by example without inflating cardinality. Only the **AI-fallback** path emits these — the template path has no LLM value to shadow.

Review disagreements during the bake with this CloudWatch Logs Insights query over `/aws/lambda/po-email-processor`:

```
fields @timestamp, Field, Agreement, LlmValue, PythonValue, po_number
| filter ispresent(DerivedFieldAgreement)
| filter Agreement = "disagree"
| sort @timestamp desc
| limit 200
```

Or aggregate overall agreement per field: `| filter ispresent(DerivedFieldAgreement) | stats count(*) by Field, Agreement`.

> **Post-bake follow-up:** once agreement is acceptable, make Python authoritative on the AI-fallback path too (stop keeping the LLM value) and **drop the `site_code`/`trade`/`fiscal_year` rule sections from `EXTRACTION_PROMPT`** — the LLM stays authoritative on the fallback path only until then.

**Lambdas** (`lambdas/po/`):
| Function | Trigger | Purpose |
|---|---|---|
| `po-email-processor` | S3 ObjectCreated | Claude extraction + DynamoDB write |
| `po-ingest-site-extractor` | DynamoDB Streams | Site code/address extraction -> `verified-sites` |
| `po-web-ui` | Manual invoke | HTML dashboard (public Function URL removed 2026-06-08, INFRA-74) |

**Tables:**
- `purchase-orders` (PK: `po_number`, Streams: NEW_IMAGE) — shared with seahaven-slack-bot (read-only; see Shared Resources)
- `verified-sites` (PK: `siteCode`) — ~1,100 unique Amazon facility sites (`by-state` GSI removed 2026-06-03, audit M-20)
- `pending-site-review` (PK: `po_number`) — unresolvable POs for manual Payee Central verification

### Work Orders (`WorkorderIngestStack` stack)

Amazon APM work order emails (from Hexagon EAM / HxGN SmartCloud) are received at `apm@int.seahaven.com`, parsed **deterministic-template-first with a Claude-on-Bedrock fallback**, and written to the `WorkOrders` DynamoDB table.

**Flow:**
1. Hexagon EAM sends email notifications (new assignments, comments, updates, cancellations) to `amazon@seahavenind.com`.
2. Gmail filter forwards APM emails to `apm@int.seahaven.com` (SES).
3. SES drops the raw MIME into `s3://workorder-ingest-emails-{AccountId}/inbound/`.
4. S3 triggers the `workorder-email-processor` Lambda.
5. Fail-closed sender authentication (INFRA-107): the SES-stamped `Authentication-Results` header must show `dkim=pass` for the domain that re-signs the forward (currently allowlisted as `seahaven.com` — see the validation caveat under [Sender authentication](#sender-authentication-infra-107)); otherwise the email is logged and dropped.
6. **Parse:** a pure, offline template parser (`template_parser.py`) tries the two known Hexagon templates first, behind a strict fail-closed validation gate. Only on a miss/invalid result does the Lambda fall back to the Claude-on-Bedrock AI extractor. Both paths emit the identical structured-JSON contract (work order ID, site code, severity, priority, dates, assigned technician).
7. Work order upserted to `WorkOrders`, event/comment appended to `WorkOrderComments`.

**Deterministic template parser.** ~93.6% of WO traffic is the plain-text "AMAZON UPDATE WO DETAILS \<id\>" comment template (T1) and ~6.4% is the HTML "AMAZON assign Work Order \<id\> on building \<SITE\>" assignment template (T2). `template_parser.try_deterministic_parse()` classifies by subject, extracts the contract fields, and returns a parsed result **only if** it passes a strict validation gate (exact contract-key set; `work_order_id` matches the subject and is all-digits; the T1 `Work Order:  <id>` double-space is literally present; `email_type` matches the template; `site_code` shape; per-type required fields; and a label-bleed guard so a value that over-ran into another field fails). Anything that fails — the rare update/cancellation shapes, Hexagon template drift, or an extractor exception — falls back to the AI extractor. The AI path is gated too: the untrusted email reaches Bedrock inside a neutralized `<email>` data block (tag lookalikes in the body are defanged), and the raw model output must pass the fail-closed `validate_ai_fallback()` schema/enum/date gate before any DynamoDB write — output that fails is dropped and paged (see the `ai-fallback-rejected` alarm below), a prompt-injection defence for DKIM-passing but attacker-influenced mail. **Data is never corrupted; only the fallback rate rises.** Every record emits one CloudWatch EMF metric (see below).

**Lambdas** (`lambdas/wo/`):
| Function | Trigger | Purpose |
|---|---|---|
| `workorder-email-processor` | S3 ObjectCreated | Claude extraction + DynamoDB write |
| `workorder-web-ui` | Manual invoke | HTML dashboard (public Function URL removed 2026-06-08, INFRA-74) |

**Tables:**
- `WorkOrders` (PK: `work_order_id`) — `site-code-index` and `status-index` GSIs removed 2026-06-03 (audit M-20)
- `WorkOrderComments` (PK: `work_order_id`, SK: `comment_id`) — see the `comment_id` format note below

## Architecture

**IaC:** AWS CDK (Python), two stacks in one app, region `us-east-1`.

All Lambdas: Python 3.12, ARM64, 60-day log retention.

**LLM provider — Amazon Bedrock.** Both email processors call Claude Haiku 4.5 through the Bedrock inference profile `us.anthropic.claude-haiku-4-5-20251001-v1:0` (`bedrock-runtime` `InvokeModel`), configured via the `BEDROCK_MODEL_ID` env var. There is **no Anthropic API key and no Secrets Manager secret** any more. Each processor role is granted `bedrock:InvokeModel` + `bedrock:InvokeModelWithResponseStream` on **both** the inference-profile ARN **and** the per-region foundation-model ARNs for `us-east-1` / `us-east-2` / `us-west-2` (empty-account foundation-model ARNs) — the `us.*` profile can route cross-region, so a profile-only grant would `AccessDenied` at runtime under load.

> **Retired secrets (manual cleanup outstanding):** the former secrets `po-ingest/anthropic-api-key` and `workorder-ingest/anthropic-api-key` had `RemovalPolicy.RETAIN`, so removing them from the CDK stacks **orphans** them rather than deleting them. Delete both by hand post-deploy and revoke the stored keys at the provider. The processors no longer read any `ANTHROPIC_API_KEY_SECRET_ARN` — authentication to Bedrock is via the Lambda execution-role IAM grant, so there is no provider API key or Secrets Manager fetch on the parse path.

**SES:** Both stacks add rules to the shared `INBOUND_MAIL` receipt rule set on `int.seahaven.com`.

### Sender authentication (INFRA-107)

The `From` header and any `Authentication-Results` header inside the raw MIME are attacker-forgeable, so neither is trusted. Instead, both email processors (`lambdas/*/email_processor/ses_auth.py`) authenticate the sender against the verdicts SES itself stamps at delivery time, failing closed:

1. Take **only the topmost** `Authentication-Results` header (SES prepends its trace headers; any lower copies arrived inside the message and are ignored).
2. Require its authserv-id to be `amazonses.com`.
3. Require a `dkim=pass` clause whose `header.d=`/`header.i=` domain is in the pipeline's allowlist.

The allowlist is the `ALLOWED_DKIM_DOMAINS` Lambda environment variable (comma-separated, set per stack in CDK — no code change needed to adjust):

| Pipeline | `ALLOWED_DKIM_DOMAINS` | Why |
|---|---|---|
| Work orders | `seahaven.com` | APM mail reaches `apm@int.seahaven.com` via a forward off `amazon@seahavenind.com`; the allowlist trusts the domain that **re-signs** DKIM on that forward (the original `hxgnsmartcloud.com` signature does not survive it). **Validated against live SES-stamped headers (2026-07-16)** — real APM deliveries carry `dkim=pass header.i=@seahaven.com`. Note a plain Gmail auto-forward re-signs under the *sending Workspace* domain (`seahavenind.com` / a `*.gappssmtp.com` key), **not** `seahaven.com`; only a Google Group (or Workspace routing) with "sign as `seahaven.com`" produces `dkim=pass header.i=@seahaven.com`. If the observed re-signing domain differs, update this value (do **not** widen it to a shared key like `*.gappssmtp.com`, which any Google customer's mail would pass). The `workorder-email-processor-sender-auth-rejected` alarm pages if this assumption is wrong instead of silently dropping every work order. |
| Purchase orders | `amazon.coupahost.com` | Coupa signs as `amazon.coupahost.com`. `amazonses.com` also passes but is deliberately not allowlisted — every SES customer's mail passes for it |

> **Clause-injection hardening:** SES echoes attacker-controlled SMTP-session tokens (`envelope-from`, `helo`, `header.from`) into its own `Authentication-Results` value, and an RFC 5321 quoted-local-part MAIL FROM may legally contain `;` and spaces. The parser therefore tokenises comment- and quoted-string-aware (RFC 8601 / RFC 5322): CFWS comments `(...)` are stripped and clauses are split only on semicolons **outside** a quoted string, so a `;` inside a quoted `envelope-from=` value can never be torn into a forged `dkim=pass` clause. The DKIM signer domain is read from `header.d=` when present (falling back to `header.i=`, taking the domain after the AUID's last top-level `@` so a quoted local-part cannot smuggle an allowlisted domain). Two latent comment-parsing edge cases (early comment-close, no-separator-on-strip) are tracked as hardening follow-ups — see the SES-AR-01/02 issue; neither is reachable through SES's real header encoding today.

> **Risk acceptance — forwarder-domain binding (INFRA-107, accepted 2026-07-16):** for work orders this control authenticates the domain that *re-signs* the `apm@` forward (`seahaven.com`), not the Hexagon originator (`hxgnsmartcloud.com`, whose signature does not survive the forward). Its strength therefore rests on the `apm@` Google Group's posting policy being restricted to trusted internal senders — that restriction is the **load-bearing control** and is accepted as documented risk. **If the `apm@` group is ever opened to external posting, this finding escalates to HIGH** (anyone able to post to the group could inject a forged work order) and the correct fix is to bind acceptance to the originator via DMARC alignment rather than the forwarder's re-signature. The PO pipeline is unaffected — `amazon.coupahost.com` is an external domain an attacker cannot get SES to sign.

On any failure (env var unset, header missing/unparseable, verdict fail, unaligned domain) the processor logs a structured `sender_auth_rejected` warning with the reason and S3 key, skips the email, and returns normally — rejected mail never triggers Lambda retries or DLQ messages, but the `<fn>-sender-auth-rejected` CloudWatch alarm (see [CloudWatch alarms](#cloudwatch-alarms)) pages on a rejection spike so a drift-induced outage is not silent. Unit tests live in `tests/test_ses_auth.py`.

**Failure handling (INFRA-41):** Each email-processor is async-invoked (S3 → Lambda). Both have a CDK-managed SQS dead-letter queue (`dead_letter_queue=`, 14-day retention, SSL-enforced) so a failed parse is captured rather than silently dropped after Lambda's retries.

### CloudWatch alarms

Every alarm is **ALARM-only** (no OK action), sends to the shared `site-alerts` SNS topic (imported once per stack via `Topic.from_topic_arn`), and uses `TreatMissingData.NOT_BREACHING`.

**Lambda alarms** (`AWS/Lambda`, `FunctionName` dimension):

| Alarm | Functions | Metric / config |
|---|---|---|
| `<fn>-errors` | `po-email-processor`, `po-ingest-site-extractor`, `workorder-email-processor` | `Errors` Sum, 5 min, `> 0`, eval 1 |
| `<fn>-throttles` | `po-email-processor`, `po-ingest-site-extractor`, `po-web-ui`, `workorder-email-processor` | `Throttles` Sum, 5 min, `> 0`, eval 1 |
| `<fn>-duration` | `po-email-processor`, `po-ingest-site-extractor`, `po-web-ui` (p99); `workorder-email-processor` (p95) | `Duration` percentile, 5 min, `>= 45000` ms (75% of the 60s timeout), eval 3 / datapoints 2 |
| `<fn>-sender-auth-rejected` | `po-email-processor`, `workorder-email-processor` | Log-metric-filter count (namespace `Seahaven/ProcurementIngest`, `default_value=0`) on `sender_auth_rejected` warnings, `Sum` 5 min, `>= 1`, eval 3 / datapoints 2 |

The `<fn>-sender-auth-rejected` alarm closes the silent-drop gap in INFRA-107: a rejected email returns normally (no error, no retry, no DLQ message), so without a log-metric filter a signing-domain drift or a wrong allowlist would discard 100% of legitimate mail while every other alarm stayed green. It counts `sender_auth_rejected` warnings per 5-minute period (`default_value=0` keeps the series continuous) and pages when 2 of the last 3 periods each see at least one rejection — a lone stray spoof probe to the internal ingest address self-clears, but a sustained false-reject storm pages within ~10–15 minutes even at low mail volume; the config is easy to tune in the CDK helper. (A residual gap remains for a *very* sparse total-reject outage — see the SES-AR-01/02 hardening issue.)

The `<fn>-duration` and `<fn>-throttles` alarms for `po-email-processor` and `workorder-email-processor` supersede the orphaned, CLI-created `Lambda-Duration-*` / `Lambda-Throttles-*` alarms (deleted post-deploy).

**DLQ alarms** (`AWS/SQS`): `po-email-processor-dlq-messages` and `workorder-email-processor-dlq-messages` fire when any message is visible on an email-processor DLQ (`ApproximateNumberOfMessagesVisible` Maximum, 5 min, `> 0`, eval 1) — a message there means an email was dropped after Lambda exhausted its async retries.

**Parse-outcome metric + fallback-rate alarm (workorder-ingest):** the WO processor writes one CloudWatch **EMF** line per email to namespace `Seahaven/WorkorderIngest`, metric `ParseOutcome` (Unit Count, value 1), dimensioned by `ParseMethod` (`template` | `ai_fallback` | `ai_fallback_rejected`) and `TemplateId` (`update_plaintext` | `assign_html` | `unknown`). `ai_fallback_rejected` counts AI-fallback output that failed the fail-closed `validate_ai_fallback()` gate (schema/enum/date contract on raw Bedrock output — prompt-injection defence) and was dropped without a DynamoDB write. Non-dimension EMF properties `ReasonCode` and `work_order_id` are queryable in Logs Insights but not promoted to metrics (kept low-cardinality). EMF is used instead of `PutMetricData` so there is no extra sync call / latency / IAM grant on the async hot path (the role already has `logs:PutLogEvents`). The alarm `workorder-email-processor-template-fallback-rate` fires when the AI-fallback share of parses — rejected fallback parses included, so a drift outage whose AI output also fails the gate cannot lower the observed rate while dropping mail — exceeds **15%** sustained (a `MathExpression` with `FILL(...,0)` and a ≥10-sample volume floor over 15-minute periods, eval 3 / datapoints 2) — catching Hexagon template-drift coverage collapse while the volume floor + `FILL` prevent low-volume false pages / `INSUFFICIENT_DATA`. ALARM-only `SnsAction` to `site-alerts`, no OK action, `NOT_BREACHING`. The 15-minute period is a deliberate deviation from the 5-minute house style to accumulate a stable denominator at the low ~760/day volume. A second alarm, `workorder-email-processor-ai-fallback-rejected`, pages on the rejected series itself (≥1 rejection per 5-min period, 2 of the last 6 periods — the sender-auth-rejected sparse-arrival idiom) because a gate rejection drops mail without error/retry/DLQ and would otherwise be silent.

**Parse-outcome metric + fallback-rate alarm (po-ingest):** the PO processor emits the same EMF shape to namespace `Seahaven/PoIngest`, metric `ParseOutcome`, dimensioned by `ParseMethod` (`template` | `ai_fallback` | `ai_fallback_rejected`) and `TemplateId` (`coupa_new_po` | `coupa_cancellation` | `unknown`), with `ReasonCode` (the fail-closed gate reason) and `po_number` as Logs-Insights ride-alongs. `ai_fallback_rejected` counts AI-fallback output that failed the fail-closed `validate_ai_fallback()` gate (nested key-set contract, `po_number` shape, `email_type` allow-list, `Decimal` money typing) and was dropped without a DynamoDB write.

**Deliberate double-count:** unlike WO, PO emits `ParseMethod=ai_fallback` *before* the Bedrock call (so a Bedrock-side error still records the outcome) — a rejected email therefore always emits **both** an `ai_fallback` datapoint (pre-call) and an `ai_fallback_rejected` datapoint (post-gate), never just the latter. This is intentional and load-bearing, not a bug; the fallback-rate math below treats `fb` as already inclusive of every rejection.

The alarm `po-email-processor-template-fallback-rate` is **deliberately retuned for PO volume — do NOT copy the WO numbers**: at ~57 emails/day a 15-minute period holds ~0.6 emails, so the WO ≥10-sample floor would never be met and the alarm would be structurally dead. Instead: **6-hour periods** (~14.25 expected emails each), an `IF((fb+tmpl)>=8, …)` volume floor (at the floor a single fallback email is 12.5% < the threshold, so one email can never breach a datapoint; a breach needs ≥2 fallbacks in one window, or ≥3 at typical volume), threshold **>20%** (expected baseline fallback ≈1%: comments 0.55% + multi-line 0.18% + non-USD 0), **eval 4 / datapoints 2** (a 24h span — isolated noise self-clears while total template drift at 100% fallback pages within ~12h). Sparse overnight/weekend windows below the floor evaluate to 0 (non-breaching by design; accepted trade: a Friday-evening drift may not page until weekend volume accrues). The `ai_fallback_rejected` series (`rej`) is **deliberately excluded** from this expression's numerator, denominator, and volume floor: because the pre-call emit already counts every rejected email once inside `fb`, folding WO's `fb+rej` math in verbatim would double-count each rejection in both terms and inflate the observed rate toward 100% — `fb/(fb+tmpl)` alone is already exact for PO. Same idiom otherwise: ALARM-only `SnsAction` to `site-alerts`, `NOT_BREACHING`, and no element-wise `MAX` in the math expression (the post-#102 rule — the `IF` floor guarantees the non-zero denominator).

A second alarm, `po-email-processor-ai-fallback-rejected`, monitors the rejected series on its own — **retuned for ~57 emails/day, not WO's 5-minute sparse idiom** (which needs two rejections inside one 30-minute window and would be structurally dead at PO volume). It uses the same 6h/`IF`-floor/eval-4/datapoints-2 idiom as the fallback-rate alarm above, but as a plain count-floor on the rejected series itself (`IF(FILL(rej,0)>=1, …)`, no denominator so no divide guard is needed): threshold ≥1, over **6-hour periods**, **eval 4 / datapoints 2** — a lone stray rejection self-clears, while ≥2 rejections landing in ≥2 distinct 6h windows within 24h (sustained prompt-injection probing, or template drift whose AI output also fails the gate) pages within ~12–24h. ALARM-only `SnsAction` to `site-alerts`, `NOT_BREACHING`. Accepted residual: a single isolated rejected email never pages this alarm by itself — it is still visible as an `ai_fallback_rejected` datapoint and in the `ReasonCode` log line, and it has already raised the fallback-rate numerator above via its pre-call `ai_fallback` emit.

**DynamoDB alarms** (`AWS/DynamoDB`): each owned table gets `<table>-throttles` (`ThrottledRequests`) and `<table>-system-errors` (`SystemErrors`). These metrics emit only at the `TableName` + `Operation` dimension set, so each alarm is a `Sum` math expression across the operations the table uses (Get/BatchGet/Query/Scan/Put/Update/Delete/BatchWrite). Tables covered: `purchase-orders`, `verified-sites`, `pending-site-review` (po-ingest); `WorkOrders`, `WorkOrderComments` (workorder-ingest).

## Deploy-Pipeline Guards (Phase 0)

**Goal:** a broken Lambda bundle fails the deploy job, not Monday's first email.

**Healthcheck direct-invoke contract.** Both `po-email-processor` and `workorder-email-processor` recognize a top-level direct-invoke probe payload `{"healthcheck": true}`. In each `handler(event, context)`, the **very first statements** — before any S3 fetch, before `ses_auth`, before iterating `event["Records"]` — are:

```python
if isinstance(event, dict) and event.get("healthcheck") is True:
    return {"healthcheck": "ok"}
```

This placement is deliberate, not incidental: real mail always arrives as an S3 `ObjectCreated` event whose top-level keys (`Records`) AWS controls, so email content can never set a top-level `healthcheck` key — the branch creates no accept path for forged mail. It also emits **no EMF metric and no log line**, so it can never match the `sender_auth_rejected` log-metric-filter pattern that feeds the `<fn>-sender-auth-rejected` alarm (see [CloudWatch alarms](#cloudwatch-alarms)) — that alarm pages at ≥1 match in its window, so repeated healthcheck invokes across deploys (two deploys in ~30 min is routine) must never contribute to it. Unit coverage: `lambdas/po/email_processor/tests/test_po_healthcheck.py` and `lambdas/wo/email_processor/tests/test_healthcheck.py`.

**Post-deploy smoke gate.** `scripts/post-deploy-smoke.sh` is wired into the CD workflow as `cd-cdk.yaml`'s `post-deploy-script` input (see [CI/CD](#cicd)) and runs synchronously after every deploy, before the workflow is considered green. It invokes both `po-email-processor` and `workorder-email-processor` with `aws lambda invoke --invocation-type RequestResponse --payload '{"healthcheck": true}'` (region `us-east-1`) and asserts, per function:

1. The invoke response's **`FunctionError` field is absent** — this is the load-bearing check. A broken bundle (e.g. an `ImportError` at module init from a missing sibling module) still returns HTTP 200 from the Lambda Invoke API with `FunctionError=Unhandled`; a bare exit-code check on `aws lambda invoke` would false-pass on exactly the failure this gate exists to catch.
2. The returned payload is **exactly** `{"healthcheck": "ok"}`.

The script runs `set -euo pipefail` and exits non-zero on any invoke failure, any `FunctionError`, or a payload mismatch on either function, failing the deploy job.

**PO bundling: glob replaces the hand-maintained allowlist.** `cdk/po_stack.py`'s asset bundling command now ships PO's Lambda source with a non-recursive glob, `cp ./*.py /asset-output/`, instead of a hand-maintained list of filenames (`cp handler.py ses_auth.py template_parser.py derived_fields.py /asset-output/`). The glob is functionally identical for today's file set — non-recursive, so `tests/` and other subdirectories are still excluded — but structurally eliminates the failure mode that shipped a broken bundle twice (PR #105 omitted `template_parser.py`; PR #2 nearly omitted `derived_fields.py`): a new sibling module the handler imports now ships automatically instead of requiring someone to remember to add it to the list. WO's bundling (`cdk/wo_stack.py`) already used a recursive `cp -r` of the whole source dir and is unaffected.

`tests/test_bundle_consistency.py` guards both bundling commands with a pure-AST check (no synth, no boto3, no handler import): it parses each handler.py's top-level first-party sibling imports, extracts the bundling `command=[...]` string from the corresponding CDK stack file, and asserts every required sibling module is guaranteed to ship — recognizing the PO glob and the WO recursive copy as unconditionally-safe shapes, and falling back to literal filename matching for any other (allowlist-style) shape. It also pins the PO command to the glob form specifically, so a future revert back to a filename allowlist that omits a sibling fails CI rather than merely relying on the general detection logic. Runs in the existing pytest step, before synth.

## Security

**Web UI auth (defense-in-depth).** The `po-web-ui` / `workorder-web-ui` handlers refuse unauthenticated requests even though their public Function URLs were removed (INFRA-74). Each requires a shared secret in the `X-Auth-Token` header (or `Authorization: Bearer <token>`), compared in constant time against the configured token. The handler **fails closed** if the token is unset or unreadable (denies all). The token lives in the Secrets Manager secret `procurement-ingest/web-ui-auth-token`; only its ARN is passed to the Lambda (`WEB_UI_AUTH_TOKEN_SECRET_ARN`), and the value is fetched at runtime — never embedded in the CloudFormation template or Lambda env vars. The fetched value is cached in the warm container with a short TTL (5 min) so a rotated secret propagates without waiting for the execution environment to recycle. This is a defense-in-depth floor for a detached URL, not primary auth.

**Output escaping.** All caller-influenced values (including prompt-injectable strings Claude may return for `total_amount`/line-item amounts) are HTML-escaped before interpolation to prevent stored XSS.

## Shared Resources

### `purchase-orders` table (owned here)

The `purchase-orders` DynamoDB table is **owned by this repo's `po-ingest` stack** (defined in `cdk/po_stack.py` with `RemovalPolicy.RETAIN`, `StreamViewType.NEW_IMAGE`, and SSE-KMS encryption with the shared customer-managed CMK `alias/seahaven-dynamodb`, INFRA-95 / M-3). The `po-email-processor` Lambda is the authoritative writer — it performs the merge inserts, field-level revision merges, and cancellation updates described above.

**Consumers (read-only):**

| Repo | How it reads | Purpose |
|---|---|---|
| `seahaven-slack-bot` | `po-sync` (DynamoDB Streams + daily scan) and `wo-po-lookup` | Daily KB sync + Bedrock agent PO lookups |

The consumer imports the table via `Table.fromTableName(...)` and is granted read-only access (`grantReadData`); it does not own or define it.

**Schema-coordination rule:** Any change to the `purchase-orders` schema (partition key, item shape, attribute names, streams view type) must be coordinated with `seahaven-slack-bot`. The owner here ships the change; the consumer must be updated in lockstep so its readers do not break. Treat schema changes as a cross-repo migration, not a local edit.

**Known exception (INFRA-51):** `amazon-po-parser` currently writes directly to `purchase-orders` outside this stack (backfill/enrichment scripts). This second writer is being folded into the `po-ingest` pipeline so this stack is the sole writer; until INFRA-51 closes, coordinate any schema change with `amazon-po-parser` as well.

### `WorkOrders` and `WorkOrderComments` tables (owned here)

Both tables are **owned by this repo's `WorkorderIngestStack`** (`cdk/wo_stack.py`, `RemovalPolicy.RETAIN`):

- `WorkOrders` — PK `work_order_id` (S).
- `WorkOrderComments` — PK `work_order_id` (S), SK `comment_id` (S).

> **`comment_id` format change (issue #23).** The `WorkOrderComments` range key is now
> `work_order_id#<comment_time|nocomment>#<sha256(s3_object_key)[:12]>`
> (e.g. `11144580730#2026-04-27T23:51:48#a1b2c3d4e5f6`, or `…#nocomment#…` when the source
> email carries no comment time). Previously it was `work_order_id#<timestamp>`, where two
> emails on the same WO with an identical/absent comment time collided and overwrote each other.
> The 12-hex suffix is derived from the **S3 object key alone** — deterministic, so a Lambda
> async **retry** of the same object produces a byte-identical key (idempotent, no duplicate row),
> while two distinct emails on the same WO get distinct keys. Wall-clock `now()` is kept **out** of
> the key. Consumers that split on `#` and read index `[0]`/`[1]` are unaffected; anything that
> treated "everything after the first `#`" as a bare timestamp now also captures the hash segment.

> **Timestamp format shift.** All stored ISO timestamps (`created_at`, `updated_at`, `ingested_at`
> on WO; `processed_at`, `cancelled_at` on PO) moved from naive `datetime.utcnow().isoformat()`
> to timezone-aware `datetime.now(timezone.utc).isoformat()`, so they now carry a `+00:00` suffix
> (e.g. `2026-07-15T12:00:00+00:00`). Downstream parsers that assumed a naive/no-offset string
> must accept the offset.

Both currently use default DynamoDB encryption — they are **not** yet on the shared customer-managed CMK (`alias/seahaven-dynamodb`, INFRA-95 / M-3); that migration is tracked in INFRA-6. The `workorder-email-processor` role no longer holds a pre-emptive encrypt/decrypt grant on that CMK (removed in the 2026-06-17 security sweep — it was unused while the tables are unencrypted and extended the role's decrypt reach to the CMK protecting `purchase-orders`). Re-add the grant as part of the INFRA-6 migration, at which point `grant_read_write_data` on the then-encrypted tables propagates the needed key permissions automatically.

**Consumer (read-only) — data contract:** `seahaven-slack-bot` imports both tables via `Table.fromTableName(...)` (`grantReadData`) and reads them from two Lambdas: `workorder-sync` (daily full-table scan into the Bedrock knowledge base) and `wo-po-lookup` (the Bedrock agent's direct WO lookup action group). The bot depends on the PK/SK schema above and these attributes: on `WorkOrders` — `description`, `wo_status`, `customer`, `site_code`, `building`, `address`, `severity`, `priority`, `assigned_to`, `date_reported`, `scheduled_start`, `due_date`, `updated_at`; on `WorkOrderComments` — `created_at` (used to sort comments), `commenter`, `text`. Any change to table name, key schema, these attribute names, or the encryption configuration (e.g. the INFRA-6 CMK migration) must be coordinated with `seahaven-slack-bot` before it ships, or the Bedrock agent breaks at runtime (not at deploy — the tables are imported by name, so there is no compile-time link).

### `verified-sites` table (owned here)

Owned by this repo's `po-ingest` stack (`cdk/po_stack.py`). PK `siteCode` (S); default DynamoDB encryption (NOT the shared CMK).

**Consumer (read-only) — data contract:** `seahaven-slack-bot`'s `wo-po-lookup` Lambda imports this table via `Table.fromTableName(...)` for the Bedrock agent's `lookup_site` action. It does point lookups by `siteCode` and reads `address`, `fullAddress`, `city`, `state`, `zip`, `latitude`, `longitude`, `notes`. Coordinate any change to the table name, key schema, or these attribute names with `seahaven-slack-bot`.

> **GSI drift (INFRA-138):** the `by-state` GSI was removed here on 2026-06-03 (audit M-20, "0 reads in 30d"), but `seahaven-slack-bot` still queries `IndexName: 'by-state'` for its state-listing path, so that path fails at runtime today. Restoring the GSI or removing the consumer's state path needs to be reconciled cross-repo. This is the kind of silent owner-side lifecycle change this data-contract note exists to prevent.

## Documentation

The canonical map of Sea Haven's AWS infrastructure lives in Confluence. This project's `po-ingest` and `WorkorderIngestStack` stacks are represented there as Mermaid subgraphs.

- **[AWS Architecture Map](https://seahaven.atlassian.net/wiki/spaces/IT/pages/1540098)** (Confluence, IT space, page 1540098)

## CI/CD

GitHub Actions with reusable workflows from `Sea-Haven-Industries/.github` (all pinned to a commit SHA of `main`):
- **CI** (`ci.yaml`, PR to `main`): linting + `cdk synth` via `ci-python-sam.yaml`
- **CD** (`deploy.yaml`, push to `main`): CDK deploy via `cd-cdk.yaml` (OIDC auth), followed by the synchronous `post-deploy-script: scripts/post-deploy-smoke.sh` healthcheck gate (see [Deploy-Pipeline Guards](#deploy-pipeline-guards-phase-0)) — `cd-cdk.yaml`'s `stack-name` input only accepts one stack, so the smoke script itself enumerates both `po-email-processor` and `workorder-email-processor`
- Plus dependency review and PR labeler workflows on every PR

Branch protection on `main` — all changes through PR.

## Setup

1. Bootstrap CDK: `cdk bootstrap aws://{AccountId}/us-east-1`
2. Ensure the Bedrock inference profile `us.anthropic.claude-haiku-4-5-20251001-v1:0` is enabled in `us-east-1` (it is; the CDK grants cover cross-region routing to `us-east-2`/`us-west-2`). No API key or secret to set — the processors authenticate to Bedrock via their IAM roles.
3. Create the web UI auth-gate shared secret. This secret is **imported by name**
   (`Secret.from_secret_name_v2`), not CDK-managed, so it must exist before deploy
   or the web-ui Lambdas fail closed:
   ```bash
   aws secretsmanager create-secret --name procurement-ingest/web-ui-auth-token --secret-string "$(openssl rand -hex 32)"
   ```
4. Deploy both stacks:
   ```bash
   cd cdk
   pip install -r requirements.txt
   cdk deploy --all
   ```
5. **Post-deploy cleanup (one-time):** the retired `RETAIN`-policy secrets `po-ingest/anthropic-api-key` and `workorder-ingest/anthropic-api-key` are orphaned by this deploy, not deleted. Remove them and revoke the keys at the provider:
   ```bash
   aws secretsmanager delete-secret --secret-id po-ingest/anthropic-api-key --force-delete-without-recovery
   aws secretsmanager delete-secret --secret-id workorder-ingest/anthropic-api-key --force-delete-without-recovery
   ```
6. Dashboards: `po-web-ui` and `workorder-web-ui` have no public endpoint (the Function URLs were removed 2026-06-08, INFRA-74). Invoke them through an authenticated path that forwards the `X-Auth-Token` header, e.g. `aws lambda invoke --function-name po-web-ui /tmp/out.json`.

## Tests

Offline unit tests (no AWS, no network) run via pytest from the repo root:

```bash
pip install pytest
pytest
```

Coverage:
- `tests/` — shared handler + cross-pipeline tests: `parse_raw_email` and `pad_zip`, the PO merge-write semantics (`test_po_merge.py`, #97, moto-backed), the fail-closed sender-authentication parser (`test_ses_auth.py`, INFRA-107), and the Phase 0 CDK-bundling/handler-import AST consistency check (`test_bundle_consistency.py` — see [Deploy-Pipeline Guards](#deploy-pipeline-guards-phase-0)).
- `lambdas/wo/email_processor/tests/` — the deterministic WO parser suite: golden-file tests over 55 real scrubbed `.eml` fixtures (`test_parser.py`), fail-closed validation-gate rules and adversarial/injection cases (`test_validation_gate.py`), the issue #23 `comment_id` idempotency invariants (`test_comment_id.py`), the Bedrock-fallback dispatch/EMF-metric behavior with a mocked `invoke_model` (`test_bedrock_fallback.py`), and the Phase 0 direct-invoke healthcheck contract (`test_healthcheck.py`). Golden JSON lives under `tests/fixtures/expected/`.
- `lambdas/po/email_processor/tests/` — the deterministic PO parser suite: golden-file tests over real scrubbed `.eml` fixtures (17 single-line new-PO + 8 cancellations, exact `Decimal`-aware golden comparison via `parse_float=Decimal`), fail-closed validation-gate coverage for **every** gate reason code (fixture-driven for body-level triggers under `fixtures/adversarial/`, direct `validate()` unit tests for candidate-level mutations), real multi-line and comment/non-Coupa fallback fixtures under `fixtures/ai-fallback/`, dual line-ending (CRLF/LF) parse-identity, two-path `enrich_parsed`/`save_new_po` parity (the site-extractor stream-contract guard), fixture hygiene (`ses_auth` pass + scrub-marker leak sweep), the Bedrock-fallback dispatch/EMF-metric behavior (`test_po_bedrock_fallback.py`), and the Phase 0 direct-invoke healthcheck contract (`test_po_healthcheck.py`).

All three roots are discovered by `pytest.ini` (`testpaths`).

## Scripts

**Reprocess PO emails** (re-run parser against all emails still in S3):
```bash
python scripts/reprocess.py            # dry-run
python scripts/reprocess.py --execute  # invoke po-email-processor for each
```

**Backfill verified sites** (one-time scan of historical POs):
```bash
python scripts/backfill_sites.py
```

## Directory Structure

```
cdk/
  app.py               # Two stacks: po-ingest + WorkorderIngestStack
  po_stack.py          # Purchase order pipeline resources
  wo_stack.py          # Work order pipeline resources
lambdas/
  po/                  # PO pipeline Lambdas
    email_processor/
      handler.py         # template-first + Bedrock fallback, EMF metric, merge writes, {"healthcheck": true} early-return
      template_parser.py # pure deterministic Coupa parser + fail-closed validation gate
      tests/             # golden-file + validation-gate + fallback-dispatch + healthcheck tests + fixtures
    site_extractor/
    web_ui/
  wo/                  # WO pipeline Lambdas
    email_processor/
      handler.py         # template-first + Bedrock fallback, EMF metric, #23 comment_id, {"healthcheck": true} early-return
      template_parser.py # pure deterministic parser + fail-closed validation gate
      tests/             # golden-file + validation-gate + comment_id + fallback + healthcheck tests
    web_ui/
scripts/
  reprocess.py
  backfill_sites.py
  post-deploy-smoke.sh   # CD gate: synchronous healthcheck invoke of both processors, checks FunctionError
test_local.py          # Parse sample emails through Bedrock locally (no AWS mutation)
tests/
  requirements.txt         # Test-only deps (moto)
  conftest.py              # AWS env stubs + per-pipeline module loader
  test_pad_zip.py          # PO zip-code padding tests
  test_parse_raw_email.py  # MIME parsing tests (PO + WO handlers)
  test_po_merge.py         # PO merge-write semantics tests (#97)
  test_ses_auth.py         # Sender-authentication parser tests (INFRA-107)
  test_bundle_consistency.py # AST check: bundling command ships every handler.py sibling import
```