procurement-ingest/README.md

# Procurement Ingest

![Python](https://img.shields.io/badge/Python-3776AB?logo=python&logoColor=white)
![AWS CDK](https://img.shields.io/badge/AWS-CDK-FF9900?logo=amazonaws&logoColor=white)
![CI](https://github.com/Sea-Haven-Industries/procurement-ingest/actions/workflows/ci.yaml/badge.svg)

Unified email ingestion pipelines for Amazon procurement data. Two independent pipelines — purchase orders (Coupa) and work orders (APM/Hexagon EAM) — share a single repo and CDK app but deploy as separate CloudFormation stacks.

## Pipelines

### Purchase Orders (`po-ingest` stack)

Coupa PO emails are received at `amazon_po@int.seahaven.com`, parsed **deterministic-template-first with a Claude-Haiku-4.5-on-Bedrock fallback**, and written to the `purchase-orders` DynamoDB table.

**Flow:**
1. Coupa sends a PO email (new, revision, or cancellation).
2. SES (`INBOUND_MAIL` rule set) drops the raw MIME into `s3://po-ingest-emails-{AccountId}/inbound/`.
3. S3 `ObjectCreated` triggers the `po-email-processor` Lambda.
4. Fail-closed sender authentication (INFRA-107): the SES-stamped `Authentication-Results` header must show `dkim=pass` for `amazon.coupahost.com` (see [Sender authentication](#sender-authentication-infra-107)); otherwise the email is logged and dropped.
5. **Parse:** a pure, offline template parser (`template_parser.py`) tries the two known Coupa templates first — `coupa_new_po` (95.5% of traffic) and `coupa_cancellation` (2.9%) — behind a strict fail-closed validation gate. Only on a miss/invalid result does the Lambda fall back to the Claude-on-Bedrock AI extractor (`InvokeModel`), which extracts the identical structured-JSON contract (PO number, status, supplier, nested ship-to, line items). Numeric amounts are `Decimal` on both paths (the AI decode uses `parse_float=Decimal`; DynamoDB rejects floats). The derived classification fields (`site_code`, `trade`, `fiscal_year`) are computed by a deterministic Python classifier in the shared `enrich_parsed()` post-stage (zip padding, `ship_to_raw`/`state` promotion, metadata) which runs identically on both paths — Python-authoritative on the template path and an LLM-authoritative-with-Python-shadow bake on the AI-fallback path (see the **Derived fields** note below).
6. Merge write to DynamoDB:
   - `new_po` — merge insert. Creates the PO, or backfills data into a pre-existing `Cancelled` skeleton left by an out-of-order cancellation (preserving the `Cancelled` status). No longer silently dropped when a record already exists.
   - `revision` — field-level merge (`update_item` SETs only the fields present in the revision). A revision that omits `line_items`/`supplier` no longer deletes them. Will not un-cancel a `Cancelled` PO.
   - `cancellation` — marks the row `Cancelled` (creating a minimal skeleton if the cancellation arrives before the `new_po`).
7. DynamoDB Streams feeds the **site extractor** (`po-ingest-site-extractor`) — real-time site address extraction into the `verified-sites` table. (LedgerFlow / `seahaven-slack-bot/po-sync`, the former second stream consumer, was decommissioned 2026-07-23.)

**Deterministic template parser.** `template_parser.try_deterministic_parse()` classifies by exact subject regex, extracts the nested contract (`supplier{}`, `ship_to{}` — 8 keys, `line_items[]` — 10 keys per item), and returns a parsed result **only if** it passes a fail-closed validation gate: recursive exact key-set at every nesting level; `email_type` emitted **only** on the exact new-PO subject *and* a confirmed-safe `Status` (never defaulted — a cancellation misrouted as `new_po` would defeat the sticky-`Cancelled` guard); `po_number` shape + byte-equality with the subject, the body `PO ID`, the `Amazon Purchase Order #` heading, and the `orders/<id>` URL; duplicate-label anchor integrity (`Supplier`/`Shipping`/`Total` each appear twice, the first `Shipping` must be the literal `None` placeholder); money fidelity (every `Decimal` re-serializes byte-identically to its source token with a digit/comma border check — the thousands-separator-truncation kill switch — plus `sum(line amounts) == total`, proven against **both** `Total` blocks); bullet-metadata label discipline (closed label set, assigned by leading label, never ordinal — immune to the optional `Part Number` segment); USPS address shape on the raw pre-enrichment value; and rejection of any unparseable sentinel or residual `\r`/`\xa0` artifact. Multi-line-item (0.18%) and non-USD (0 observed) new-POs, comment emails, and anything else falls back to the AI extractor. The AI path is gated too: the untrusted email reaches Bedrock inside a neutralized `<email>` data block (forged tag lookalikes in the body are defanged) with `temperature=0`, and the raw model output must pass the fail-closed `validate_ai_fallback()` gate — the PO-specific nested contract (exact key-set at every level, with missing keys normalized rather than rejected), a `po_number` shape check hardened against fullwidth-digit and trailing-artifact injection (the same regex family protecting the DynamoDB partition key the handler builds from it), an `email_type` allow-list enforced *before* dispatch so a miss can never fall into the `new_po` default branch, and `Decimal`/`int`/`None` money typing (PO decodes with `parse_float=Decimal`) — before any DynamoDB write. **The template path is gate-enforced end-to-end; the AI-fallback path is validated and fail-closed — output that fails the gate is skipped, never written (see `ai_fallback_rejected` below), so a malformed or injected email raises the fallback rate rather than corrupting a record.** Every record emits one CloudWatch EMF metric (see below).

**Derived fields (`site_code`, `trade`, `fiscal_year`).** These three are **classified**, not verbatim-extracted, so neither parse path emits them directly — instead a deterministic Python classifier (`derived_fields.derive_all()`) computes them inside the shared `enrich_parsed()` post-stage, which runs identically on both paths. `derived_fields.py` is byte-frozen this phase (shadow-bake freeze, per the constraint-9 module-diff gate); note for accuracy that its in-file docstring's "FAITHFUL v1 port" self-description is aspirational, not descriptive — the module is a deliberate **spec-superset** of the original `EXTRACTION_PROMPT` rule text (its own inline backtest-tuning comments diverge from those three prompt sections), with the LLM staying runtime-authoritative on the AI-fallback path during the bake regardless. The docstring reword is deferred to the first post-bake PR, since even a text-only edit is barred by this phase's machine-checked empty-diff gate on the file. `trade` is a closed label set (the same one the AI prompt enumerates): `Plumbing - PM`, `Plumbing - Reactive`, `Electrical`, `HVAC`, `Dock Doors`, `Doors`, `Signage`, `Carpentry`, `Fencing/Gates`, `Conveyance/MHE`, `Painting`, `Flooring`, `Janitorial`, `Fire/Life Safety`, `Landscaping/Yard`, `Roofing`, `Security/Locksmith`, `Snow Removal`, `PO Uplift`, `General Building - Emergency`, `General Building - Handyman`, `General Building - Project`, `General Building`. Authority differs by path:

- **Template path** — the parser leaves all three `null`, so Python is authoritative: it fills them.
- **AI-fallback path** — the LLM value stays **authoritative during the bake** (Python fills only a gap the LLM left `null`, never overwrites), and Python additionally runs in **shadow mode**: `enrich_parsed()` emits one `DerivedFieldAgreement` EMF record per field comparing the two.

**`DerivedFieldAgreement` metric** — namespace `Seahaven/PoIngest`, metric name `DerivedFieldAgreement` (Unit Count, value 1), dimensioned **only** by `Field` × `Agreement` (cardinality fixed at 3 × 4). `Agreement` ∈ {`agree` (both non-null, equal after str-strip), `disagree` (both non-null, different), `llm_null_python_filled` (LLM null, Python supplied a value), `python_null` (LLM non-null, Python null)}; the record is skipped entirely when both are null. `po_number`, `PythonValue`, and `LlmValue` ride along as non-dimensioned Logs-Insights properties so a disagreement can be reviewed by example without inflating cardinality. Only the **AI-fallback** path emits these — the template path has no LLM value to shadow.

Review disagreements during the bake with this CloudWatch Logs Insights query over `/aws/lambda/po-email-processor`:

```
fields @timestamp, Field, Agreement, LlmValue, PythonValue, po_number
| filter ispresent(DerivedFieldAgreement)
| filter Agreement = "disagree"
| sort @timestamp desc
| limit 200
```

Or aggregate overall agreement per field: `| filter ispresent(DerivedFieldAgreement) | stats count(*) by Field, Agreement`.

> **Post-bake follow-up:** once agreement is acceptable, make Python authoritative on the AI-fallback path too (stop keeping the LLM value) and **drop the `site_code`/`trade`/`fiscal_year` rule sections from `EXTRACTION_PROMPT`** — the LLM stays authoritative on the fallback path only until then.

**Lambdas** (`lambdas/po/`):
| Function | Trigger | Purpose |
|---|---|---|
| `po-email-processor` | S3 ObjectCreated | Claude extraction + DynamoDB write |
| `po-ingest-site-extractor` | DynamoDB Streams | Site code/address extraction -> `verified-sites` |
| `po-web-ui` | Manual invoke (authenticated — see Setup §6) | HTML dashboard (public Function URL removed 2026-06-08, INFRA-74) |

**Tables:**
- `purchase-orders` (PK: `po_number`, Streams: NEW_IMAGE) — read by `procurement-api` (see Shared Resources; the former `seahaven-slack-bot` reader was decommissioned 2026-07-23)
- `verified-sites` (PK: `siteCode`) — ~1,100 unique Amazon facility sites (`by-state` GSI removed 2026-06-03, audit M-20)
- `pending-site-review` (PK: `po_number`) — unresolvable POs for manual Payee Central verification

### Work Orders (`WorkorderIngestStack` stack)

Amazon APM work order emails (from Hexagon EAM / HxGN SmartCloud) are received at `apm@int.seahaven.com`, parsed **deterministic-template-first with a Claude-on-Bedrock fallback**, and written to the `WorkOrders` DynamoDB table.

**Flow:**
1. Hexagon EAM sends email notifications (new assignments, comments, updates, cancellations) to `amazon@seahavenind.com`.
2. Gmail filter forwards APM emails to `apm@int.seahaven.com` (SES).
3. SES drops the raw MIME into `s3://workorder-ingest-emails-{AccountId}/inbound/`.
4. S3 triggers the `workorder-email-processor` Lambda.
5. Fail-closed sender authentication (INFRA-107): the SES-stamped `Authentication-Results` header must show `dkim=pass` for the domain that re-signs the forward (currently allowlisted as `seahaven.com` — see the validation caveat under [Sender authentication](#sender-authentication-infra-107)); otherwise the email is logged and dropped.
6. **Parse:** a pure, offline template parser (`template_parser.py`) tries the two known Hexagon templates first, behind a strict fail-closed validation gate. Only on a miss/invalid result does the Lambda fall back to the Claude-on-Bedrock AI extractor. Both paths emit the identical structured-JSON contract (work order ID, site code, severity, priority, dates, assigned technician).
7. Work order upserted to `WorkOrders`, event/comment appended to `WorkOrderComments`.

**Deterministic template parser.** ~93.6% of WO traffic is the plain-text "AMAZON UPDATE WO DETAILS \<id\>" comment template (T1) and ~6.4% is the HTML "AMAZON assign Work Order \<id\> on building \<SITE\>" assignment template (T2). `template_parser.try_deterministic_parse()` classifies by subject, extracts the contract fields, and returns a parsed result **only if** it passes a strict validation gate (exact contract-key set; `work_order_id` matches the subject and is all-digits; the T1 `Work Order:  <id>` double-space is literally present; `email_type` matches the template; `site_code` shape; per-type required fields; and a label-bleed guard so a value that over-ran into another field fails). Anything that fails — the rare update/cancellation shapes, Hexagon template drift, or an extractor exception — falls back to the AI extractor. The AI path is gated too: the untrusted email reaches Bedrock inside a neutralized `<email>` data block (tag lookalikes in the body are defanged), and the raw model output must pass the fail-closed `validate_ai_fallback()` schema/enum/date gate before any DynamoDB write — output that fails is dropped and paged (see the `ai-fallback-rejected` alarm below), a prompt-injection defence for DKIM-passing but attacker-influenced mail. **Data is never corrupted; only the fallback rate rises.** Every record emits one CloudWatch EMF metric (see below).

**Lambdas** (`lambdas/wo/`):
| Function | Trigger | Purpose |
|---|---|---|
| `workorder-email-processor` | S3 ObjectCreated | Claude extraction + DynamoDB write |
| `workorder-web-ui` | Manual invoke (authenticated — see Setup §6) | HTML dashboard (public Function URL removed 2026-06-08, INFRA-74) |
| `workorder-shoc-emitter` | DynamoDB Streams, both WO tables (**ESMs ship disabled** — see [SHOC webhook feed](#shoc-webhook-feed-workorder-shoc-emitter)) | HMAC-signed webhook push of every WO mutation to the SHOC backend |
| `workorder-shoc-hmac-rotator` | Secrets Manager rotation schedule (30 days) | Rotates the webhook HMAC signing keys (dual-key overlap) |

**Tables:**
- `WorkOrders` (PK: `work_order_id`, Streams: NEW_AND_OLD_IMAGES) — `site-code-index` and `status-index` GSIs removed 2026-06-03 (audit M-20)
- `WorkOrderComments` (PK: `work_order_id`, SK: `comment_id`, Streams: NEW_AND_OLD_IMAGES) — see the `comment_id` format note below

### SHOC webhook feed (`workorder-shoc-emitter`)

**Flow:** `WorkOrders` / `WorkOrderComments` DynamoDB Streams (`NEW_AND_OLD_IMAGES` — the OLD image is what lets the emitter detect the `wo_status → cancelled` transition) → `workorder-shoc-emitter` → HMAC-signed HTTPS POST → SHOC backend. Every WO mutation becomes one webhook event (`work_order.created` / `.updated` / `.cancelled` / `.comment_added`) within seconds of the DynamoDB commit; DynamoDB stays the source of truth.

- **Contract:** `docs/shoc-webhook-contract.md` (Rev 2026-07-23) is the producer/consumer contract SHOC builds its receiver against, and `docs/shoc-webhook-test-vectors.json` is the shared receiver-verification vector set — both sides pin their HMAC implementation against the same vectors (producer-side via the golden-vector tests over `delivery.sign_body`).
- **Ships DARK.** Both event-source mappings deploy with `enabled=False`: the full stack (Lambda, queues, alarms, secret, rotation) deploys and is testable with zero deliveries while SHOC has no receiver, so this repo's merge cadence never depends on SHOC's. Activation is a deliberate one-line `enabled=True` PR, gated on the SHOC receiver passing the shared test vectors. The ESMs start at `LATEST` — no historical flood; SHOC loads history through the `procurement-api` read API at activation instead.
- **Ordering/retry semantics.** `parallelization_factor=1`, `bisect_batch_on_error=False`, `retry_attempts=-1`, `maximum_record_age=24h`: a retryable failure (429/5xx/timeout/connection error) blocks the shard and retries from the failed record — per-work-order commit order is the guarantee, and blocking is the intended behavior when SHOC is down. `report_batch_item_failures` keeps earlier in-batch successes from being re-delivered. Records that exhaust the 24h age are parked as ESM **failure metadata** (not full records) on `workorder-shoc-emitter-failures` (`on_failure` destination); a non-retryable 4xx (a contract bug, never worth blocking the shard for 24h) parks the **full `{envelope, response_status}` payload** on `workorder-shoc-emitter-rejected` and the loop continues. Both queues: 14-day retention, SSL-enforced, alarmed (see [CloudWatch alarms](#cloudwatch-alarms)); recovery is `scripts/replay_shoc_webhooks.py` (see [Scripts](#scripts)).
- **Secret + KMS.** The HMAC signing keys live in Secrets Manager secret `workorder-ingest/shoc-webhook-hmac` (value `{"keys": [{"kid", "secret"}, ...]}`, newest first, max 2), encrypted with the dedicated CMK `workorder-ingest-shoc-webhook-kms` — deliberately **not** `alias/seahaven-dynamodb`, so the SHOC cross-account grant's decrypt reach covers exactly this one secret and nothing else. The secret's removal policy is **`DESTROY`, deliberately not `RETAIN`**: the value is machine-generated HMAC material, fully regenerable by a single rotation, so `RETAIN` buys nothing and would expose the fixed-name RETAIN-orphan deadlock (a failed create orphans an empty shell holding the global name; every later create fails `AlreadyExists`). **Accidental-deletion recovery runbook:** redeploy to recreate the secret, force a rotation (`aws secretsmanager rotate-secret --secret-id workorder-ingest/shoc-webhook-hmac`), notify the SHOC team — receivers re-fetch within their ≤5-minute cache TTL, so no coordination window is needed — then watch the `-failures` queue and replay the gap with the replay script.
- **Rotation.** `workorder-shoc-hmac-rotator` runs on a 30-day schedule: it prepends a fresh 64-hex-char key as `keys[0]` and truncates the list to 2 entries (one overlap cycle). `kid` format is `YYYY-MM-DDTHH`. The emitter always signs with `keys[0]` behind a 5-minute TTL cache; the receiver accepts any listed `kid` and re-fetches on an unknown one — there is no delivery window in which signatures can't verify.
- **Cross-account grants (exact ARN only):** `arn:aws:iam::396287094661:role/shoc-backend-dev` is granted `secretsmanager:GetSecretValue` on the secret's resource policy **and** `kms:Decrypt` on the CMK's key policy — both halves are required; either one alone fails silently at the receiver. Future staging/prod receiver roles are each a deliberate, individually-reviewed policy addition — no wildcard/prefix trust.

### Procurement API (`procurement-api` stack)

A read-only REST API (API Gateway + the `procurement-api` Lambda, `lambdas/api/`) over both pipelines' tables, plus a token-gated OpenAPI docs page. Primary consumer: the SHOC backend, for which this replaces the retired SyncController cross-account DynamoDB scan as the reconciliation/backfill path (and the initial-history load for the outbound work-order webhook, which starts at LATEST).

| Method + path | Auth | Backing read |
|---|---|---|
| `GET /work-orders` · `/purchase-orders` · `/verified-sites` | IAM SigV4 | unordered paginated Scan (`limit` 1–500, opaque `cursor` = base64 `LastEvaluatedKey`; malformed cursor → 400) |
| `GET /work-orders/{id}` · `/purchase-orders/{id}` · `/verified-sites/{siteCode}` | IAM SigV4 | GetItem (404 on miss) |
| `GET /work-orders/{id}/comments` | IAM SigV4 | Query on the partition key, paginated |
| `POST /work-orders/{id}/comments`, `PATCH /work-orders/{id}` | — | **phase-2 planned** (`x-planned` in the spec); handler answers 501 |
| `GET /docs`, `GET /openapi.json` | shared docs token (`X-Auth-Token` header or `?token=` in a browser) | Redoc reference docs (vendored offline, no CDN) with the spec inlined / the committed spec |

- **Spec is source of truth:** `lambdas/api/openapi.json` (OpenAPI 3.1). Its top-level `webhooks` section documents the outbound SHOC work-order feed, so one page describes both directions (call + be-called). `tests/test_api_spec_drift.py` pins the spec's paths to the router table, so spec and implementation cannot drift.
- **Auth:** data routes use API Gateway `AWS_IAM` (SigV4) plus a resource policy allowing exactly `arn:aws:iam::396287094661:role/shoc-backend-dev` on `GET/*`; same-account admin callers authorize via identity policy (Postman signs SigV4 natively). Docs routes are auth `NONE` at the gateway (resource-policy carve-out for exactly those two GETs) but the handler fails closed on the shared token (`lambdas/shared/web_ui_auth.py`, secret `procurement-ingest/web-ui-auth-token`) — not an unauthenticated data path (INFRA-74 posture).
- **Custom domain:** `https://procurement-api.seahaven.com` (REGIONAL API Gateway domain, TLS 1.2, empty base-path mapping to the `prod` stage, so callers hit `/work-orders` with no `/prod` segment). The stable SHOC-facing endpoint; the raw `*.execute-api.us-east-1.amazonaws.com/prod` URL still works. **Cross-account DNS:** the `seahaven.com` public zone is in the mgmt account (`328440206208`), so the ACM cert's validation record and the A-alias are added there out of band — `scripts/setup_procurement_api_domain.sh cert` issues the cert (DNS-validated against the mgmt zone) and writes its ARN to prod SSM `/procurement-api/custom-domain/certificate-arn`, which the stack reads (`value_for_string_parameter`, CFN-resolved at deploy); after `cdk deploy procurement-api`, `scripts/setup_procurement_api_domain.sh alias` adds the A-alias from the stack's `ProcurementApiAliasTarget`/`ProcurementApiAliasHostedZoneId` outputs. SigV4 is unaffected (same underlying API id + resource policy); the docs page injects whichever host served the request into `servers[0].url`.
- **KMS:** `purchase-orders` is CMK-encrypted; the imported-by-name table doesn't carry the key association, so the stack grants `kms:Decrypt`/`DescribeKey` on the CMK from SSM `/seahaven/dynamodb/cmk-arn` explicitly (the INFRA-104 failure class).
- **No access logging in v1** (keeps the `?token=` shim out of any log and avoids the account-level API Gateway CloudWatch role); rotate the docs token before ever enabling it. No CORS (server-to-server + Postman callers).
- **Alarms:** `procurement-api-errors`/`-throttles`/`-duration` (p99 ≥ 22.5 s) + gateway `procurement-api-5xx`, all ALARM-only → `site-alerts`. No 4XX alarm (401/403/404 are expected traffic).
- **Docs page:** `/docs` serves Redoc (read-only reference docs; live calls go through Postman since data routes need SigV4) as ONE token-gated response: the handler inlines the vendored `redoc.standalone.js`, the design-system fonts (`fonts.css`, data-URI `@font-face` — nothing may fetch from Google Fonts, test-pinned), and the spec into `docs.html`, with `</script>`/`</style>` breakout guards on every blob. Theme = SHOC tokens (Montserrat/DM Sans/JetBrains Mono, primary `#1c75bc`, 64px gradient topbar). The right-panel gradient and the topbar's Hide/Show-samples toggle target styled-components class names that are deterministic for the pinned Redoc bundle but change on any bump — re-derive them then (headless probe: elements whose computed background equals the `rightPanel` color); stale selectors degrade to a solid panel / inert toggle, cosmetic only. Tooling: `npm run lint:api` lints the spec against `redocly.yaml` (CI job `spec-lint`; deliberate exceptions live in `.redocly.lint-ignore.yaml`), `npm run docs:preview` renders the real handler output locally.

## Architecture

**IaC:** AWS CDK (Python), three stacks in one app, region `us-east-1`. The `cdk.Environment` is deliberately **account-agnostic** (region-only, no `account=`): the stacks deploy to whichever account the deploy credentials target (`011934824531` seahaven-prod since the 2026-07 account migration; formerly mgmt `328440206208`), and every account-derived template value — bucket names, `Lambda::Permission` source account, the site-alerts SNS action ARN, the Bedrock ARN below — renders as the CloudFormation `AWS::AccountId` pseudo-parameter rather than a literal. Pinning `account=` was evaluated in Phase 4 and rejected: it would resolve those tokens to literals, and against the deployed (account-agnostic) templates CloudFormation flags the `RemovalPolicy.RETAIN` email buckets as requiring replacement — a data-loss risk — for no functional gain.

All Lambdas: Python 3.12, ARM64, 60-day log retention.

**LLM provider — Amazon Bedrock.** Both email processors call Claude Haiku 4.5 through the Bedrock inference profile `us.anthropic.claude-haiku-4-5-20251001-v1:0` (`bedrock-runtime` `InvokeModel`), configured via the `BEDROCK_MODEL_ID` env var. There is **no Anthropic API key and no Secrets Manager secret** any more. Each processor role is granted `bedrock:InvokeModel` + `bedrock:InvokeModelWithResponseStream` on **both** the inference-profile ARN **and** the per-region foundation-model ARNs for `us-east-1` / `us-east-2` / `us-west-2` (empty-account foundation-model ARNs) — the `us.*` profile can route cross-region, so a profile-only grant would `AccessDenied` at runtime under load.

> **Retired secrets (manual cleanup outstanding):** the former secrets `po-ingest/anthropic-api-key` and `workorder-ingest/anthropic-api-key` had `RemovalPolicy.RETAIN`, so removing them from the CDK stacks **orphans** them rather than deleting them. Delete both by hand post-deploy and revoke the stored keys at the provider. The processors no longer read any `ANTHROPIC_API_KEY_SECRET_ARN` — authentication to Bedrock is via the Lambda execution-role IAM grant, so there is no provider API key or Secrets Manager fetch on the parse path.

**SES:** Both stacks add rules to the shared `INBOUND_MAIL` receipt rule set on `int.seahaven.com`.

### Shared CDK helpers (`cdk/common.py`, Phase 4)

The ~379 lines that `po_stack.py` and `wo_stack.py` both defined identically (DynamoDB alarms, the sender-auth-rejected metric filter + alarm, the standard per-Lambda alarm set, the Bedrock `InvokeModel` grant, the raw-email bucket, the async-invoke DLQ, and the template-fallback-rate math alarm) are collapsed into `cdk/common.py`.

**Logical-ID-safety rule (load-bearing).** Every helper is a **plain function** taking `(scope, id, ...)` — never a `Construct` subclass. Each stack calls a helper with its **own Stack instance as `scope`** and the **exact same literal construct id** it used inline before the extraction, so every synthesized logical ID is byte-stable. A `Construct` subclass would insert an extra tree node, reparent every child's logical ID, and CloudFormation would attempt to **replace** the `RemovalPolicy.RETAIN`-protected `purchase-orders` / `WorkOrders` / `WorkOrderComments` tables and the named S3 buckets — a data-loss event. Nothing in `common.py` subclasses `Construct`, and it imports only the CDK constructs its helpers touch (`dynamodb`, `cloudwatch`/`cw_actions`, `iam`, `logs`, `s3`, `sqs` — no `kms`, `ssm`, or Lambda event-source imports).

Extracted helpers: `add_ddb_alarms`, `add_sender_auth_rejected_alarm`, `add_standard_lambda_alarms` (bespoke `descriptions=` dict passed through verbatim per call site — no alarm text is generated or homogenized), `make_bedrock_invoke_statement`, `make_email_bucket`, `make_processor_dlq`, `make_fallback_rate_alarm`. Per-function alarm variance is preserved exactly through call-site arguments, not baked into the helpers: PO's email-processor duration alarm uses `p99`, WO's uses `p95`; `po-web-ui` gets throttles + duration only (no errors, no DLQ); `po-ingest-site-extractor` has no DLQ alarm (it's a DynamoDB-stream consumer, not async-invoked); `workorder-web-ui` gets **zero** alarms — the helper is simply never called for it, so it cannot silently add any. The two "AI-fallback-rejected" alarms (PO's 6-hour count-floor `IF(FILL(rej,0)>=1,...)`, WO's 5-minute/2-of-6 sparse idiom) are a different shape from `make_fallback_rate_alarm` and stay as distinct inline call sites in each stack rather than being forced through the shared helper.

**Bedrock ARN, now account/region-derived.** `make_bedrock_invoke_statement(scope)` builds the inference-profile ARN and the `us-east-1` foundation-model ARN from `Stack.of(scope).account` / `.region` instead of the previous hardcoded `328440206208`/`us-east-1` literals (the `us-east-2`/`us-west-2` foundation-model ARNs stay literal — they're fixed cross-region reach targets, not the stack's own region). Because the environment is account-agnostic (above), `stack.account` is the `AWS::AccountId` pseudo-parameter, so the derived ARN synthesizes as an `Fn::Sub`/`Ref` that **resolves at deploy time to the same ARN** the hardcoded literal named in-account. Versus the deployed stack this is a benign **in-place** IAM policy update (IAM policies never require replacement) — and it makes the grant account-portable instead of pinned to the management account. This IAM PolicyStatement change is the surface the mandatory GPT-4.1 cross-family review covers.

**New `CfnOutput`s.** Each stack now exports its Lambda function ARNs and the DynamoDB table names it owns/consumes, all net-new/additive (no existing output changes): `po-ingest` — `EmailProcessorFunctionArn`, `WebUiFunctionArn`, `SiteExtractorFunctionArn`, `PurchaseOrdersTableName`, `PendingSiteReviewTableName` (the existing `VerifiedSitesTableName` output is unchanged); `workorder-ingest` — `EmailProcessorFunctionArn`, `WebUiFunctionArn`, `WorkOrdersTableName`, `WorkOrderCommentsTableName`.

### Sender authentication (INFRA-107)

The `From` header and any `Authentication-Results` header inside the raw MIME are attacker-forgeable, so neither is trusted. Instead, both email processors authenticate the sender against the verdicts SES itself stamps at delivery time, failing closed. The authenticator is **single-sourced** at `lambdas/shared/ses_auth.py` (Phase 3) — previously duplicated byte-for-byte in each pipeline's `email_processor/` dir and kept in sync by a fixture-hygiene test; now one copy, so a future hardening fix to the fail-closed logic lands **once** instead of needing two identical edits. The CDK bundling `cp`s it flat beside each handler so the handlers' unchanged `from ses_auth import authenticate_inbound_email` still resolves at runtime (see [`lambdas/shared/`](#deploy-pipeline-guards-phase-0)). The gate:

1. Take **only the topmost** `Authentication-Results` header (SES prepends its trace headers; any lower copies arrived inside the message and are ignored).
2. Require its authserv-id to be `amazonses.com`.
3. Require a `dkim=pass` clause whose `header.d=`/`header.i=` domain is in the pipeline's allowlist.

The allowlist is the `ALLOWED_DKIM_DOMAINS` Lambda environment variable (comma-separated, set per stack in CDK — no code change needed to adjust):

| Pipeline | `ALLOWED_DKIM_DOMAINS` | Why |
|---|---|---|
| Work orders | `seahaven.com` | APM mail reaches `apm@int.seahaven.com` via a forward off `amazon@seahavenind.com`; the allowlist trusts the domain that **re-signs** DKIM on that forward (the original `hxgnsmartcloud.com` signature does not survive it). **Validated against live SES-stamped headers (2026-07-16)** — real APM deliveries carry `dkim=pass header.i=@seahaven.com`. Note a plain Gmail auto-forward re-signs under the *sending Workspace* domain (`seahavenind.com` / a `*.gappssmtp.com` key), **not** `seahaven.com`; only a Google Group (or Workspace routing) with "sign as `seahaven.com`" produces `dkim=pass header.i=@seahaven.com`. If the observed re-signing domain differs, update this value (do **not** widen it to a shared key like `*.gappssmtp.com`, which any Google customer's mail would pass). The `workorder-email-processor-sender-auth-rejected` alarm pages if this assumption is wrong instead of silently dropping every work order. |
| Purchase orders | `amazon.coupahost.com` | Coupa signs as `amazon.coupahost.com`. `amazonses.com` also passes but is deliberately not allowlisted — every SES customer's mail passes for it |

> **Clause-injection hardening:** SES echoes attacker-controlled SMTP-session tokens (`envelope-from`, `helo`, `header.from`) into its own `Authentication-Results` value, and an RFC 5321 quoted-local-part MAIL FROM may legally contain `;` and spaces. The parser therefore tokenises comment- and quoted-string-aware (RFC 8601 / RFC 5322): CFWS comments `(...)` are stripped and clauses are split only on semicolons **outside** a quoted string, so a `;` inside a quoted `envelope-from=` value can never be torn into a forged `dkim=pass` clause. The DKIM signer domain is read from `header.d=` when present (falling back to `header.i=`, taking the domain after the AUID's last top-level `@` so a quoted local-part cannot smuggle an allowlisted domain). Two latent comment-parsing edge cases (early comment-close, no-separator-on-strip) are tracked as hardening follow-ups — see the SES-AR-01/02 issue; neither is reachable through SES's real header encoding today.

> **Risk acceptance — forwarder-domain binding (INFRA-107, accepted 2026-07-16):** for work orders this control authenticates the domain that *re-signs* the `apm@` forward (`seahaven.com`), not the Hexagon originator (`hxgnsmartcloud.com`, whose signature does not survive the forward). Its strength therefore rests on the `apm@` Google Group's posting policy being restricted to trusted internal senders — that restriction is the **load-bearing control** and is accepted as documented risk. **If the `apm@` group is ever opened to external posting, this finding escalates to HIGH** (anyone able to post to the group could inject a forged work order) and the correct fix is to bind acceptance to the originator via DMARC alignment rather than the forwarder's re-signature. The PO pipeline is unaffected — `amazon.coupahost.com` is an external domain an attacker cannot get SES to sign.

On any failure (env var unset, header missing/unparseable, verdict fail, unaligned domain) the processor logs a structured `sender_auth_rejected` warning with the reason and S3 key, skips the email, and returns normally — rejected mail never triggers Lambda retries or DLQ messages, but the `<fn>-sender-auth-rejected` CloudWatch alarm (see [CloudWatch alarms](#cloudwatch-alarms)) pages on a rejection spike so a drift-induced outage is not silent. Unit tests live in `tests/test_ses_auth.py`.

**Failure handling (INFRA-41):** Each email-processor is async-invoked (S3 → Lambda). Both have a CDK-managed SQS dead-letter queue (`dead_letter_queue=`, 14-day retention, SSL-enforced) so a failed parse is captured rather than silently dropped after Lambda's retries.

### CloudWatch alarms

Every alarm is **ALARM-only** (no OK action), sends to the shared `site-alerts` SNS topic (imported once per stack via `Topic.from_topic_arn`), and uses `TreatMissingData.NOT_BREACHING`.

**Lambda alarms** (`AWS/Lambda`, `FunctionName` dimension):

| Alarm | Functions | Metric / config |
|---|---|---|
| `<fn>-errors` | `po-email-processor`, `po-ingest-site-extractor`, `workorder-email-processor`, `workorder-shoc-emitter`, `workorder-shoc-hmac-rotator` | `Errors` Sum, 5 min, `> 0`, eval 1 |
| `<fn>-throttles` | `po-email-processor`, `po-ingest-site-extractor`, `po-web-ui`, `workorder-email-processor`, `workorder-shoc-emitter`, `workorder-shoc-hmac-rotator` | `Throttles` Sum, 5 min, `> 0`, eval 1 |
| `<fn>-duration` | `po-email-processor`, `po-ingest-site-extractor`, `po-web-ui`, `workorder-shoc-emitter`, `workorder-shoc-hmac-rotator` (p99); `workorder-email-processor` (p95) | `Duration` percentile, 5 min, `>= 45000` ms (75% of the 60s timeout), eval 3 / datapoints 2 |
| `<fn>-sender-auth-rejected` | `po-email-processor`, `workorder-email-processor` | Log-metric-filter count (namespace `Seahaven/ProcurementIngest`, `default_value=0`) on `sender_auth_rejected` warnings, `Sum` 5 min, `>= 1`, eval 3 / datapoints 2 |

The `<fn>-sender-auth-rejected` alarm closes the silent-drop gap in INFRA-107: a rejected email returns normally (no error, no retry, no DLQ message), so without a log-metric filter a signing-domain drift or a wrong allowlist would discard 100% of legitimate mail while every other alarm stayed green. It counts `sender_auth_rejected` warnings per 5-minute period (`default_value=0` keeps the series continuous) and pages when 2 of the last 3 periods each see at least one rejection — a lone stray spoof probe to the internal ingest address self-clears, but a sustained false-reject storm pages within ~10–15 minutes even at low mail volume; the config is easy to tune in the CDK helper. (A residual gap remains for a *very* sparse total-reject outage — see the SES-AR-01/02 hardening issue.)

The `<fn>-duration` and `<fn>-throttles` alarms for `po-email-processor` and `workorder-email-processor` supersede the orphaned, CLI-created `Lambda-Duration-*` / `Lambda-Throttles-*` alarms (deleted post-deploy).

**DLQ alarms** (`AWS/SQS`): `po-email-processor-dlq-messages` and `workorder-email-processor-dlq-messages` fire when any message is visible on an email-processor DLQ (`ApproximateNumberOfMessagesVisible` Maximum, 5 min, `> 0`, eval 1) — a message there means an email was dropped after Lambda exhausted its async retries.  Recovery from a DLQ message (no console redrive) is documented in the [DLQ recovery runbook](docs/runbook-dlq-recovery.md).

**SHOC emitter alarms** (bespoke — these don't fit the standard-Lambda-alarm helper's shape): `workorder-shoc-emitter-iterator-age` (`IteratorAge` Maximum, 5 min, `>= 600000` ms, eval 3 / datapoints 2) fires when the stream lags ≥ 10 minutes — SHOC is likely down and the shard is blocking on retries, which is exactly the ordered-backpressure design working, but an operator should know. `workorder-shoc-emitter-failures-messages` and `workorder-shoc-emitter-rejected-messages` (`ApproximateNumberOfMessagesVisible` Maximum, 5 min, `> 0`, eval 1) page the replay runbook: the `-failures` queue receives ESM failure **metadata** for retry-exhausted records, the `-rejected` queue receives the **full parked payloads** of non-retryable 4xx deliveries (see the [SHOC webhook feed](#shoc-webhook-feed-workorder-shoc-emitter) section and `scripts/replay_shoc_webhooks.py`). All three: ALARM-only → `site-alerts`, `NOT_BREACHING`, per the house rules above.

**Parse-outcome metric + fallback-rate alarm (workorder-ingest):** the WO processor writes one CloudWatch **EMF** line per email to namespace `Seahaven/WorkorderIngest`, metric `ParseOutcome` (Unit Count, value 1), dimensioned by `ParseMethod` (`template` | `ai_fallback` | `ai_fallback_rejected`) and `TemplateId` (`update_plaintext` | `assign_html` | `unknown`). `ai_fallback_rejected` counts AI-fallback output that failed the fail-closed `validate_ai_fallback()` gate (schema/enum/date contract on raw Bedrock output — prompt-injection defence) and was dropped without a DynamoDB write. Non-dimension EMF properties `ReasonCode` and `work_order_id` are queryable in Logs Insights but not promoted to metrics (kept low-cardinality). EMF is used instead of `PutMetricData` so there is no extra sync call / latency / IAM grant on the async hot path (the role already has `logs:PutLogEvents`). The alarm `workorder-email-processor-template-fallback-rate` fires when the AI-fallback share of parses — rejected fallback parses included, so a drift outage whose AI output also fails the gate cannot lower the observed rate while dropping mail — exceeds **15%** sustained (a `MathExpression` with `FILL(...,0)` and a ≥10-sample volume floor over 15-minute periods, eval 3 / datapoints 2) — catching Hexagon template-drift coverage collapse while the volume floor + `FILL` prevent low-volume false pages / `INSUFFICIENT_DATA`. ALARM-only `SnsAction` to `site-alerts`, no OK action, `NOT_BREACHING`. The 15-minute period is a deliberate deviation from the 5-minute house style to accumulate a stable denominator at the low ~760/day volume. A second alarm, `workorder-email-processor-ai-fallback-rejected`, pages on the rejected series itself (≥1 rejection per 5-min period, 2 of the last 6 periods — the sender-auth-rejected sparse-arrival idiom) because a gate rejection drops mail without error/retry/DLQ and would otherwise be silent.

**WO Bedrock transport-error metric (Phase 8).** A Bedrock-side transport error (throttling, malformed response, non-JSON model text) during the AI-fallback attempt previously emitted **zero** `ParseOutcome` datapoints — the only emit sites were post-gate. `handler.py` now wraps the `extract_with_bedrock` call in a try/except that emits exactly one `ParseMethod=ai_fallback` / `ReasonCode=bedrock_error` datapoint and then re-raises (the exception still propagates into the errors alarm / DLQ path unchanged). This is an except-and-reraise, not a reorder: a gate-rejected email (Bedrock *returns* successfully, `validate_ai_fallback()` then rejects it) still emits only the single `ai_fallback_rejected` datapoint and nothing else — the except branch never fires because Bedrock did not raise — so the `workorder-email-processor-ai-fallback-rejected` "a rejected email emits nothing else" alarm contract holds with no double-count.

**Parse-outcome metric + fallback-rate alarm (po-ingest):** the PO processor emits the same EMF shape to namespace `Seahaven/PoIngest`, metric `ParseOutcome`, dimensioned by `ParseMethod` (`template` | `ai_fallback` | `ai_fallback_rejected`) and `TemplateId` (`coupa_new_po` | `coupa_cancellation` | `unknown`), with `ReasonCode` (the fail-closed gate reason) and `po_number` as Logs-Insights ride-alongs. `ai_fallback_rejected` counts AI-fallback output that failed the fail-closed `validate_ai_fallback()` gate (nested key-set contract, `po_number` shape, `email_type` allow-list, `Decimal` money typing) and was dropped without a DynamoDB write. The `ai_fallback_rejected` emission's `po_number` ride-along is clamped to 64 chars (`handler.py:118`, pinned by the PO-DC-02 regression test) — `telemetry.py`'s `DerivedFieldAgreement` `PythonValue`/`LlmValue` properties clamp the same way.

**Deliberate double-count:** unlike WO, PO emits `ParseMethod=ai_fallback` *before* the Bedrock call (so a Bedrock-side error still records the outcome) — a rejected email therefore always emits **both** an `ai_fallback` datapoint (pre-call) and an `ai_fallback_rejected` datapoint (post-gate), never just the latter. This is intentional and load-bearing, not a bug; the fallback-rate math below treats `fb` as already inclusive of every rejection.

The alarm `po-email-processor-template-fallback-rate` is **deliberately retuned for PO volume — do NOT copy the WO numbers**: at ~57 emails/day a 15-minute period holds ~0.6 emails, so the WO ≥10-sample floor would never be met and the alarm would be structurally dead. Instead: **6-hour periods** (~14.25 expected emails each), an `IF((fb+tmpl)>=8, …)` volume floor (at the floor a single fallback email is 12.5% < the threshold, so one email can never breach a datapoint; a breach needs ≥2 fallbacks in one window, or ≥3 at typical volume), threshold **>20%** (expected baseline fallback ≈1%: comments 0.55% + multi-line 0.18% + non-USD 0), **eval 4 / datapoints 2** (a 24h span — isolated noise self-clears while total template drift at 100% fallback pages within ~12h). Sparse overnight/weekend windows below the floor evaluate to 0 (non-breaching by design; accepted trade: a Friday-evening drift may not page until weekend volume accrues). The `ai_fallback_rejected` series (`rej`) is **deliberately excluded** from this expression's numerator, denominator, and volume floor: because the pre-call emit already counts every rejected email once inside `fb`, folding WO's `fb+rej` math in verbatim would double-count each rejection in both terms and inflate the observed rate toward 100% — `fb/(fb+tmpl)` alone is already exact for PO. Same idiom otherwise: ALARM-only `SnsAction` to `site-alerts`, `NOT_BREACHING`, and no element-wise `MAX` in the math expression (the post-#102 rule — the `IF` floor guarantees the non-zero denominator).

A second alarm, `po-email-processor-ai-fallback-rejected`, monitors the rejected series on its own — **retuned for ~57 emails/day, not WO's 5-minute sparse idiom** (which needs two rejections inside one 30-minute window and would be structurally dead at PO volume). It uses the same 6h/`IF`-floor/eval-4/datapoints-2 idiom as the fallback-rate alarm above, but as a plain count-floor on the rejected series itself (`IF(FILL(rej,0)>=1, …)`, no denominator so no divide guard is needed): threshold ≥1, over **6-hour periods**, **eval 4 / datapoints 2** — a lone stray rejection self-clears, while ≥2 rejections landing in ≥2 distinct 6h windows within 24h (sustained prompt-injection probing, or template drift whose AI output also fails the gate) pages within ~12–24h. ALARM-only `SnsAction` to `site-alerts`, `NOT_BREACHING`. Accepted residual: a single isolated rejected email never pages this alarm by itself — it is still visible as an `ai_fallback_rejected` datapoint and in the `ReasonCode` log line, and it has already raised the fallback-rate numerator above via its pre-call `ai_fallback` emit.

**DynamoDB alarms** (`AWS/DynamoDB`): each owned table gets `<table>-throttles` (`ThrottledRequests`) and `<table>-system-errors` (`SystemErrors`). These metrics emit only at the `TableName` + `Operation` dimension set, so each alarm is a `Sum` math expression across the operations the table uses (Get/BatchGet/Query/Scan/Put/Update/Delete/BatchWrite). Tables covered: `purchase-orders`, `verified-sites`, `pending-site-review` (po-ingest); `WorkOrders`, `WorkOrderComments` (workorder-ingest).

## Deploy-Pipeline Guards (Phase 0)

**Goal:** a broken Lambda bundle fails the deploy job, not Monday's first email.

**Healthcheck direct-invoke contract.** Both `po-email-processor` and `workorder-email-processor` recognize a top-level direct-invoke probe payload `{"healthcheck": true}`. In each `handler(event, context)`, the **very first statements** — before any S3 fetch, before `ses_auth`, before iterating `event["Records"]` — are:

```python
if isinstance(event, dict) and event.get("healthcheck") is True:
    return {"healthcheck": "ok"}
```

This placement is deliberate, not incidental: real mail always arrives as an S3 `ObjectCreated` event whose top-level keys (`Records`) AWS controls, so email content can never set a top-level `healthcheck` key — the branch creates no accept path for forged mail. It also emits **no EMF metric and no log line**, so it can never match the `sender_auth_rejected` log-metric-filter pattern that feeds the `<fn>-sender-auth-rejected` alarm (see [CloudWatch alarms](#cloudwatch-alarms)) — that alarm pages at ≥1 match in its window, so repeated healthcheck invokes across deploys (two deploys in ~30 min is routine) must never contribute to it. Unit coverage: `lambdas/po/email_processor/tests/test_po_healthcheck.py` and `lambdas/wo/email_processor/tests/test_healthcheck.py`.

**Post-deploy smoke gate.** `scripts/post-deploy-smoke.sh` is wired into the CD workflow as `cd-cdk.yaml`'s `post-deploy-script` input (see [CI/CD](#cicd)) and runs synchronously after every deploy, before the workflow is considered green. It invokes `po-email-processor`, `workorder-email-processor`, and `procurement-api` with `aws lambda invoke --invocation-type RequestResponse --payload '{"healthcheck": true}'` (region `us-east-1`) and asserts, per function:

1. The invoke response's **`FunctionError` field is absent** — this is the load-bearing check. A broken bundle (e.g. an `ImportError` at module init from a missing sibling module) still returns HTTP 200 from the Lambda Invoke API with `FunctionError=Unhandled`; a bare exit-code check on `aws lambda invoke` would false-pass on exactly the failure this gate exists to catch.
2. The returned payload is **exactly** `{"healthcheck": "ok"}`.

The script runs `set -euo pipefail` and exits non-zero on any invoke failure, any `FunctionError`, or a payload mismatch on either function, failing the deploy job.

**PO bundling: glob replaces the hand-maintained allowlist.** `cdk/po_stack.py`'s asset bundling command ships PO's Lambda source with a non-recursive glob instead of a hand-maintained list of filenames (`cp handler.py ses_auth.py template_parser.py derived_fields.py /asset-output/`). The glob is functionally identical for today's file set — non-recursive, so `tests/` and other subdirectories are still excluded — but structurally eliminates the failure mode that shipped a broken bundle twice (PR #105 omitted `template_parser.py`; PR #2 nearly omitted `derived_fields.py`): a new sibling module the handler imports now ships automatically instead of requiring someone to remember to add it to the list.

**Phase 2: widened asset root, both processors on the glob.** Both `cdk/po_stack.py` and `cdk/wo_stack.py` widen their bundled email-processor's `Code.from_asset` root from the per-pipeline dir (`../lambdas/po/email_processor`, `../lambdas/wo/email_processor`) to the shared parent, `../lambdas` — the prerequisite for the Phase 3 `lambdas/shared/` extraction, which needs a bundling root able to reach a sibling `shared/` package outside either pipeline's own dir (this move ships **zero handler code changes** — `git diff -- lambdas/` is empty for this PR). With bundling present, CDK mounts the asset root as the container's working directory, so both the `pip install -r` path and the `cp` source operand became repo-relative to `lambdas/`: `-r po/email_processor/requirements.txt` (resp. `wo/email_processor/requirements.txt`) and `cp po/email_processor/*.py /asset-output/` (resp. `cp wo/email_processor/*.py /asset-output/`). The pip `--platform manylinux2014_aarch64 --only-binary=:all:` pin — removing it once shipped x86 wheels into the ARM64 function and caused a total outage (PR #34) — is preserved byte-for-byte on both.

Both bundled `from_asset` calls also gain `exclude=['**/__pycache__/**', '**/tests/**', '**/package/**']`. This is load-bearing, not cosmetic: `Code.from_asset` does not honor `.gitignore`, and widening the root to `../lambdas` means the untracked, 44 MB `lambdas/po/email_processor/package/` dir (a stale vendored dependency tree; deletion is a separate, deliberate call — not part of this change) would otherwise be staged into the *source fingerprint* `from_asset` hashes to decide whether to re-bundle. Because CI never has that local-only directory, an un-excluded root would diverge the local vs. CI asset hash on every synth/deploy and force spurious redeploys; the `**/tests/**` and `__pycache__` excludes keep the hash stable for the same reason. Note the exclude does **not** decouple the two pipelines' asset hashes: `from_asset` hashes with its default `AssetHashType.SOURCE`, so the fingerprint is computed over *all* of `../lambdas` minus only the excluded `__pycache__`/`tests`/`package` paths — PO's and WO's first-party source (both `email_processor` trees, plus the two `web_ui`s and the `site_extractor`) therefore both feed **both** email-processors' hash. Editing any non-excluded file under `lambdas/` changes both email-processors' source fingerprint and redeploys both functions with byte-identical bundles. That coupling is an accepted cost of the shared-root design (the bundling `cp` glob still copies only each pipeline's own `*.py` into the zip); the excludes exist solely to strip local-only/irrelevant cruft that would diverge local vs. CI, not to isolate PO's tree from WO's — which `SOURCE` hashing cannot do here.

**WO bundling: glob replaces the whole-dir copy — deliberate prod-zip shrinkage.** WO's bundling command changes from a recursive `cp -r . /asset-output/` (the entire `wo/email_processor/` source dir, copied into the deployed zip) to the same scoped, non-recursive glob PO uses: `cp wo/email_processor/*.py /asset-output/`. This intentionally drops from the production zip:
- `requirements.txt` — needed only at bundle time (`pip install -r ...`), never at runtime;
- the entire `tests/` tree (`lambdas/wo/email_processor/tests/`) — real scrubbed `.eml` fixtures, golden JSON, and test modules;
- any first-party `__pycache__/*.pyc` a local `cp -r .` would have picked up (the currently-deployed zip carries none, but the exclude keeps future local builds equally clean).

This is cleanup, not a regression: none of those file classes are imported at runtime by `handler.handler`, so the acceptance bar for this change on WO is "the runtime-imported module set is unchanged, plus a post-deploy smoke pass" — not a byte-identical zip diff (that stricter bar applies to PO only, whose deployed zip was already this tight before this change). All four first-party top-level `.py` files WO's handler needs — `__init__.py`, `handler.py`, `ses_auth.py`, `template_parser.py` — are still shipped; the glob retains `__init__.py` because it is itself a top-level `.py` file, not a special case requiring a separate copy rule.

**Plain (non-bundled) `from_asset` calls gain `exclude` too.** The three non-bundled Lambda assets — `po-web-ui`, `po-ingest-site-extractor`, `workorder-web-ui` — each add `exclude=['**/__pycache__/**']`. Nothing else about these three changes: each keeps its own scoped asset path (`../lambdas/po/web_ui`, etc.) rather than widening to `../lambdas`, and none gains bundling. Without the exclude, a developer's local `__pycache__` — again invisible to `from_asset`'s `.gitignore`-blind staging — makes that function's asset hash nondeterministic across machines and forces spurious redeploys.

`tests/test_bundle_consistency.py` guards all of the above with a pure-AST check (no synth, no boto3, no handler import): it parses each handler.py's top-level first-party sibling imports, extracts the bundling `command=[...]` string from the corresponding CDK stack file, and asserts every required sibling module is guaranteed to ship. It recognizes both the scoped glob (`cp po/email_processor/*.py` / `cp wo/email_processor/*.py`, with or without a path prefix) and a whole-dir recursive copy (`cp -r . /asset-output/`) as unconditionally-safe shapes, and falls back to literal filename matching for any other (allowlist-style) shape. It pins each stack's command to the scoped-glob form specifically — a future revert to a narrowed single-file copy, a commented-out glob, or a filename allowlist missing a sibling all fail CI loudly instead of silently shipping a broken bundle. Runs in the existing pytest step, before synth.

### Phase 3: shared module extraction (`lambdas/shared/`)

Four first-party modules that were previously duplicated per pipeline (or inlined in each handler) are now **single-sourced** under `lambdas/shared/`, following the handbook's `lambdas/shared/` convention:

| Module | What it is | Imported by |
|---|---|---|
| `ses_auth.py` | fail-closed SES sender-authentication (INFRA-107) | both email processors |
| `web_ui_auth.py` | fail-closed `X-Auth-Token` gate + token cache (INFRA-74) | both web_ui handlers |
| `email_parsing.py` | `parse_raw_email` (the WO superset that returns `cc` unconditionally; PO simply ignores `cc`) | both email processors |
| `emf.py` | generic CloudWatch EMF emitter (`emit_metric`, `emit_parse_outcome`) parameterized by namespace / dimension-sets / properties | both email processors (PO also uses `emit_metric` for `DerivedFieldAgreement`) |

**Flat-landing import rule.** The shared dir has **no `__init__.py`** — the modules are consumed by bare name (`from ses_auth import ...`, `from emf import emit_parse_outcome`), exactly as when they were siblings. This works because the bundling `cp` lands them **flat in `/asset-output/`** beside `handler.py`, so at runtime each shared module sits on the function's own `sys.path` under its bare name — the handler import lines are unchanged, which is what keeps the byte-identical fail-closed `ses_auth` behavior through the move. The load-bearing `emf` dimension-set list `[["ParseMethod"], ["ParseMethod", "TemplateId"]]` is now pinned **once** in `emf.py` (one-sided dimension drift between the two pipelines becomes structurally impossible), while the deliberate per-pipeline **emission-ordering** differences stay in the handlers (PO emits `ai_fallback` *before* the Bedrock call with an intentional double-count; WO emits mutually-exclusive `ai_fallback`/`ai_fallback_rejected` after its gate).

**Bundling — email processors.** Both email-processor commands append a second glob, `cp shared/*.py /asset-output/`, after their own `cp <pipeline>/email_processor/*.py`. This ships all four shared modules flat into each email-processor zip. `web_ui_auth.py` therefore rides along into both email-processor bundles even though the email handlers never import it — a harmless, deliberate consequence of the all-of-`shared/` glob (`PO_EXPECTED_TOP_LEVEL_MODULES` and the bundle-parity expectations account for it). The base is cp-only (Phase 7 removed the pip install / `manylinux` pin from both email-processor commands, and this phase moves only pure first-party modules with no new dependencies, so it **stays** cp-only — no pip step is reintroduced).

**Bundling — web UIs.** Both `po-web-ui` and `workorder-web-ui` gain the same widened-root Docker bundling mechanism: their `Code.from_asset` root widens to `../lambdas` and their command copies the function's own dir contents plus **only** `shared/web_ui_auth.py` (`cp shared/web_ui_auth.py`, *not* `cp shared/*.py`) — the web UIs need only the auth module, and shipping the email-processor-only modules would break the "deployed set + `web_ui_auth`, nothing else" parity. The per-stack INFRA-74 comments stay in each handler (their wording is deliberately pipeline-specific and is not unified). `site_extractor`'s asset is untouched.

`tests/test_bundle_consistency.py` is updated in lockstep without losing teeth: `_first_party_sibling_imports` resolves shared-sourced imports under `lambdas/shared/`; the command extractor selects the email-processor command now that each stack has two bundled functions; `_bundling_ships_all` accumulates shipped module stems across **both** globs; `PO_EXPECTED_TOP_LEVEL_MODULES` gains the four shared modules; and a new pin + mutation test require the `cp shared/*.py` line to be actually executed (a commented-out or removed shared `cp` fails CI).

### Phase 5: handler decomposition + lazy boto3 clients

Both email-processor God-handlers are decomposed along the seams that already work into **flat sibling modules** in the same directory (bare-name imports, exactly like the `ses_auth`/`template_parser`/`derived_fields` pattern), so the Phase 0/2/3 `cp <pipeline>/email_processor/*.py` glob ships every new sibling automatically — no bundling change beyond the exact-set pin. Every move is a pure delete-here/add-there; `derived_fields.py`, both `template_parser.py`, the `validate_ai_fallback` gates, `lambdas/shared/`, and `cdk/` are byte-untouched.

**PO** (`handler.py` → 5 siblings + `prompts.py`): `handler.py` keeps the event loop, fail-closed SES auth, and `email_type` routing; `extraction.py` owns `extract_with_claude` + `_EMAIL_TAG_RE`; `enrichment.py` owns `enrich_parsed` + `pad_zip` (PO-only — WO has no enrichment stage) as a byte-identical move including the derived-field shadow block; `telemetry.py` owns the EMF `ParseMethod`/`DerivedFieldAgreement` emit wrappers; `persistence.py` owns `_write_fields`/`_merge_update`/`save_*` — with the two byte-identical `save_new_po`/`save_revision` **collapsed into one `_save_merge`** plus two thin wrappers differing only in the log verb (behavior-identical to both originals, sticky-cancel `ConditionExpression` guard intact; `save_cancellation` stays its own function). **WO** splits into ~5 concerns (no `enrichment`), keeping `validate_ai_fallback` **and** the `re.fullmatch(r"[0-9]+", work_order_id)` key guard in the handler loop AHEAD of both `save_work_order` and `save_event` (it protects the partition key and the `#`-delimited `comment_id` range-key segment), and keeps `_header_date_iso`/`comment_id` determinism together with `save_event` in `persistence.py`.

`EXTRACTION_PROMPT` (the ~181-line prompt) moves to `prompts.py` with a cross-reference header to the second authoritative copy of the trade/site/fiscal rule tables in `derived_fields.py`; `handler.py` keeps a `from prompts import EXTRACTION_PROMPT` re-export so `handler.EXTRACTION_PROMPT` still resolves for the tests that dereference it.

**Lazy cached boto3 clients.** Each I/O module initializes its client cache to `None` and populates it through a private `_get_<client>()` accessor on first call (`extraction.bedrock`, `persistence.dynamodb`, `handler.s3`); pure modules (`enrichment`, `telemetry`, `prompts`) import no boto3. The cache attribute keeps its original public name, so a test patches the same attribute — only the owning **module** moved (e.g. `setattr(persistence, "dynamodb", fake)`). Building the client at first *call* (deep inside a test) rather than at import also strengthens the moto-before-handler invariant.

**Behavior preserved (pinned by new tests).** PO emits `ParseMethod=ai_fallback` **before** the Bedrock call (a throttle that raises still leaves the pre-call datapoint), and a gate rejection is an additive second `ai_fallback_rejected` datapoint (PO's deliberate double-count); WO emits **after** its gate, mutually exclusive; the `DerivedFieldAgreement` shadow telemetry stays `ai_fallback`-only; `_save_merge` issues byte-identical `update_item` calls for both `new_po` and `revision`; and the WO key guard fires before either save.

## Security

**Web UI auth (defense-in-depth).** The `po-web-ui` / `workorder-web-ui` handlers refuse unauthenticated requests even though their public Function URLs were removed (INFRA-74). Each requires a shared secret in the `X-Auth-Token` header (or `Authorization: Bearer <token>`), compared in constant time against the configured token. The handler **fails closed** if the token is unset or unreadable (denies all). The token lives in the Secrets Manager secret `procurement-ingest/web-ui-auth-token`; only its ARN is passed to the Lambda (`WEB_UI_AUTH_TOKEN_SECRET_ARN`), and the value is fetched at runtime — never embedded in the CloudFormation template or Lambda env vars. The fetched value is cached in the warm container with a short TTL (5 min) so a rotated secret propagates without waiting for the execution environment to recycle. This is a defense-in-depth floor for a detached URL, not primary auth.

**Output escaping.** All caller-influenced values (including prompt-injectable strings Claude may return for `total_amount`/line-item amounts) are HTML-escaped before interpolation to prevent stored XSS.

## Shared Resources

### `purchase-orders` table (owned here)

The `purchase-orders` DynamoDB table is **owned by this repo's `po-ingest` stack** (defined in `cdk/po_stack.py` with `RemovalPolicy.RETAIN`, `StreamViewType.NEW_IMAGE`, and SSE-KMS encryption with the shared customer-managed CMK `alias/seahaven-dynamodb`, INFRA-95 / M-3). The `po-email-processor` Lambda is the authoritative writer — it performs the merge inserts, field-level revision merges, and cancellation updates described above.

**Consumers (read-only):**

| Consumer | How it reads | Purpose |
|---|---|---|
| `procurement-api` (this repo) | `Table.from_table_name` + `grant_read_data` | REST reads (`GET /purchase-orders*`) |

`seahaven-slack-bot`, the former external consumer, was decommissioned 2026-07-23 and its cross-account grants removed. External consumers now read through the `procurement-api` REST contract (`lambdas/api/openapi.json`), never by importing the table directly.

**Schema-coordination rule:** Any change to the `purchase-orders` schema (partition key, item shape, attribute names, streams view type) must be reflected in the API's OpenAPI schemas in the same PR (the spec-drift test pins paths, not item shapes — the schema fields are maintained by hand). Treat schema changes as a contract migration, not a local edit.

**Known exception (INFRA-51):** `amazon-po-parser` currently writes directly to `purchase-orders` outside this stack (backfill/enrichment scripts). This second writer is being folded into the `po-ingest` pipeline so this stack is the sole writer; until INFRA-51 closes, coordinate any schema change with `amazon-po-parser` as well.

### `WorkOrders` and `WorkOrderComments` tables (owned here)

Both tables are **owned by this repo's `WorkorderIngestStack`** (`cdk/wo_stack.py`, `RemovalPolicy.RETAIN`):

- `WorkOrders` — PK `work_order_id` (S).
- `WorkOrderComments` — PK `work_order_id` (S), SK `comment_id` (S).

> **`comment_id` format change (issue #23).** The `WorkOrderComments` range key is now
> `work_order_id#<comment_time|nocomment>#<sha256(s3_object_key)[:12]>`
> (e.g. `11144580730#2026-04-27T23:51:48#a1b2c3d4e5f6`, or `…#nocomment#…` when the source
> email carries no comment time). Previously it was `work_order_id#<timestamp>`, where two
> emails on the same WO with an identical/absent comment time collided and overwrote each other.
> The 12-hex suffix is derived from the **S3 object key alone** — deterministic, so a Lambda
> async **retry** of the same object produces a byte-identical key (idempotent, no duplicate row),
> while two distinct emails on the same WO get distinct keys. Wall-clock `now()` is kept **out** of
> the key. Consumers that split on `#` and read index `[0]`/`[1]` are unaffected; anything that
> treated "everything after the first `#`" as a bare timestamp now also captures the hash segment.

> **Timestamp format shift.** All stored ISO timestamps (`created_at`, `updated_at`, `ingested_at`
> on WO; `processed_at`, `cancelled_at` on PO) moved from naive `datetime.utcnow().isoformat()`
> to timezone-aware `datetime.now(timezone.utc).isoformat()`, so they now carry a `+00:00` suffix
> (e.g. `2026-07-15T12:00:00+00:00`). Downstream parsers that assumed a naive/no-offset string
> must accept the offset.

Both currently use default DynamoDB encryption — they are **not** yet on the shared customer-managed CMK (`alias/seahaven-dynamodb`, INFRA-95 / M-3); that migration is tracked in INFRA-6. The `workorder-email-processor` role no longer holds a pre-emptive encrypt/decrypt grant on that CMK (removed in the 2026-06-17 security sweep — it was unused while the tables are unencrypted and extended the role's decrypt reach to the CMK protecting `purchase-orders`). Re-add the grant as part of the INFRA-6 migration, at which point `grant_read_write_data` on the then-encrypted tables propagates the needed key permissions automatically.

**Consumers (read-only) — data contract:** `seahaven-slack-bot` (the former external reader) was decommissioned 2026-07-23; its grants are gone. Current consumers: the `procurement-api` Lambda (this repo, `Table.from_table_name` + `grant_read_data`, serving `GET /work-orders*`), and the `workorder-shoc-emitter` stream consumer (this repo) — both tables now stream `NEW_AND_OLD_IMAGES`, and the emitter is a live consumer of those streams, dark (ESMs disabled) until the SHOC activation PR. External readers (SHOC) consume through the REST contract (`lambdas/api/openapi.json`) and the webhook contract (`docs/shoc-webhook-contract.md`); both documents' field lists mirror `lambdas/wo/email_processor/persistence.py`. Any change to table name, key schema, attribute names, or encryption configuration (e.g. the INFRA-6 CMK migration) must update those two contracts in the same PR — the tables are imported by name, so there is no compile-time link and breakage surfaces at runtime.

**Stream field contract (`site_code`).** Three definitions of "is this a valid site code" have existed in this repo at once. The canonical shape is `derived_fields._STRICT_CODE_RE` (`[A-Z][A-Z0-9]{2,4}`, `fullmatch`) plus its skip-list semantics (a code-shaped token is only a real site code if it is *not* skip-listed, e.g. `LLC`/`INC`/`CORP`/`LTD`/`ATTN`), used by `derive_site_code()` (the `enrich_parsed()` classifier, see the **Derived fields** note in the Purchase Orders flow above). Honestly noted, not papered over: `lambdas/po/site_extractor/handler.py`'s own `SITE_CODE_PATTERN` (`[A-Z]{2,4}\d{1,2}`, prefix-anchored `.match`, digit-requiring) still diverges from the canonical shape as of this phase — it rejects valid all-letter codes like `KLAL` (which surface as permanent `pending-site-review` rows) and accepts overlong junk like `DLI6X`/`SNY55` that the canonical `fullmatch` would not. Reconciling `po-ingest-site-extractor`'s direct-field validation onto the canonical `derived_fields` shape is Phase 6 scope, tracked separately — this paragraph will be updated when it lands.

### `verified-sites` table (owned here)

Owned by this repo's `po-ingest` stack (`cdk/po_stack.py`). PK `siteCode` (S); default DynamoDB encryption (NOT the shared CMK).

**Consumer (read-only) — data contract:** `seahaven-slack-bot` (former reader, incl. the INFRA-138/INFRA-180 `by-state` GSI drift saga) was decommissioned 2026-07-23 — the GSI-drift issue died with it. Current consumer: the `procurement-api` Lambda (`GET /verified-sites*`). The API's `VerifiedSite` schema deliberately promises only pipeline-written fields (`siteCode`, `address`, `city`, `state`, `zip`, `fullAddress`, `locationCode`, `poCount`, `sourcePOs`) — `latitude`/`longitude`/`notes` were manually curated legacy attributes the extractor never writes, and are not part of the contract.

## Documentation

The canonical map of Sea Haven's AWS infrastructure lives in Confluence. This project's `po-ingest` and `WorkorderIngestStack` stacks are represented there as Mermaid subgraphs.

- **[AWS Architecture Map](https://seahaven.atlassian.net/wiki/spaces/IT/pages/1540098)** (Confluence, IT space, page 1540098)

## CI/CD

GitHub Actions with reusable workflows from `Sea-Haven-Industries/.github` (all pinned to a commit SHA of `main`):
- **CI** (`ci.yaml`, PR to `main`): linting + `cdk synth` via `ci-python-sam.yaml`. `cdk synth`'s Docker-bundled asset build for `po-email-processor` and `workorder-email-processor` mounts the widened `../lambdas` asset root (Phase 2, see [Deploy-Pipeline Guards](#deploy-pipeline-guards-phase-0)) as build context, not just each function's own subdirectory — the `exclude` list on both `from_asset` calls strips local-only `__pycache__`/`package/` (and `tests/`) cruft from that wider mount's source fingerprint, so CI's asset hash matches a clean local checkout, and each function's scoped `cp` glob copies only its own pipeline's `*.py` into the zip. (The exclude does not, and under `SOURCE` hashing cannot, keep the *other* pipeline's tracked source out of the fingerprint (see the PO bundling note above on `SOURCE` hashing) — but that source is identical in CI and local, so it does not cause hash divergence.)
- **CD** (`deploy.yaml`, push to `main`): CDK deploy via `cd-cdk.yaml` (OIDC auth), followed by the synchronous `post-deploy-script: scripts/post-deploy-smoke.sh` healthcheck gate (see [Deploy-Pipeline Guards](#deploy-pipeline-guards-phase-0)) — `cd-cdk.yaml`'s `stack-name` input only accepts one stack, so the smoke script itself enumerates `po-email-processor`, `workorder-email-processor`, and `procurement-api`
- Plus dependency review and PR labeler workflows on every PR

Branch protection on `main` — all changes through PR.

## Setup

**Account prerequisites** — the stacks import four dependencies by name, so all
of these must exist in the target account BEFORE the first deploy (in
seahaven-prod they are provisioned by the seahaven-org-baseline repo and the
migration Phase 0 runbook):

- SES receipt rule set `INBOUND_MAIL` (may be inactive; the stacks attach
  their receipt rules to it) + a verified `int.seahaven.com` domain identity
- SNS topic `site-alerts` (with the `alias/seahaven-alarm-topics` CMK)
- SSM param `/seahaven/dynamodb/cmk-arn` -> KMS `alias/seahaven-dynamodb`
- Secrets Manager secret `procurement-ingest/web-ui-auth-token` (step 3)
- OIDC deploy role `githubdeploy-procurement-ingest` (see `infra/deploy-role/`)
  with the `smoke-invoke-lambda` policy, or the post-deploy smoke gate fails
  closed

1. Bootstrap CDK: `cdk bootstrap aws://{AccountId}/us-east-1`
2. Ensure the Bedrock inference profile `us.anthropic.claude-haiku-4-5-20251001-v1:0` is enabled in `us-east-1` (it is; the CDK grants cover cross-region routing to `us-east-2`/`us-west-2`). No API key or secret to set — the processors authenticate to Bedrock via their IAM roles.
3. Create the web UI auth-gate shared secret. This secret is **imported by name**
   (`Secret.from_secret_name_v2`), not CDK-managed, so it must exist before deploy
   or the web-ui Lambdas fail closed:
   ```bash
   aws secretsmanager create-secret --name procurement-ingest/web-ui-auth-token --secret-string "$(openssl rand -hex 32)"
   ```
4. Deploy both stacks:
   ```bash
   cd cdk
   pip install -r requirements.txt
   cdk deploy --all
   ```
5. **Post-deploy cleanup (one-time):** the retired `RETAIN`-policy secrets `po-ingest/anthropic-api-key` and `workorder-ingest/anthropic-api-key` are orphaned by this deploy, not deleted. Remove them and revoke the keys at the provider:
   ```bash
   aws secretsmanager delete-secret --secret-id po-ingest/anthropic-api-key --force-delete-without-recovery
   aws secretsmanager delete-secret --secret-id workorder-ingest/anthropic-api-key --force-delete-without-recovery
   ```
6. Dashboards: `po-web-ui` and `workorder-web-ui` have no public endpoint (the Function URLs were removed 2026-06-08, INFRA-74). A bare `aws lambda invoke --function-name po-web-ui /tmp/out.json` with no headers in the event is guaranteed a `401` — the handler fails closed (see [Web UI auth](#security)). Fetch the token and forward it in the event's `headers`:
   ```bash
   TOKEN=$(aws secretsmanager get-secret-value --secret-id procurement-ingest/web-ui-auth-token --query SecretString --output text)
   aws lambda invoke --function-name po-web-ui --payload "{\"headers\":{\"x-auth-token\":\"$TOKEN\"}}" /tmp/out.json
   ```

## Tests

Offline unit tests (no AWS, no network) run via pytest from the repo root:

```bash
pip install -r tests/requirements.txt
pytest
```

**Test-root consolidation (Phase 8).** Both test roots are kept (`tests/` for genuinely cross-pipeline suites, plus each pipeline's own `lambdas/*/email_processor/tests/`), but loading is now single-sourced. A new repo-root `conftest.py` — not `tests/conftest.py`, which is never an ancestor of the pipeline test roots and so cannot load for a standalone `pytest lambdas/po/email_processor/tests` run — sets the dummy AWS env, imports `moto` before any handler import (registers moto's botocore stubber hook so boto3 sessions created afterwards are stubbable), and exposes the one `load_lambda_module(pipeline, name)` loader (implemented in `tests/support/loader.py`) that every handler exec in the suite goes through, including its `sys.modules` save/restore dance for the per-pipeline duplicated bare names (`template_parser`, `persistence`, etc. — `derived_fields`/`prompts`/`telemetry`/`extraction`/`enrichment` too). `tests/support/` is a small shared package (`FakeTable`, `FakeDynamoResource`, `FakeS3`, `load_email`, `load_golden`) used by both pipelines' `_po_parser_support.py`/`_wo_parser_support.py` shims — `load_golden` decodes JSON money with `parse_float=Decimal` unconditionally (load-bearing for PO's exact-money goldens; proven safe for WO, whose 55 golden fixtures contain zero float-typed JSON numbers). `test_local.py` (a root-level manual script that imported a handler at collection time, bypassing the loader gate, and globbed a nonexistent `samples/` dir) is deleted — the golden suites cover its role.

Coverage:
- `tests/` — shared handler + cross-pipeline tests: `parse_raw_email` (`test_parse_raw_email.py`), the fail-closed sender-authentication parser (`test_ses_auth.py`, INFRA-107), the Phase 0 CDK-bundling/handler-import AST consistency check (`test_bundle_consistency.py` — see [Deploy-Pipeline Guards](#deploy-pipeline-guards-phase-0)), the `scripts/reprocess.py` synthetic S3-event-shape contract (`test_reprocess_contract.py`, Phase 7), and, new in Phase 8: the handler-level SES-auth reject seam per pipeline (`test_handler_auth_seam.py` — no auth monkeypatch + an empty `ALLOWED_DKIM_DOMAINS` must yield zero Bedrock calls, zero writes, no raise), and both web_ui functions' first coverage — fail-closed auth (`test_web_ui_auth.py`) and the handlers themselves, including a 401-without-a-table-scan assertion and a hostile-field escaping regression lock (`test_web_ui_handlers.py`).
- `lambdas/wo/email_processor/tests/` — the deterministic WO parser suite: golden-file tests over 55 real scrubbed `.eml` fixtures (`test_parser.py`), fail-closed validation-gate rules and adversarial/injection cases (`test_validation_gate.py`), the issue #23 `comment_id` idempotency invariants (`test_comment_id.py`), the Bedrock-fallback dispatch/EMF-metric behavior with a mocked `invoke_model` (`test_bedrock_fallback.py`, which also carries the Phase 5 behavior pins — WO's mutually-exclusive emit and the `[0-9]+` key guard ahead of both saves), and the Phase 0 direct-invoke healthcheck contract (`test_healthcheck.py`). Golden JSON lives under `tests/fixtures/expected/`. New in Phase 8: `test_wo_bedrock_transport.py` (transport errors — throttle, missing `content`, empty content, non-JSON model text — plus the hand-computed no-double-count pin for the `handler.py` except-and-reraise metric wrap, and the multi-record failure-isolation pin) and `test_wo_merge.py` (a moto-backed mirror of `test_po_merge.py` against the `WorkOrders` table — null-status never clobbers `wo_status`, `created_at` immutable, `status`→`wo_status` mapping, `None` fields absent from `SET`, `record_type` only-when-present). After Phase 5 the suite patches accessors on the owning siblings (`extraction.bedrock`, `persistence.dynamodb`, `persistence.datetime`, `persistence.save_*`) rather than on `handler`.
- `lambdas/po/email_processor/tests/` — the deterministic PO parser suite: golden-file tests over real scrubbed `.eml` fixtures (17 single-line new-PO + 8 cancellations, exact `Decimal`-aware golden comparison via `parse_float=Decimal`), fail-closed validation-gate coverage for **every** gate reason code (fixture-driven for body-level triggers under `fixtures/adversarial/`, direct `validate()` unit tests for candidate-level mutations), real multi-line and comment/non-Coupa fallback fixtures under `fixtures/ai-fallback/`, dual line-ending (CRLF/LF) parse-identity, two-path `enrich_parsed`/`save_new_po` parity (the site-extractor stream-contract guard), fixture hygiene (`ses_auth` pass + scrub-marker leak sweep), the Bedrock-fallback dispatch/EMF-metric behavior plus the Phase 5 behavior pins (`test_po_bedrock_fallback.py` — the pre-Bedrock `ai_fallback` emit and the additive rejected double-count), the `_save_merge` collapse parity to both `save_new_po`/`save_revision` (`test_po_save_merge_parity.py`), the `ai_fallback`-only shadow telemetry after the `enrich_parsed` move (`test_po_derived_wiring.py`, which gains the PO-DC-02 64-char `po_number` clamp regression pin in Phase 8), and the Phase 0 direct-invoke healthcheck contract (`test_po_healthcheck.py`). New in Phase 8: `test_po_bedrock_transport.py` (the same four transport-error cases as WO, asserting the pre-call `ai_fallback` metric survives and no partial write occurs, plus the multi-record failure-isolation pin) and, moved here from `tests/`: `test_po_merge.py` (#97, moto-backed merge-write semantics) and `test_pad_zip.py` (zip-code padding). After Phase 5 the suite patches accessors/constants on the owning siblings (`po_extraction.bedrock`/`.BEDROCK_MODEL_ID`/`._EMAIL_TAG_RE`, `po_persistence.dynamodb`/`.PO_TABLE`, `po_enrichment.derive_all`/`.pad_zip`, `po_telemetry.DERIVED_METRIC_NAME`) rather than on `po_handler`; re-exported names it calls (`save_*`, `extract_with_claude`, `enrich_parsed`, `_emit_parse_method_metric`, `EXTRACTION_PROMPT`) stay on `po_handler`.

All three roots are discovered by `pytest.ini` (`testpaths`).

**Coverage floor (Phase 8).** `pytest.ini`'s `addopts` runs `pytest-cov` with an explicit `--cov` path per first-party package (`lambdas/po/email_processor`, `lambdas/wo/email_processor`, `lambdas/po/web_ui`, `lambdas/wo/web_ui`, `lambdas/po/site_extractor`, `lambdas/shared`) rather than relying on an `__init__.py` package marker — none of these dirs have one, and adding one would perturb the CDK bundling asset-hash fingerprint for zero runtime benefit; `pytest-cov`'s path form measures by source file regardless of package markers. `site_extractor` is deliberately included even though it measures 0% until Phase 6 lands — coverage honesty, not a silent skip. `.coveragerc` omits `*/tests/*` and `*/cdk.out/*` so the pipeline test dirs (which sit inside their own `--cov` path) don't dilute the number. `--cov-fail-under` is the new permanent CI floor (constraint 10 — never ratcheted down).

**Ruff C901/PLR floor (Phase 8).** `ruff.toml` adds `extend-select = ["C901", "PLR"]` with `max-complexity = 12`, making the tree's pre-existing `# noqa: PLR09xx` suppressions load-bearing instead of inert (no config previously enabled the rules they suppressed). `scripts/` is in the lint scope. Findings that could not be split or were out of this phase's file-ownership were resolved with a per-file `[lint.per-file-ignores]` entry carrying a written justification — most notably `derived_fields.py` (shadow-bake freeze: even an in-file `noqa` comment is a barred edit) and the other Phase 3/5/6/7 frozen modules (`enrichment.py`, WO `template_parser.py`, both `ses_auth.py`/`emf.py`, `site_extractor/handler.py`, `scripts/reprocess.py`). PO's `template_parser.py` — this phase's one in-scope split target — clears the ceiling by decomposing `extract_new_po` and `_validate_new_po_values` into per-rule helpers instead of an ignore.

## Scripts

**Reprocess emails** (re-invoke an email-processor with a synthetic S3 event). `reprocess.py` is now **pipeline-general**: `--pipeline po|wo` selects the function + raw-email bucket. **Targeted replay** (`--key` one object, `--prefix`, or `--since` a `LastModified` timestamp) is the default, preferred mode; the full-prefix sweep is demoted behind an explicit `--all`. Every mode is dry-run unless `--execute`. See the [DLQ recovery runbook](docs/runbook-dlq-recovery.md) for the targeted single-key re-invoke flow.
```bash
python scripts/reprocess.py --pipeline po --key inbound/2026/msg.eml            # targeted, dry-run
python scripts/reprocess.py --pipeline po --key inbound/2026/msg.eml --execute  # targeted, re-invoke one
python scripts/reprocess.py --pipeline wo --all --execute                       # demoted full-prefix sweep (see --all caveats)
```

**Backfill verified sites** (one-time scan of historical POs):
```bash
python scripts/backfill_sites.py
```

**Replay SHOC webhooks** (rebuild work-order webhook events from DynamoDB and re-POST them, marked `"replay": true`). `replay_shoc_webhooks.py` is the operator runbook for the `workorder-shoc-emitter-failures` / `-rejected` alarms: when deliveries were parked (SHOC down past the 24h retry window, or a contract-bug 4xx), replay re-sends the affected work orders from **current table state** — receivers dedupe on the deterministic `delivery_id`, so overlapping or repeated runs are harmless. Selection is exactly one of `--work-order-id` (repeatable, targeted) or `--since` (a full table Scan — a count banner prints per table); `--events` narrows to state rows, comments, or both. `--url` is required with no default — replay must be a deliberate act against a known receiver. Every mode is dry-run unless `--execute`.
```bash
python scripts/replay_shoc_webhooks.py --url https://... --work-order-id 11144580730            # targeted, dry-run first
python scripts/replay_shoc_webhooks.py --url https://... --work-order-id 11144580730 --execute  # then re-POST
python scripts/replay_shoc_webhooks.py --url https://... --since 2026-07-24T02:00:00Z --execute # everything touched since (full Scan)
```

## Directory Structure

```
cdk/
  app.py               # Three stacks: po-ingest + WorkorderIngestStack + procurement-api (region-only env)
  common.py            # Phase 4: shared plain-function CDK helpers (alarms, Bedrock grant,
                        # email bucket, processor DLQ, fallback-rate alarm) -- called with
                        # each stack's own scope + literal construct ids, logical-ID-safe
  po_stack.py          # Purchase order pipeline resources
  wo_stack.py          # Work order pipeline resources + the SHOC webhook feed (secret/CMK,
                        # rotator, emitter, disabled ESMs, queues, alarms)
  procurement_api_stack.py # REST API (IAM SigV4 + resource policy) over both pipelines' tables + token-gated /docs
lambdas/               # Phase 2: shared Code.from_asset("../lambdas") bundling root for
                        # BOTH po-email-processor and workorder-email-processor (and, since
                        # Phase 3, both web_ui functions) -- each email-processor command
                        # `cp`s its own po/email_processor/*.py (or wo/...) subset PLUS
                        # `cp shared/*.py`; each web_ui command `cp`s its own dir PLUS only
                        # `shared/web_ui_auth.py`; site_extractor keeps its narrower, non-
                        # bundled asset root
  shared/              # Phase 3: single-sourced first-party modules, landed FLAT (no
                        # __init__.py -- bare-name imports) into each bundle by the cp above
    ses_auth.py          # fail-closed SES sender-auth (INFRA-107) -- one copy, one fix
    web_ui_auth.py       # fail-closed X-Auth-Token gate + token cache (INFRA-74; also gates the API /docs routes)
    email_parsing.py     # parse_raw_email (WO superset; returns cc unconditionally)
    emf.py               # generic CloudWatch EMF emitter (dimension-sets pinned once)
  api/                 # procurement-api Lambda (REST reads over both pipelines' tables)
    handler.py           # healthcheck early-return + docs-token gate + route dispatch + error mapping
    router.py            # single route table (spec-drift test pins it to openapi.json)
    pagination.py        # opaque base64(LastEvaluatedKey) cursor encode/validate (hostile cursor -> 400)
    serialization.py     # Decimal-safe JSON responses
    wo_repo.py           # WorkOrders/WorkOrderComments reads (paginated Scan / PK Query)
    po_repo.py           # purchase-orders/verified-sites reads (no VendorReplies -- dead table)
    openapi.json         # OpenAPI 3.1 source of truth (paths + outbound `webhooks` section)
    docs.html            # Redoc shell, SHOC design-system theme; handler inlines bundle/fonts/spec at request time (single token-gated request, no CDN)
    redoc.standalone.js  # vendored Redoc bundle (redoc 2.5.3, MIT), inlined into /docs; read-only docs, no try-it-out
    fonts.css            # SHOC fonts (DM Sans/Montserrat/JetBrains Mono, @fontsource latin subsets) as data URIs, inlined into /docs
  po/                  # PO pipeline Lambdas
    email_processor/     # Phase 5: God-handler decomposed into flat siblings (bare-name
                         # imports; the Phase 0/2/3 `cp <pipeline>/email_processor/*.py`
                         # glob ships every new sibling automatically)
      handler.py         # event loop + fail-closed SES auth + email_type routing + {"healthcheck": true} early-return; lazy s3 accessor; re-exports EXTRACTION_PROMPT (4 tests deref handler.EXTRACTION_PROMPT)
      extraction.py      # extract_with_claude + _EMAIL_TAG_RE + EXTRACTION_PROMPT import; lazy bedrock accessor
      enrichment.py      # enrich_parsed + pad_zip (PO-only; no boto3); derived-field shadow block
      telemetry.py       # EMF ParseMethod + DerivedFieldAgreement emit wrappers (stdout EMF; no boto3)
      persistence.py     # _write_fields/_merge_update + save_cancellation; save_new_po/save_revision collapsed into one behavior-identical _save_merge (sticky-cancel guard intact); lazy dynamodb accessor
      prompts.py         # EXTRACTION_PROMPT (~181 lines); cross-ref header to derived_fields' rule tables
      template_parser.py # pure deterministic Coupa parser + fail-closed validation gate; Phase 8:
                         # extract_new_po/_validate_new_po_values split into C901/PLR-clean per-rule
                         # helpers (behavior byte-identical; see the ruff C901/PLR floor below)
      derived_fields.py  # deterministic site_code/trade/fiscal_year classifier (untouched by Phase 3/5/8 -- shadow-bake freeze; ruff per-file-ignore instead of an in-file noqa)
      tests/             # golden-file + validation-gate + fallback-dispatch + healthcheck + save-merge-parity + behavior-pin tests + fixtures
        conftest.py                  # fake_dynamo fixture, patches persistence.dynamodb
        _po_parser_support.py        # Phase 8: thin shim -- re-exports tests.support fakes/loader; PO module handles (template_parser, prompts, telemetry, extraction, enrichment, persistence)
        test_po_derived_fields.py    # derived_fields.derive_all() unit tests
        test_po_derived_wiring.py    # ai_fallback-only shadow-telemetry wiring; Phase 8 gains the PO-DC-02 64-char po_number clamp regression pin
        test_po_bedrock_transport.py # Phase 8: Bedrock transport errors (throttle/missing-content/empty-content/non-JSON) + pre-call metric survival + multi-record failure-isolation pin
        test_po_merge.py             # Phase 8: moved from tests/ -- PO merge-write semantics tests (#97), moto-backed
        test_pad_zip.py              # Phase 8: moved from tests/ -- PO zip-code padding tests
    site_extractor/
    web_ui/              # handler.py imports `from web_ui_auth import is_authenticated` (shared)
  wo/                  # WO pipeline Lambdas
    email_processor/     # Phase 5: decomposed into ~5 flat siblings (no enrichment stage);
                         # validate_ai_fallback + the [0-9]+ work_order_id key guard stay in
                         # the handler loop AHEAD of both saves
      handler.py         # event loop + fail-closed SES auth + [0-9]+ work_order_id key guard + {"healthcheck": true} early-return; lazy s3 accessor; re-exports EXTRACTION_PROMPT; Phase 8: the Bedrock call is wrapped in try/except so a transport error emits one ai_fallback/bedrock_error datapoint then re-raises (a gate rejection still emits only ai_fallback_rejected -- no double-count)
      extraction.py      # extract_with_bedrock + _EMAIL_TAG_RE + EXTRACTION_PROMPT import; lazy bedrock accessor
      telemetry.py       # emit_parse_metric EMF wrapper (stdout EMF; no boto3)
      persistence.py     # save_work_order + _header_date_iso + save_event (#23 comment_id determinism kept WITH persistence); lazy dynamodb accessor
      prompts.py         # EXTRACTION_PROMPT; cross-ref header to template_parser.CONTRACT_KEYS
      template_parser.py # pure deterministic parser + fail-closed validation gate; Phase 8: the
                         # template-path bad-status branch now returns "invalid_status" (was a
                         # copy-paste "malformed_site_code") -- matches validate_ai_fallback's code
                         # for the identical condition
      tests/             # golden-file + validation-gate + comment_id + fallback + healthcheck + behavior-pin tests
        conftest.py                  # fake_dynamo fixture, patches persistence.dynamodb (module-top `from _wo_parser_support import wo_persistence`, Phase 8)
        _wo_parser_support.py        # Phase 8: rewritten OFF the bare `import handler` strategy -- re-exports tests.support fakes/loader; WO module handles (template_parser, persistence, extraction, telemetry)
        test_wo_merge.py             # Phase 8: moto-backed WorkOrders save_work_order merge-semantics mirror of test_po_merge (null-status never clobbers wo_status, created_at immutable, status->wo_status mapping, None fields absent, record_type only-when-present)
        test_wo_bedrock_transport.py # Phase 8: Bedrock transport errors + the hand-computed emitted-series no-double-count pin for the handler.py except-and-reraise wrap + multi-record failure-isolation pin
        fixtures/
          ses-stamped/
            auth-pass-01.eml         # Phase 8: the one new fixture allowed this phase -- a synthesized Authentication-Results header block (from WO_SES_HEADER in test_ses_auth.py), not scraped mail
    web_ui/              # handler.py imports `from web_ui_auth import is_authenticated` (shared)
    shoc_emitter/        # SHOC webhook emitter (ships dark -- ESMs disabled; see the SHOC
                         # webhook feed section). Flat siblings, bare-name imports
      handler.py         # thin per-record event loop + partial-batch failure report + rejected-queue parking
      envelope.py        # pure stream-record -> envelope mapping + event classification (incl. echo guard)
      delivery.py        # TTL-cached secret fetch + HMAC signing (sign_body) + POST + response classification
    shoc_hmac_rotator/
      handler.py         # 30-day Secrets Manager rotation (single-user: createSecret/finishSecret;
                         # dual-key overlap, kid YYYY-MM-DDTHH)
scripts/
  reprocess.py
  backfill_sites.py
  replay_shoc_webhooks.py # rebuild + re-POST webhook events ("replay": true) -- the
                          # -failures/-rejected alarm runbook; dry-run unless --execute
  post-deploy-smoke.sh   # CD gate: synchronous healthcheck invoke of the three smoke-gated functions, checks FunctionError
conftest.py             # Phase 8: THE repo-root session-invariant conftest. rootdir is pinned by
                        # pytest.ini at repo root, so this loads for every pytest invocation shape --
                        # including a standalone `pytest lambdas/po/email_processor/tests` -- before
                        # any collection import. Sets the dummy AWS env, imports moto BEFORE any
                        # handler import (registers moto's botocore stubber hook so boto3 sessions
                        # created afterwards are stubbable), and re-exports
                        # `tests.support.loader.load_lambda_module`. Owns the session fixtures
                        # (po_handler, wo_handler, email_handler, ses_auth, po_persistence,
                        # po_enrichment, wo_persistence). Supersedes the old tests/conftest.py
                        # (deleted -- it was never an ancestor of the pipeline test roots, so it
                        # could not carry these invariants for a standalone pipeline-root run).
tests/
  requirements.txt          # Test-only deps (moto, pytest-cov)
  support/                   # Phase 8: shared test package (tests/ itself has no __init__.py --
                             # PEP-420 namespace resolution via the root conftest's sys.path insert)
    __init__.py                # load_lambda_module + REPO_ROOT re-export, FIXTURE_DIRS, stems/
                                # load_raw/load_email/load_golden (parse_float=Decimal, pinned safe
                                # for both pipelines), and the superset FakeTable/FakeDynamoResource/FakeS3
    loader.py                   # the ONE load_lambda_module + sys.modules save/restore dance
                                # (dependency-topological sibling tuple + web_ui_auth); every
                                # pipeline handler exec in the suite goes through this
  test_parse_raw_email.py   # MIME parsing tests (PO + WO handlers) -- stays at root, cross-pipeline
  test_ses_auth.py          # Sender-authentication parser tests (INFRA-107) -- stays at root, cross-pipeline
  test_handler_auth_seam.py # Phase 8: handler-level SES-auth reject seam, per pipeline -- no auth
                             # monkeypatch + empty ALLOWED_DKIM_DOMAINS -> zero Bedrock calls, zero
                             # writes, no raise (closes the "delete the gate line, tests still pass" hole)
  test_web_ui_auth.py       # Phase 8: shared/web_ui_auth.py -- fail-closed on unset ARN / Secrets
                             # Manager exception, TTL cache refresh, header-matrix case-insensitivity,
                             # non-ASCII-token documenting pin
  test_web_ui_handlers.py   # Phase 8: both web_ui handlers (0% coverage before this phase) -- 401
                             # without a table scan, authenticated render path, hostile-field escaping
                             # regression lock
  test_bundle_consistency.py  # AST check: bundling command ships every handler.py sibling import
  test_reprocess_contract.py  # reprocess.py synthetic S3-event-shape contract (Phase 7)
```