procurement-ingest/README.md

# Procurement Ingest

![Python](https://img.shields.io/badge/Python-3776AB?logo=python&logoColor=white)
![Terraform](https://img.shields.io/badge/Terraform-844FBA?logo=terraform&logoColor=white)
![CI](https://github.com/Sea-Haven-Industries/procurement-ingest/actions/workflows/ci.yaml/badge.svg)

Unified email ingestion pipelines for Amazon procurement data. Two independent pipelines — purchase orders (Coupa) and work orders (APM/Hexagon EAM) — share a single repo and deploy to **seahaven-prod** via HCP Terraform workspace `procurement-ingest-prod` (PLAT-86).

## Pipelines

### Purchase Orders (`po-ingest` stack)

Coupa PO emails are received at `amazon_po@int.seahaven.com`, parsed **deterministic-template-first with a Claude-Haiku-4.5-on-Bedrock fallback**, and written to the `purchase-orders` DynamoDB table.

**Flow:**
1. Coupa sends a PO email (new, revision, or cancellation).
2. SES (`INBOUND_MAIL` rule set) drops the raw MIME into `s3://po-ingest-emails-{AccountId}/inbound/`.
3. S3 `ObjectCreated` triggers the `po-email-processor` Lambda.
4. Fail-closed sender authentication (INFRA-107): the SES-stamped `Authentication-Results` header must show `dkim=pass` for `amazon.coupahost.com` (see [Sender authentication](#sender-authentication-infra-107)); otherwise the email is logged and dropped.
5. **Parse:** a pure, offline template parser (`template_parser.py`) tries the two known Coupa templates first — `coupa_new_po` (95.5% of traffic) and `coupa_cancellation` (2.9%) — behind a strict fail-closed validation gate. Only on a miss/invalid result does the Lambda fall back to the Claude-on-Bedrock AI extractor (`InvokeModel`), which extracts the identical structured-JSON contract (PO number, status, supplier, nested ship-to, line items). Numeric amounts are `Decimal` on both paths (the AI decode uses `parse_float=Decimal`; DynamoDB rejects floats). The derived classification fields (`site_code`, `trade`, `fiscal_year`) are computed by a deterministic Python classifier in the shared `enrich_parsed()` post-stage (zip padding, `ship_to_raw`/`state` promotion, metadata) which runs identically on both paths — Python-authoritative on the template path and an LLM-authoritative-with-Python-shadow bake on the AI-fallback path (see the **Derived fields** note below).
6. Merge write to DynamoDB:
   - `new_po` — merge insert. Creates the PO, or backfills data into a pre-existing `Cancelled` skeleton left by an out-of-order cancellation (preserving the `Cancelled` status). No longer silently dropped when a record already exists.
   - `revision` — field-level merge (`update_item` SETs only the fields present in the revision). A revision that omits `line_items`/`supplier` no longer deletes them. Will not un-cancel a `Cancelled` PO.
   - `cancellation` — marks the row `Cancelled` (creating a minimal skeleton if the cancellation arrives before the `new_po`).
7. DynamoDB Streams feeds the **site extractor** (`po-ingest-site-extractor`) — real-time site address extraction into the `verified-sites` table. (LedgerFlow / `seahaven-slack-bot/po-sync`, the former second stream consumer, was decommissioned 2026-07-23.)

**Deterministic template parser.** `template_parser.try_deterministic_parse()` classifies by exact subject regex, extracts the nested contract (`supplier{}`, `ship_to{}` — 8 keys, `line_items[]` — 10 keys per item), and returns a parsed result **only if** it passes a fail-closed validation gate: recursive exact key-set at every nesting level; `email_type` emitted **only** on the exact new-PO subject *and* a confirmed-safe `Status` (never defaulted — a cancellation misrouted as `new_po` would defeat the sticky-`Cancelled` guard); `po_number` shape + byte-equality with the subject, the body `PO ID`, the `Amazon Purchase Order #` heading, and the `orders/<id>` URL; duplicate-label anchor integrity (`Supplier`/`Shipping`/`Total` each appear twice, the first `Shipping` must be the literal `None` placeholder); money fidelity (every `Decimal` re-serializes byte-identically to its source token with a digit/comma border check — the thousands-separator-truncation kill switch — plus `sum(line amounts) == total`, proven against **both** `Total` blocks); bullet-metadata label discipline (closed label set, assigned by leading label, never ordinal — immune to the optional `Part Number` segment); USPS address shape on the raw pre-enrichment value; and rejection of any unparseable sentinel or residual `\r`/`\xa0` artifact. Multi-line-item (0.18%) and non-USD (0 observed) new-POs, comment emails, and anything else falls back to the AI extractor. The AI path is gated too: the untrusted email reaches Bedrock inside a neutralized `<email>` data block (forged tag lookalikes in the body are defanged) with `temperature=0`, and the raw model output must pass the fail-closed `validate_ai_fallback()` gate — the PO-specific nested contract (exact key-set at every level, with missing keys normalized rather than rejected), a `po_number` shape check hardened against fullwidth-digit and trailing-artifact injection (the same regex family protecting the DynamoDB partition key the handler builds from it), an `email_type` allow-list enforced *before* dispatch so a miss can never fall into the `new_po` default branch, and `Decimal`/`int`/`None` money typing (PO decodes with `parse_float=Decimal`) — before any DynamoDB write. **The template path is gate-enforced end-to-end; the AI-fallback path is validated and fail-closed — output that fails the gate is skipped, never written (see `ai_fallback_rejected` below), so a malformed or injected email raises the fallback rate rather than corrupting a record.** Every record emits one CloudWatch EMF metric (see below).

**Derived fields (`site_code`, `trade`, `fiscal_year`).** These three are **classified**, not verbatim-extracted, so neither parse path emits them directly — instead a deterministic Python classifier (`derived_fields.derive_all()`) computes them inside the shared `enrich_parsed()` post-stage, which runs identically on both paths. `derived_fields.py` is byte-frozen this phase (shadow-bake freeze, per the constraint-9 module-diff gate); note for accuracy that its in-file docstring's "FAITHFUL v1 port" self-description is aspirational, not descriptive — the module is a deliberate **spec-superset** of the original `EXTRACTION_PROMPT` rule text (its own inline backtest-tuning comments diverge from those three prompt sections), with the LLM staying runtime-authoritative on the AI-fallback path during the bake regardless. The docstring reword is deferred to the first post-bake PR, since even a text-only edit is barred by this phase's machine-checked empty-diff gate on the file. `trade` is a closed label set (the same one the AI prompt enumerates): `Plumbing - PM`, `Plumbing - Reactive`, `Electrical`, `HVAC`, `Dock Doors`, `Doors`, `Signage`, `Carpentry`, `Fencing/Gates`, `Conveyance/MHE`, `Painting`, `Flooring`, `Janitorial`, `Fire/Life Safety`, `Landscaping/Yard`, `Roofing`, `Security/Locksmith`, `Snow Removal`, `PO Uplift`, `General Building - Emergency`, `General Building - Handyman`, `General Building - Project`, `General Building`. Authority differs by path:

- **Template path** — the parser leaves all three `null`, so Python is authoritative: it fills them.
- **AI-fallback path** — the LLM value stays **authoritative during the bake** (Python fills only a gap the LLM left `null`, never overwrites), and Python additionally runs in **shadow mode**: `enrich_parsed()` emits one `DerivedFieldAgreement` EMF record per field comparing the two.

**`DerivedFieldAgreement` metric** — namespace `Seahaven/PoIngest`, metric name `DerivedFieldAgreement` (Unit Count, value 1), dimensioned **only** by `Field` × `Agreement` (cardinality fixed at 3 × 4). `Agreement` ∈ {`agree` (both non-null, equal after str-strip), `disagree` (both non-null, different), `llm_null_python_filled` (LLM null, Python supplied a value), `python_null` (LLM non-null, Python null)}; the record is skipped entirely when both are null. `po_number`, `PythonValue`, and `LlmValue` ride along as non-dimensioned Logs-Insights properties so a disagreement can be reviewed by example without inflating cardinality. Only the **AI-fallback** path emits these — the template path has no LLM value to shadow.

Review disagreements during the bake with this CloudWatch Logs Insights query over `/aws/lambda/po-email-processor`:

```
fields @timestamp, Field, Agreement, LlmValue, PythonValue, po_number
| filter ispresent(DerivedFieldAgreement)
| filter Agreement = "disagree"
| sort @timestamp desc
| limit 200
```

Or aggregate overall agreement per field: `| filter ispresent(DerivedFieldAgreement) | stats count(*) by Field, Agreement`.

> **Post-bake follow-up:** once agreement is acceptable, make Python authoritative on the AI-fallback path too (stop keeping the LLM value) and **drop the `site_code`/`trade`/`fiscal_year` rule sections from `EXTRACTION_PROMPT`** — the LLM stays authoritative on the fallback path only until then.

**Lambdas** (`lambdas/po/`):
| Function | Trigger | Purpose |
|---|---|---|
| `po-email-processor` | S3 ObjectCreated | Claude extraction + DynamoDB write |
| `po-ingest-site-extractor` | DynamoDB Streams | Site code/address extraction -> `verified-sites` |
| `po-web-ui` | Manual invoke (authenticated — see Setup §6) | HTML dashboard (public Function URL removed 2026-06-08, INFRA-74) |

**Tables:**
- `purchase-orders` (PK: `po_number`, Streams: NEW_IMAGE) — read by `procurement-api` (see Shared Resources; the former `seahaven-slack-bot` reader was decommissioned 2026-07-23)
- `verified-sites` (PK: `siteCode`) — ~1,100 unique Amazon facility sites (`by-state` GSI removed 2026-06-03, audit M-20)
- `pending-site-review` (PK: `po_number`) — unresolvable POs for manual Payee Central verification

### Work Orders (`WorkorderIngestStack` stack)

Amazon APM work order emails (from Hexagon EAM / HxGN SmartCloud) are received at `apm@int.seahaven.com`, parsed **deterministic-template-first with a Claude-on-Bedrock fallback**, and written to the `work-orders` DynamoDB table.

**Flow:**
1. Hexagon EAM sends email notifications (new assignments, comments, updates, cancellations) to `amazon@seahavenind.com`.
2. Gmail filter forwards APM emails to `apm@int.seahaven.com` (SES).
3. SES drops the raw MIME into `s3://workorder-ingest-emails-{AccountId}/inbound/`.
4. S3 triggers the `workorder-email-processor` Lambda.
5. Fail-closed sender authentication (INFRA-107): the SES-stamped `Authentication-Results` header must show `dkim=pass` for the domain that re-signs the forward (currently allowlisted as `seahaven.com` — see the validation caveat under [Sender authentication](#sender-authentication-infra-107)); otherwise the email is logged and dropped.
6. **Parse:** a pure, offline template parser (`template_parser.py`) tries the two known Hexagon templates first, behind a strict fail-closed validation gate. Only on a miss/invalid result does the Lambda fall back to the Claude-on-Bedrock AI extractor. Both paths emit the identical structured-JSON contract (work order ID, site code, severity, priority, dates, assigned technician).
7. Work order upserted to `work-orders`, event/comment appended to `work-order-comments`.

**Deterministic template parser.** ~93.6% of WO traffic is the plain-text "AMAZON UPDATE WO DETAILS \<id\>" comment template (T1) and ~6.4% is the HTML "AMAZON assign Work Order \<id\> on building \<SITE\>" assignment template (T2). `template_parser.try_deterministic_parse()` classifies by subject, extracts the contract fields, and returns a parsed result **only if** it passes a strict validation gate (exact contract-key set; `work_order_id` matches the subject and is all-digits; the T1 `Work Order:  <id>` double-space is literally present; `email_type` matches the template; `site_code` shape; per-type required fields; and a label-bleed guard so a value that over-ran into another field fails). Anything that fails — the rare update/cancellation shapes, Hexagon template drift, or an extractor exception — falls back to the AI extractor. The AI path is gated too: the untrusted email reaches Bedrock inside a neutralized `<email>` data block (tag lookalikes in the body are defanged), and the raw model output must pass the fail-closed `validate_ai_fallback()` schema/enum/date gate before any DynamoDB write — output that fails is dropped and paged (see the `ai-fallback-rejected` alarm below), a prompt-injection defence for DKIM-passing but attacker-influenced mail. **Data is never corrupted; only the fallback rate rises.** Every record emits one CloudWatch EMF metric (see below).

**Lambdas** (`lambdas/wo/`):
| Function | Trigger | Purpose |
|---|---|---|
| `workorder-email-processor` | S3 ObjectCreated | Claude extraction + DynamoDB write |
| `workorder-web-ui` | Manual invoke (authenticated — see Setup §6) | HTML dashboard (public Function URL removed 2026-06-08, INFRA-74) |
| `workorder-shoc-emitter` | DynamoDB Streams, both WO tables (**ESMs ship disabled** — see [SHOC webhook feed](#shoc-webhook-feed-workorder-shoc-emitter)) | HMAC-signed webhook push of every WO mutation to the SHOC backend |
| `workorder-shoc-hmac-rotator` | Secrets Manager rotation schedule (30 days) | Rotates the webhook HMAC signing keys (dual-key overlap) |

**Tables:**
- `work-orders` (PK: `work_order_id`, Streams: NEW_AND_OLD_IMAGES) — `site-code-index` and `status-index` GSIs removed 2026-06-03 (audit M-20)
- `work-order-comments` (PK: `work_order_id`, SK: `comment_id`, Streams: NEW_AND_OLD_IMAGES) — see the `comment_id` format note below

### SHOC webhook feed (`workorder-shoc-emitter`)

**Flow:** `work-orders` / `work-order-comments` DynamoDB Streams (`NEW_AND_OLD_IMAGES` — the OLD image is what lets the emitter detect the `wo_status → cancelled` transition) → `workorder-shoc-emitter` → HMAC-signed HTTPS POST → SHOC backend. Every WO mutation becomes one webhook event (`work_order.created` / `.updated` / `.cancelled` / `.comment_added`) within seconds of the DynamoDB commit; DynamoDB stays the source of truth.

- **Contract:** `docs/shoc-webhook-contract.md` (Rev 2026-07-23) is the producer/consumer contract SHOC builds its receiver against, and `docs/shoc-webhook-test-vectors.json` is the shared receiver-verification vector set — both sides pin their HMAC implementation against the same vectors (producer-side via the golden-vector tests over `delivery.sign_body`).
- **ACTIVE since 2026-07-30.** Both event-source mappings run with `enabled=True`, flipped after the SHOC receiver on `api.dev.seahaven.com` passed the shared test vectors live (valid current-kid signature accepted, duplicate `delivery_id` deduplicated, tampered/stale/unknown-kid all rejected 401). The stack originally shipped dark (`enabled=False`) so it could deploy and be tested with zero deliveries while SHOC had no receiver. The ESMs start at `LATEST` — no historical flood; SHOC backfills history through the `procurement-api` read API, not the stream.
- **Ordering/retry semantics.** `parallelization_factor=1`, `bisect_batch_on_error=False`, `retry_attempts=-1`, `maximum_record_age=24h`: a retryable failure (429/5xx/timeout/connection error) blocks the shard and retries from the failed record — per-work-order commit order is the guarantee, and blocking is the intended behavior when SHOC is down. `report_batch_item_failures` keeps earlier in-batch successes from being re-delivered. Records that exhaust the 24h age are parked as ESM **failure metadata** (not full records) on `workorder-shoc-emitter-failures` (`on_failure` destination); a non-retryable 4xx (a contract bug, never worth blocking the shard for 24h) parks the **full `{envelope, response_status}` payload** on `workorder-shoc-emitter-rejected` and the loop continues. Both queues: 14-day retention, SSL-enforced, alarmed (see [CloudWatch alarms](#cloudwatch-alarms)); recovery is `scripts/replay_shoc_webhooks.py` (see [Scripts](#scripts)).
- **Secret + KMS.** The HMAC signing keys live in Secrets Manager secret `workorder-ingest/shoc-webhook-hmac` (value `{"keys": [{"kid", "secret"}, ...]}`, newest first, max 2), encrypted with the dedicated CMK `workorder-ingest-shoc-webhook-kms` — deliberately **not** `alias/seahaven-dynamodb`, so the SHOC cross-account grant's decrypt reach covers exactly this one secret and nothing else. The secret's removal policy is **`DESTROY`, deliberately not `RETAIN`**: the value is machine-generated HMAC material, fully regenerable by a single rotation, so `RETAIN` buys nothing and would expose the fixed-name RETAIN-orphan deadlock (a failed create orphans an empty shell holding the global name; every later create fails `AlreadyExists`). **Accidental-deletion recovery runbook:** redeploy to recreate the secret, force a rotation (`aws secretsmanager rotate-secret --secret-id workorder-ingest/shoc-webhook-hmac`), notify the SHOC team — receivers re-fetch within their ≤5-minute cache TTL, so no coordination window is needed — then watch the `-failures` queue and replay the gap with the replay script.
- **Rotation.** `workorder-shoc-hmac-rotator` runs on a 30-day schedule: it prepends a fresh 64-hex-char key as `keys[0]` and truncates the list to 2 entries (one overlap cycle). `kid` format is `YYYY-MM-DDTHH`. The emitter always signs with `keys[0]` behind a 5-minute TTL cache; the receiver accepts any listed `kid` and re-fetches on an unknown one — there is no delivery window in which signatures can't verify.
- **Cross-account grants (exact ARN only):** `arn:aws:iam::396287094661:role/shoc-backend-dev` is granted `secretsmanager:GetSecretValue` on the secret's resource policy **and** `kms:Decrypt` on the CMK's key policy — both halves are required; either one alone fails silently at the receiver. Future staging/prod receiver roles are each a deliberate, individually-reviewed policy addition — no wildcard/prefix trust.

### Procurement API (`procurement-api` stack)

A read-only REST API (API Gateway + the `procurement-api` Lambda, `lambdas/api/`) over both pipelines' tables, plus a token-gated OpenAPI docs page. Primary consumer: the SHOC backend, for which this replaces the retired SyncController cross-account DynamoDB scan as the reconciliation/backfill path (and the initial-history load for the outbound work-order webhook, which starts at LATEST).

| Method + path | Auth | Backing read |
|---|---|---|
| `GET /work-orders` · `/purchase-orders` · `/verified-sites` | IAM SigV4 | unordered paginated Scan (`limit` 1–500, opaque `cursor` = base64 `LastEvaluatedKey`; malformed cursor → 400) |
| `GET /work-orders/{id}` · `/purchase-orders/{id}` · `/verified-sites/{siteCode}` | IAM SigV4 | GetItem (404 on miss) |
| `GET /work-orders/{id}/comments` | IAM SigV4 | Query on the partition key, paginated |
| `POST /work-orders/{id}/comments`, `PATCH /work-orders/{id}` | — | **phase-2 planned** (`x-planned` in the spec); handler answers 501 |
| `GET /docs`, `GET /openapi.json` | shared docs token (`X-Auth-Token` header or `?token=` in a browser) | Redoc reference docs (vendored offline, no CDN) with the spec inlined / the committed spec |

- **Spec is source of truth:** `lambdas/api/openapi.json` (OpenAPI 3.1). Its top-level `webhooks` section documents the outbound SHOC work-order feed, so one page describes both directions (call + be-called). `tests/test_api_spec_drift.py` pins the spec's paths to the router table, so spec and implementation cannot drift.
- **Auth:** data routes use API Gateway `AWS_IAM` (SigV4) plus a resource policy allowing exactly `arn:aws:iam::396287094661:role/shoc-backend-dev` on `GET/*`; same-account admin callers authorize via identity policy (Postman signs SigV4 natively). Docs routes are auth `NONE` at the gateway (resource-policy carve-out for exactly those two GETs) but the handler fails closed on the shared token (`lambdas/shared/web_ui_auth.py`, secret `procurement-ingest/web-ui-auth-token`) — not an unauthenticated data path (INFRA-74 posture).
- **Custom domain:** `https://procurement-api.seahaven.com` (REGIONAL API Gateway domain, TLS 1.2, empty base-path mapping to the `prod` stage, so callers hit `/work-orders` with no `/prod` segment). The stable SHOC-facing endpoint; the raw `*.execute-api.us-east-1.amazonaws.com/prod` URL still works. **Cross-account DNS:** the `seahaven.com` public zone is in the mgmt account (`328440206208`), so the ACM cert's validation record and the A-alias are added there out of band — `scripts/setup_procurement_api_domain.sh cert` issues the cert (DNS-validated against the mgmt zone) and writes its ARN to prod SSM `/procurement-api/custom-domain/certificate-arn`, which Terraform reads (`data.aws_ssm_parameter` in `terraform/api.tf`); after an HCP apply that owns the API Gateway DomainName, `scripts/setup_procurement_api_domain.sh alias` adds the A-alias from API Gateway `get-domain-name` (`regionalDomainName` / `regionalHostedZoneId`). SigV4 is unaffected (same underlying API id + resource policy); the docs page injects whichever host served the request into `servers[0].url`.
- **KMS:** `purchase-orders` is CMK-encrypted; the imported-by-name table doesn't carry the key association, so the stack grants `kms:Decrypt`/`DescribeKey` on the CMK from SSM `/seahaven/dynamodb/cmk-arn` explicitly (the INFRA-104 failure class).
- **No access logging in v1** (keeps the `?token=` shim out of any log and avoids the account-level API Gateway CloudWatch role); rotate the docs token before ever enabling it. No CORS (server-to-server + Postman callers).
- **Alarms:** `procurement-api-errors`/`-throttles`/`-duration` (p99 ≥ 22.5 s) + gateway `procurement-api-5xx`, all ALARM-only → `site-alerts`. No 4XX alarm (401/403/404 are expected traffic).
- **Docs page:** `/docs` serves Redoc (read-only reference docs; live calls go through Postman since data routes need SigV4) as ONE token-gated response: the handler inlines the vendored `redoc.standalone.js`, the design-system fonts (`fonts.css`, data-URI `@font-face` — nothing may fetch from Google Fonts, test-pinned), and the spec into `docs.html`, with `</script>`/`</style>` breakout guards on every blob. Theme = SHOC tokens (Montserrat/DM Sans/JetBrains Mono, primary `#1c75bc`, 64px gradient topbar). The right-panel gradient and the topbar's Hide/Show-samples toggle target styled-components class names that are deterministic for the pinned Redoc bundle but change on any bump — re-derive them then (headless probe: elements whose computed background equals the `rightPanel` color); stale selectors degrade to a solid panel / inert toggle, cosmetic only. Tooling: `npm run lint:api` lints the spec against `redocly.yaml` (CI job `spec-lint`; deliberate exceptions live in `.redocly.lint-ignore.yaml`), `npm run docs:preview` renders the real handler output locally.

## Architecture

**IaC:** Terraform under `terraform/` (HCP remote apply, Manual until sealed). Workspace `procurement-ingest-prod` in project `seahaven-prod` is the sole mutate path for live resources in seahaven-prod `011934824531` / `us-east-1` (PLAT-86 import-in-place). Former CDK stacks (`po-ingest`, `WorkorderIngestStack`, `procurement-api`) are historical and removed from the tree (PLAT-89). Never apply against mgmt `328440206208` — PLAT-67 left cold-archive RETAIN leftovers there only.

All Lambdas: Python 3.12, ARM64, 60-day log retention.

**LLM provider — Amazon Bedrock.** Both email processors call Claude Haiku 4.5 through the Bedrock inference profile `us.anthropic.claude-haiku-4-5-20251001-v1:0` (`bedrock-runtime` `InvokeModel`), configured via the `BEDROCK_MODEL_ID` env var. There is **no Anthropic API key and no Secrets Manager secret** any more. Each processor role is granted `bedrock:InvokeModel` + `bedrock:InvokeModelWithResponseStream` on **both** the inference-profile ARN **and** the per-region foundation-model ARNs for `us-east-1` / `us-east-2` / `us-west-2` (empty-account foundation-model ARNs) — the `us.*` profile can route cross-region, so a profile-only grant would `AccessDenied` at runtime under load.

> **Retired secrets (manual cleanup outstanding):** the former secrets `po-ingest/anthropic-api-key` and `workorder-ingest/anthropic-api-key` were retained when removed from IaC, so they were **orphaned** rather than deleted. Delete both by hand and revoke the stored keys at the provider. The processors no longer read any `ANTHROPIC_API_KEY_SECRET_ARN` — authentication to Bedrock is via the Lambda execution-role IAM grant, so there is no provider API key or Secrets Manager fetch on the parse path.

**SES:** Both pipelines add rules to the shared `INBOUND_MAIL` receipt rule set on `int.seahaven.com`.

### Shared alarms and packaging helpers

DynamoDB alarms, the sender-auth-rejected metric filter + alarm, the standard per-Lambda alarm set, the Bedrock `InvokeModel` grant, the raw-email bucket, the async-invoke DLQ, and the template-fallback-rate math alarm live under `terraform/` (`po_*.tf`, `wo_*.tf`, `api*.tf`). Lambda zip packaging is `terraform/build_packages.sh` (non-recursive `copy_py_dir` + `copy_shared_all` / selective `copy_src_file`), gated by `tests/test_bundle_consistency.py`.

Per-function alarm variance is preserved: PO's email-processor duration alarm uses `p99`, WO's uses `p95`; `po-web-ui` gets throttles + duration only (no errors, no DLQ); `po-ingest-site-extractor` has no DLQ alarm (it's a DynamoDB-stream consumer, not async-invoked); `workorder-web-ui` gets **zero** alarms.

### Sender authentication (INFRA-107)

The `From` header and any `Authentication-Results` header inside the raw MIME are attacker-forgeable, so neither is trusted. Instead, both email processors authenticate the sender against the verdicts SES itself stamps at delivery time, failing closed. The authenticator is **single-sourced** at `lambdas/shared/ses_auth.py` (Phase 3) — previously duplicated byte-for-byte in each pipeline's `email_processor/` dir and kept in sync by a fixture-hygiene test; now one copy, so a future hardening fix to the fail-closed logic lands **once** instead of needing two identical edits. Packaging copies it flat beside each handler (`copy_shared_all` in `terraform/build_packages.sh`) so the handlers' unchanged `from ses_auth import authenticate_inbound_email` still resolves at runtime (see [`lambdas/shared/`](#deploy-pipeline-guards-phase-0)). The gate:

1. Take **only the topmost** `Authentication-Results` header (SES prepends its trace headers; any lower copies arrived inside the message and are ignored).
2. Require its authserv-id to be `amazonses.com`.
3. Require a `dkim=pass` clause whose `header.d=`/`header.i=` domain is in the pipeline's allowlist.

The allowlist is the `ALLOWED_DKIM_DOMAINS` Lambda environment variable (comma-separated, set per pipeline in Terraform — no code change needed to adjust):

| Pipeline | `ALLOWED_DKIM_DOMAINS` | Why |
|---|---|---|
| Work orders | `seahaven.com` | APM mail reaches `apm@int.seahaven.com` via a forward off `amazon@seahavenind.com`; the allowlist trusts the domain that **re-signs** DKIM on that forward (the original `hxgnsmartcloud.com` signature does not survive it). **Validated against live SES-stamped headers (2026-07-16)** — real APM deliveries carry `dkim=pass header.i=@seahaven.com`. Note a plain Gmail auto-forward re-signs under the *sending Workspace* domain (`seahavenind.com` / a `*.gappssmtp.com` key), **not** `seahaven.com`; only a Google Group (or Workspace routing) with "sign as `seahaven.com`" produces `dkim=pass header.i=@seahaven.com`. If the observed re-signing domain differs, update this value (do **not** widen it to a shared key like `*.gappssmtp.com`, which any Google customer's mail would pass). The `workorder-email-processor-sender-auth-rejected` alarm pages if this assumption is wrong instead of silently dropping every work order. |
| Purchase orders | `amazon.coupahost.com` | Coupa signs as `amazon.coupahost.com`. `amazonses.com` also passes but is deliberately not allowlisted — every SES customer's mail passes for it |

> **Clause-injection hardening:** SES echoes attacker-controlled SMTP-session tokens (`envelope-from`, `helo`, `header.from`) into its own `Authentication-Results` value, and an RFC 5321 quoted-local-part MAIL FROM may legally contain `;` and spaces. The parser therefore tokenises comment- and quoted-string-aware (RFC 8601 / RFC 5322): CFWS comments `(...)` are stripped and clauses are split only on semicolons **outside** a quoted string, so a `;` inside a quoted `envelope-from=` value can never be torn into a forged `dkim=pass` clause. The DKIM signer domain is read from `header.d=` when present (falling back to `header.i=`, taking the domain after the AUID's last top-level `@` so a quoted local-part cannot smuggle an allowlisted domain). Two latent comment-parsing edge cases (early comment-close, no-separator-on-strip) are tracked as hardening follow-ups — see the SES-AR-01/02 issue; neither is reachable through SES's real header encoding today.

> **Risk acceptance — forwarder-domain binding (INFRA-107, accepted 2026-07-16):** for work orders this control authenticates the domain that *re-signs* the `apm@` forward (`seahaven.com`), not the Hexagon originator (`hxgnsmartcloud.com`, whose signature does not survive the forward). Its strength therefore rests on the `apm@` Google Group's posting policy being restricted to trusted internal senders — that restriction is the **load-bearing control** and is accepted as documented risk. **If the `apm@` group is ever opened to external posting, this finding escalates to HIGH** (anyone able to post to the group could inject a forged work order) and the correct fix is to bind acceptance to the originator via DMARC alignment rather than the forwarder's re-signature. The PO pipeline is unaffected — `amazon.coupahost.com` is an external domain an attacker cannot get SES to sign.

On any failure (env var unset, header missing/unparseable, verdict fail, unaligned domain) the processor logs a structured `sender_auth_rejected` warning with the reason and S3 key, skips the email, and returns normally — rejected mail never triggers Lambda retries or DLQ messages, but the `<fn>-sender-auth-rejected` CloudWatch alarm (see [CloudWatch alarms](#cloudwatch-alarms)) pages on a rejection spike so a drift-induced outage is not silent. Unit tests live in `tests/test_ses_auth.py`.

**Failure handling (INFRA-41):** Each email-processor is async-invoked (S3 → Lambda). Both have a Terraform-managed SQS dead-letter queue (14-day retention, SSL-enforced) so a failed parse is captured rather than silently dropped after Lambda's retries.

### CloudWatch alarms

Every alarm is **ALARM-only** (no OK action), sends to the shared `site-alerts` SNS topic, and uses treat-missing-data not-breaching.

**Lambda alarms** (`AWS/Lambda`, `FunctionName` dimension):

| Alarm | Functions | Metric / config |
|---|---|---|
| `<fn>-errors` | `po-email-processor`, `po-ingest-site-extractor`, `workorder-email-processor`, `workorder-shoc-emitter`, `workorder-shoc-hmac-rotator` | `Errors` Sum, 5 min, `> 0`, eval 1 |
| `<fn>-throttles` | `po-email-processor`, `po-ingest-site-extractor`, `po-web-ui`, `workorder-email-processor`, `workorder-shoc-emitter`, `workorder-shoc-hmac-rotator` | `Throttles` Sum, 5 min, `> 0`, eval 1 |
| `<fn>-duration` | `po-email-processor`, `po-ingest-site-extractor`, `po-web-ui`, `workorder-shoc-emitter`, `workorder-shoc-hmac-rotator` (p99); `workorder-email-processor` (p95) | `Duration` percentile, 5 min, `>= 45000` ms (75% of the 60s timeout), eval 3 / datapoints 2 |
| `<fn>-sender-auth-rejected` | `po-email-processor`, `workorder-email-processor` | Log-metric-filter count (namespace `Seahaven/ProcurementIngest`, `default_value=0`) on `sender_auth_rejected` warnings, `Sum` 5 min, `>= 1`, eval 3 / datapoints 2 |

The `<fn>-sender-auth-rejected` alarm closes the silent-drop gap in INFRA-107: a rejected email returns normally (no error, no retry, no DLQ message), so without a log-metric filter a signing-domain drift or a wrong allowlist would discard 100% of legitimate mail while every other alarm stayed green. It counts `sender_auth_rejected` warnings per 5-minute period (`default_value=0` keeps the series continuous) and pages when 2 of the last 3 periods each see at least one rejection — a lone stray spoof probe to the internal ingest address self-clears, but a sustained false-reject storm pages within ~10–15 minutes even at low mail volume; the config is easy to tune in Terraform. (A residual gap remains for a *very* sparse total-reject outage — see the SES-AR-01/02 hardening issue.)

The `<fn>-duration` and `<fn>-throttles` alarms for `po-email-processor` and `workorder-email-processor` supersede the orphaned, CLI-created `Lambda-Duration-*` / `Lambda-Throttles-*` alarms (deleted post-deploy).

**DLQ alarms** (`AWS/SQS`): `po-email-processor-dlq-messages` and `workorder-email-processor-dlq-messages` fire when any message is visible on an email-processor DLQ (`ApproximateNumberOfMessagesVisible` Maximum, 5 min, `> 0`, eval 1) — a message there means an email was dropped after Lambda exhausted its async retries. WO also has `workorder-email-processor-dlq-age` (`ApproximateAgeOfOldestMessage` Maximum, 5 min, `>= 86400` s, eval 1) so an undrained breadcrumb pages again well before 14-day retention expiry. Recovery from a DLQ message (no console redrive) is documented in the [DLQ recovery runbook](docs/runbook-dlq-recovery.md).

**SHOC emitter alarms** (bespoke — these don't fit the standard-Lambda-alarm helper's shape): `workorder-shoc-emitter-iterator-age` (`IteratorAge` Maximum, 5 min, `>= 600000` ms, eval 3 / datapoints 2) fires when the stream lags ≥ 10 minutes — SHOC is likely down and the shard is blocking on retries, which is exactly the ordered-backpressure design working, but an operator should know. `workorder-shoc-emitter-failures-messages` and `workorder-shoc-emitter-rejected-messages` (`ApproximateNumberOfMessagesVisible` Maximum, 5 min, `> 0`, eval 1) page the replay runbook: the `-failures` queue receives ESM failure **metadata** for retry-exhausted records, the `-rejected` queue receives the **full parked payloads** of non-retryable 4xx deliveries (see the [SHOC webhook feed](#shoc-webhook-feed-workorder-shoc-emitter) section and `scripts/replay_shoc_webhooks.py`). All three: ALARM-only → `site-alerts`, `NOT_BREACHING`, per the house rules above.

**Parse-outcome metric + fallback-rate alarm (workorder-ingest):** the WO processor writes one CloudWatch **EMF** line per email to namespace `Seahaven/WorkorderIngest`, metric `ParseOutcome` (Unit Count, value 1), dimensioned by `ParseMethod` (`template` | `ai_fallback` | `ai_fallback_rejected`) and `TemplateId` (`update_plaintext` | `assign_html` | `unknown`). `ai_fallback_rejected` counts AI-fallback output that failed the fail-closed `validate_ai_fallback()` gate (schema/enum/date contract on raw Bedrock output — prompt-injection defence) and was dropped without a DynamoDB write. Non-dimension EMF properties `ReasonCode` and `work_order_id` are queryable in Logs Insights but not promoted to metrics (kept low-cardinality). EMF is used instead of `PutMetricData` so there is no extra sync call / latency / IAM grant on the async hot path (the role already has `logs:PutLogEvents`). The alarm `workorder-email-processor-template-fallback-rate` fires when the AI-fallback share of parses — rejected fallback parses included, so a drift outage whose AI output also fails the gate cannot lower the observed rate while dropping mail — exceeds **15%** sustained (a `MathExpression` with `FILL(...,0)` and a ≥10-sample volume floor over 15-minute periods, eval 3 / datapoints 2) — catching Hexagon template-drift coverage collapse while the volume floor + `FILL` prevent low-volume false pages / `INSUFFICIENT_DATA`. ALARM-only `SnsAction` to `site-alerts`, no OK action, `NOT_BREACHING`. The 15-minute period is a deliberate deviation from the 5-minute house style to accumulate a stable denominator at the low ~760/day volume. A second alarm, `workorder-email-processor-ai-fallback-rejected`, pages on the rejected series itself (≥1 rejection per 5-min period, 2 of the last 6 periods — the sender-auth-rejected sparse-arrival idiom) because a gate rejection drops mail without error/retry/DLQ and would otherwise be silent.

**WO Bedrock transport-error metric (Phase 8).** A Bedrock-side transport error (throttling, malformed response, non-JSON model text) during the AI-fallback attempt previously emitted **zero** `ParseOutcome` datapoints — the only emit sites were post-gate. `handler.py` now wraps the `extract_with_bedrock` call in a try/except that emits exactly one `ParseMethod=ai_fallback` / `ReasonCode=bedrock_error` datapoint and then re-raises (the exception still propagates into the errors alarm / DLQ path unchanged). This is an except-and-reraise, not a reorder: a gate-rejected email (Bedrock *returns* successfully, `validate_ai_fallback()` then rejects it) still emits only the single `ai_fallback_rejected` datapoint and nothing else — the except branch never fires because Bedrock did not raise — so the `workorder-email-processor-ai-fallback-rejected` "a rejected email emits nothing else" alarm contract holds with no double-count.

**Parse-outcome metric + fallback-rate alarm (po-ingest):** the PO processor emits the same EMF shape to namespace `Seahaven/PoIngest`, metric `ParseOutcome`, dimensioned by `ParseMethod` (`template` | `ai_fallback` | `ai_fallback_rejected`) and `TemplateId` (`coupa_new_po` | `coupa_cancellation` | `unknown`), with `ReasonCode` (the fail-closed gate reason) and `po_number` as Logs-Insights ride-alongs. `ai_fallback_rejected` counts AI-fallback output that failed the fail-closed `validate_ai_fallback()` gate (nested key-set contract, `po_number` shape, `email_type` allow-list, `Decimal` money typing) and was dropped without a DynamoDB write. The `ai_fallback_rejected` emission's `po_number` ride-along is clamped to 64 chars (`handler.py:118`, pinned by the PO-DC-02 regression test) — `telemetry.py`'s `DerivedFieldAgreement` `PythonValue`/`LlmValue` properties clamp the same way.

**Deliberate double-count:** unlike WO, PO emits `ParseMethod=ai_fallback` *before* the Bedrock call (so a Bedrock-side error still records the outcome) — a rejected email therefore always emits **both** an `ai_fallback` datapoint (pre-call) and an `ai_fallback_rejected` datapoint (post-gate), never just the latter. This is intentional and load-bearing, not a bug; the fallback-rate math below treats `fb` as already inclusive of every rejection.

The alarm `po-email-processor-template-fallback-rate` is **deliberately retuned for PO volume — do NOT copy the WO numbers**: at ~57 emails/day a 15-minute period holds ~0.6 emails, so the WO ≥10-sample floor would never be met and the alarm would be structurally dead. Instead: **6-hour periods** (~14.25 expected emails each), an `IF((fb+tmpl)>=8, …)` volume floor (at the floor a single fallback email is 12.5% < the threshold, so one email can never breach a datapoint; a breach needs ≥2 fallbacks in one window, or ≥3 at typical volume), threshold **>20%** (expected baseline fallback ≈1%: comments 0.55% + multi-line 0.18% + non-USD 0), **eval 4 / datapoints 2** (a 24h span — isolated noise self-clears while total template drift at 100% fallback pages within ~12h). Sparse overnight/weekend windows below the floor evaluate to 0 (non-breaching by design; accepted trade: a Friday-evening drift may not page until weekend volume accrues). The `ai_fallback_rejected` series (`rej`) is **deliberately excluded** from this expression's numerator, denominator, and volume floor: because the pre-call emit already counts every rejected email once inside `fb`, folding WO's `fb+rej` math in verbatim would double-count each rejection in both terms and inflate the observed rate toward 100% — `fb/(fb+tmpl)` alone is already exact for PO. Same idiom otherwise: ALARM-only `SnsAction` to `site-alerts`, `NOT_BREACHING`, and no element-wise `MAX` in the math expression (the post-#102 rule — the `IF` floor guarantees the non-zero denominator).

A second alarm, `po-email-processor-ai-fallback-rejected`, monitors the rejected series on its own — **retuned for ~57 emails/day, not WO's 5-minute sparse idiom** (which needs two rejections inside one 30-minute window and would be structurally dead at PO volume). It uses the same 6h/`IF`-floor/eval-4/datapoints-2 idiom as the fallback-rate alarm above, but as a plain count-floor on the rejected series itself (`IF(FILL(rej,0)>=1, …)`, no denominator so no divide guard is needed): threshold ≥1, over **6-hour periods**, **eval 4 / datapoints 2** — a lone stray rejection self-clears, while ≥2 rejections landing in ≥2 distinct 6h windows within 24h (sustained prompt-injection probing, or template drift whose AI output also fails the gate) pages within ~12–24h. ALARM-only `SnsAction` to `site-alerts`, `NOT_BREACHING`. Accepted residual: a single isolated rejected email never pages this alarm by itself — it is still visible as an `ai_fallback_rejected` datapoint and in the `ReasonCode` log line, and it has already raised the fallback-rate numerator above via its pre-call `ai_fallback` emit.

**DynamoDB alarms** (`AWS/DynamoDB`): each owned table gets `<table>-throttles` (`ThrottledRequests`) and `<table>-system-errors` (`SystemErrors`). These metrics emit only at the `TableName` + `Operation` dimension set, so each alarm is a `Sum` math expression across the operations the table uses (Get/BatchGet/Query/Scan/Put/Update/Delete/BatchWrite). Tables covered: `purchase-orders`, `verified-sites`, `pending-site-review` (po-ingest); `work-orders`, `work-order-comments` (workorder-ingest).

## Deploy-Pipeline Guards (Phase 0)

**Goal:** a broken Lambda bundle fails the deploy job, not Monday's first email.

**Healthcheck direct-invoke contract.** Both `po-email-processor` and `workorder-email-processor` recognize a top-level direct-invoke probe payload `{"healthcheck": true}`. In each `handler(event, context)`, the **very first statements** — before any S3 fetch, before `ses_auth`, before iterating `event["Records"]` — are:

```python
if isinstance(event, dict) and event.get("healthcheck") is True:
    return {"healthcheck": "ok"}
```

This placement is deliberate, not incidental: real mail always arrives as an S3 `ObjectCreated` event whose top-level keys (`Records`) AWS controls, so email content can never set a top-level `healthcheck` key — the branch creates no accept path for forged mail. It also emits **no EMF metric and no log line**, so it can never match the `sender_auth_rejected` log-metric-filter pattern that feeds the `<fn>-sender-auth-rejected` alarm (see [CloudWatch alarms](#cloudwatch-alarms)) — that alarm pages at ≥1 match in its window, so repeated healthcheck invokes across deploys (two deploys in ~30 min is routine) must never contribute to it. Unit coverage: `lambdas/po/email_processor/tests/test_po_healthcheck.py` and `lambdas/wo/email_processor/tests/test_healthcheck.py`.

**Post-apply smoke gate.** `scripts/post-deploy-smoke.sh` runs after HCP applies (manual or gated) against seahaven-prod. It invokes `po-email-processor`, `workorder-email-processor`, and `procurement-api` with `aws lambda invoke --invocation-type RequestResponse --payload '{"healthcheck": true}'` (region `us-east-1`) and asserts, per function:

1. The invoke response's **`FunctionError` field is absent** — this is the load-bearing check. A broken bundle (e.g. an `ImportError` at module init from a missing sibling module) still returns HTTP 200 from the Lambda Invoke API with `FunctionError=Unhandled`; a bare exit-code check on `aws lambda invoke` would false-pass on exactly the failure this gate exists to catch.
2. The returned payload is **exactly** `{"healthcheck": "ok"}`.

The script runs `set -euo pipefail` and exits non-zero on any invoke failure, any `FunctionError`, or a payload mismatch, failing the apply gate.

**Lambda packaging (`terraform/build_packages.sh`).** HCP plan/apply packages each function via non-recursive `copy_py_dir` (top-level `*.py` only, so `tests/` is excluded) plus `copy_shared_all` for the email processors and selective `copy_src_file` for web UI auth / API static assets. That eliminates the failure mode that shipped a broken bundle twice (PR #105 omitted `template_parser.py`; PR #2 nearly omitted `derived_fields.py`): a new sibling module the handler imports ships automatically when added under the pipeline dir, and `tests/test_bundle_consistency.py` pins the exact-set + ships-all contracts against the packaging script (no Terraform plan required). Pip installs for third-party deps still use `--platform manylinux2014_aarch64 --only-binary=:all:` — removing that pin once shipped x86 wheels into the ARM64 function and caused a total outage (PR #34).

### Phase 3: shared module extraction (`lambdas/shared/`)

Four first-party modules that were previously duplicated per pipeline (or inlined in each handler) are now **single-sourced** under `lambdas/shared/`, following the handbook's `lambdas/shared/` convention:

| Module | What it is | Imported by |
|---|---|---|
| `ses_auth.py` | fail-closed SES sender-authentication (INFRA-107) | both email processors |
| `web_ui_auth.py` | fail-closed `X-Auth-Token` gate + token cache (INFRA-74) | both web_ui handlers |
| `email_parsing.py` | `parse_raw_email` (the WO superset that returns `cc` unconditionally; PO simply ignores `cc`) | both email processors |
| `emf.py` | generic CloudWatch EMF emitter (`emit_metric`, `emit_parse_outcome`) parameterized by namespace / dimension-sets / properties | both email processors (PO also uses `emit_metric` for `DerivedFieldAgreement`) |

**Flat-landing import rule.** The shared dir has **no `__init__.py`** — the modules are consumed by bare name (`from ses_auth import ...`, `from emf import emit_parse_outcome`), exactly as when they were siblings. This works because packaging lands them **flat in the zip root** beside `handler.py`, so at runtime each shared module sits on the function's own `sys.path` under its bare name — the handler import lines are unchanged, which is what keeps the byte-identical fail-closed `ses_auth` behavior through the move. The load-bearing `emf` dimension-set list `[["ParseMethod"], ["ParseMethod", "TemplateId"]]` is now pinned **once** in `emf.py` (one-sided dimension drift between the two pipelines becomes structurally impossible), while the deliberate per-pipeline **emission-ordering** differences stay in the handlers (PO emits `ai_fallback` *before* the Bedrock call with an intentional double-count; WO emits mutually-exclusive `ai_fallback`/`ai_fallback_rejected` after its gate).

**Packaging — email processors.** Both email-processor packages run `copy_py_dir` for the pipeline dir plus `copy_shared_all`. This ships all four shared modules flat into each email-processor zip. `web_ui_auth.py` therefore rides along into both email-processor bundles even though the email handlers never import it — a harmless, deliberate consequence of copying all of `shared/` (`PO_EXPECTED_TOP_LEVEL_MODULES` and the bundle-parity expectations account for it).

**Packaging — web UIs.** Both `po-web-ui` and `workorder-web-ui` copy their own dir plus **only** `shared/web_ui_auth.py` (not all of `shared/`) — the web UIs need only the auth module. `site_extractor` packages its own dir only.

`tests/test_bundle_consistency.py` is updated in lockstep without losing teeth: `_first_party_sibling_imports` resolves shared-sourced imports under `lambdas/shared/`; packaging recipes are parsed from `terraform/build_packages.sh`; ships-all accumulates across `copy_py_dir` + `copy_shared_all` / `copy_src_file`; and pins require shared / web_ui_auth copies to be actually present (a commented-out or removed copy fails CI).

### Phase 5: handler decomposition + lazy boto3 clients

Both email-processor God-handlers are decomposed along the seams that already work into **flat sibling modules** in the same directory (bare-name imports, exactly like the `ses_auth`/`template_parser`/`derived_fields` pattern), so the packaging `copy_py_dir` ships every new sibling automatically — no packaging change beyond the exact-set pin. Every move is a pure delete-here/add-there; `derived_fields.py`, both `template_parser.py`, the `validate_ai_fallback` gates, and `lambdas/shared/` are byte-untouched.

**PO** (`handler.py` → 5 siblings + `prompts.py`): `handler.py` keeps the event loop, fail-closed SES auth, and `email_type` routing; `extraction.py` owns `extract_with_claude` + `_EMAIL_TAG_RE`; `enrichment.py` owns `enrich_parsed` + `pad_zip` (PO-only — WO has no enrichment stage) as a byte-identical move including the derived-field shadow block; `telemetry.py` owns the EMF `ParseMethod`/`DerivedFieldAgreement` emit wrappers; `persistence.py` owns `_write_fields`/`_merge_update`/`save_*` — with the two byte-identical `save_new_po`/`save_revision` **collapsed into one `_save_merge`** plus two thin wrappers differing only in the log verb (behavior-identical to both originals, sticky-cancel `ConditionExpression` guard intact; `save_cancellation` stays its own function). **WO** splits into ~5 concerns (no `enrichment`), keeping `validate_ai_fallback` **and** the `re.fullmatch(r"[0-9]+", work_order_id)` key guard in the handler loop AHEAD of both `save_work_order` and `save_event` (it protects the partition key and the `#`-delimited `comment_id` range-key segment), and keeps `_header_date_iso`/`comment_id` determinism together with `save_event` in `persistence.py`.

`EXTRACTION_PROMPT` (the ~181-line prompt) moves to `prompts.py` with a cross-reference header to the second authoritative copy of the trade/site/fiscal rule tables in `derived_fields.py`; `handler.py` keeps a `from prompts import EXTRACTION_PROMPT` re-export so `handler.EXTRACTION_PROMPT` still resolves for the tests that dereference it.

**Lazy cached boto3 clients.** Each I/O module initializes its client cache to `None` and populates it through a private `_get_<client>()` accessor on first call (`extraction.bedrock`, `persistence.dynamodb`, `handler.s3`); pure modules (`enrichment`, `telemetry`, `prompts`) import no boto3. The cache attribute keeps its original public name, so a test patches the same attribute — only the owning **module** moved (e.g. `setattr(persistence, "dynamodb", fake)`). Building the client at first *call* (deep inside a test) rather than at import also strengthens the moto-before-handler invariant.

**Behavior preserved (pinned by new tests).** PO emits `ParseMethod=ai_fallback` **before** the Bedrock call (a throttle that raises still leaves the pre-call datapoint), and a gate rejection is an additive second `ai_fallback_rejected` datapoint (PO's deliberate double-count); WO emits **after** its gate, mutually exclusive; the `DerivedFieldAgreement` shadow telemetry stays `ai_fallback`-only; `_save_merge` issues byte-identical `update_item` calls for both `new_po` and `revision`; and the WO key guard fires before either save.

## Security

**Web UI auth (defense-in-depth).** The `po-web-ui` / `workorder-web-ui` handlers refuse unauthenticated requests even though their public Function URLs were removed (INFRA-74). Each requires a shared secret in the `X-Auth-Token` header (or `Authorization: Bearer <token>`), compared in constant time against the configured token. The handler **fails closed** if the token is unset or unreadable (denies all). The token lives in the Secrets Manager secret `procurement-ingest/web-ui-auth-token`; only its ARN is passed to the Lambda (`WEB_UI_AUTH_TOKEN_SECRET_ARN`), and the value is fetched at runtime — never embedded in the CloudFormation template or Lambda env vars. The fetched value is cached in the warm container with a short TTL (5 min) so a rotated secret propagates without waiting for the execution environment to recycle. This is a defense-in-depth floor for a detached URL, not primary auth.

**Output escaping.** All caller-influenced values (including prompt-injectable strings Claude may return for `total_amount`/line-item amounts) are HTML-escaped before interpolation to prevent stored XSS.

## Shared Resources

### `purchase-orders` table (owned here)

The `purchase-orders` DynamoDB table is **owned by this repo's Terraform config** (defined in `terraform/po_ddb.tf` with retain lifecycle, `StreamViewType.NEW_IMAGE`, and SSE-KMS encryption with the shared customer-managed CMK `alias/seahaven-dynamodb`, INFRA-95 / M-3). The `po-email-processor` Lambda is the authoritative writer — it performs the merge inserts, field-level revision merges, and cancellation updates described above.

**Consumers (read-only):**

| Consumer | How it reads | Purpose |
|---|---|---|
| `procurement-api` (this repo) | `Table.from_table_name` + `grant_read_data` | REST reads (`GET /purchase-orders*`) |

`seahaven-slack-bot`, the former external consumer, was decommissioned 2026-07-23 and its cross-account grants removed. External consumers now read through the `procurement-api` REST contract (`lambdas/api/openapi.json`), never by importing the table directly.

**Schema-coordination rule:** Any change to the `purchase-orders` schema (partition key, item shape, attribute names, streams view type) must be reflected in the API's OpenAPI schemas in the same PR (the spec-drift test pins paths, not item shapes — the schema fields are maintained by hand). Treat schema changes as a contract migration, not a local edit.

**Known exception (INFRA-51):** `amazon-po-parser` currently writes directly to `purchase-orders` outside this stack (backfill/enrichment scripts). This second writer is being folded into the `po-ingest` pipeline so this stack is the sole writer; until INFRA-51 closes, coordinate any schema change with `amazon-po-parser` as well.

### `work-orders` and `work-order-comments` tables (owned here)

> **PLAT-11 cutover complete 2026-08-07; PascalCase tables destroyed in Phase 3.**
> Live physical names are kebab-case. GitHub #24 superseded (issues disabled on
> this repo); track [PLAT-11](https://seahaven.atlassian.net/browse/PLAT-11).

> Cutover record: [`docs/plat-11/cutover-runbook.md`](docs/plat-11/cutover-runbook.md).

Both tables are **owned by this repo's Terraform config** (`terraform/wo_ddb.tf`, `prevent_destroy` on kebab tables):

- `work-orders` — PK `work_order_id` (S).
- `work-order-comments` — PK `work_order_id` (S), SK `comment_id` (S).

> **`comment_id` format change (issue #23).** The `work-order-comments` range key is now
> `work_order_id#<comment_time|nocomment>#<sha256(s3_object_key)[:12]>`
> (e.g. `11144580730#2026-04-27T23:51:48#a1b2c3d4e5f6`, or `…#nocomment#…` when the source
> email carries no comment time). Previously it was `work_order_id#<timestamp>`, where two
> emails on the same WO with an identical/absent comment time collided and overwrote each other.
> The 12-hex suffix is derived from the **S3 object key alone** — deterministic, so a Lambda
> async **retry** of the same object produces a byte-identical key (idempotent, no duplicate row),
> while two distinct emails on the same WO get distinct keys. Wall-clock `now()` is kept **out** of
> the key. Consumers that split on `#` and read index `[0]`/`[1]` are unaffected; anything that
> treated "everything after the first `#`" as a bare timestamp now also captures the hash segment.

> **Timestamp format shift.** All stored ISO timestamps (`created_at`, `updated_at`, `ingested_at`
> on WO; `processed_at`, `cancelled_at` on PO) moved from naive `datetime.utcnow().isoformat()`
> to timezone-aware `datetime.now(timezone.utc).isoformat()`, so they now carry a `+00:00` suffix
> (e.g. `2026-07-15T12:00:00+00:00`). Downstream parsers that assumed a naive/no-offset string
> must accept the offset.

Both currently use default DynamoDB encryption — they are **not** yet on the shared customer-managed CMK (`alias/seahaven-dynamodb`, INFRA-95 / M-3); that migration is tracked in INFRA-6. The `workorder-email-processor` role no longer holds a pre-emptive encrypt/decrypt grant on that CMK (removed in the 2026-06-17 security sweep — it was unused while the tables are unencrypted and extended the role's decrypt reach to the CMK protecting `purchase-orders`). Re-add the grant as part of the INFRA-6 migration, at which point `grant_read_write_data` on the then-encrypted tables propagates the needed key permissions automatically.

**Consumers (read-only) — data contract:** `seahaven-slack-bot` (the former external reader) was decommissioned 2026-07-23; its grants are gone. Current consumers: the `procurement-api` Lambda (this repo, `Table.from_table_name` + `grant_read_data`, serving `GET /work-orders*`), and the `workorder-shoc-emitter` stream consumer (this repo) — both tables now stream `NEW_AND_OLD_IMAGES`, and the emitter is an active consumer of those streams (ESMs enabled 2026-07-30). External readers (SHOC) consume through the REST contract (`lambdas/api/openapi.json`) and the webhook contract (`docs/shoc-webhook-contract.md`); both documents' field lists mirror `lambdas/wo/email_processor/persistence.py`. Any change to table name, key schema, attribute names, or encryption configuration (e.g. the INFRA-6 CMK migration) must update those two contracts in the same PR — the tables are imported by name, so there is no compile-time link and breakage surfaces at runtime.

**Stream field contract (`site_code`).** Three definitions of "is this a valid site code" have existed in this repo at once. The canonical shape is `derived_fields._STRICT_CODE_RE` (`[A-Z][A-Z0-9]{2,4}`, `fullmatch`) plus its skip-list semantics (a code-shaped token is only a real site code if it is *not* skip-listed, e.g. `LLC`/`INC`/`CORP`/`LTD`/`ATTN`), used by `derive_site_code()` (the `enrich_parsed()` classifier, see the **Derived fields** note in the Purchase Orders flow above). Honestly noted, not papered over: `lambdas/po/site_extractor/handler.py`'s own `SITE_CODE_PATTERN` (`[A-Z]{2,4}\d{1,2}`, prefix-anchored `.match`, digit-requiring) still diverges from the canonical shape as of this phase — it rejects valid all-letter codes like `KLAL` (which surface as permanent `pending-site-review` rows) and accepts overlong junk like `DLI6X`/`SNY55` that the canonical `fullmatch` would not. Reconciling `po-ingest-site-extractor`'s direct-field validation onto the canonical `derived_fields` shape is Phase 6 scope, tracked separately — this paragraph will be updated when it lands.

### `verified-sites` table (owned here)

Owned by this repo's Terraform config (`terraform/po_ddb.tf`). PK `siteCode` (S); default DynamoDB encryption (NOT the shared CMK).

**Consumer (read-only) — data contract:** `seahaven-slack-bot` (former reader, incl. the INFRA-138/INFRA-180 `by-state` GSI drift saga) was decommissioned 2026-07-23 — the GSI-drift issue died with it. Current consumer: the `procurement-api` Lambda (`GET /verified-sites*`). The API's `VerifiedSite` schema deliberately promises only pipeline-written fields (`siteCode`, `address`, `city`, `state`, `zip`, `fullAddress`, `locationCode`, `poCount`, `sourcePOs`) — `latitude`/`longitude`/`notes` were manually curated legacy attributes the extractor never writes, and are not part of the contract.

## Documentation

The canonical map of Sea Haven's AWS infrastructure lives in Confluence. This project's live path is HCP Terraform workspace `procurement-ingest-prod` (PLAT-86).

- **[AWS Architecture Map](https://seahaven.atlassian.net/wiki/spaces/IT/pages/1540098)** (Confluence, IT space, page 1540098)
- Import / dispose notes: `docs/plat-86/`

## CI/CD

GitHub Actions with reusable workflows from `Sea-Haven-Industries/.github` (all pinned to a commit SHA of `main`):
- **CI** (`ci.yaml`, PR to `main`): linting + pytest via `ci-python-sam.yaml` (`run-cdk-synth: false`; HCP Terraform is the deploy path).
- **Terraform CI** (`ci-terraform.yaml`, PR to `main` when `terraform/**` changes): `fmt -check`, `init -backend=false`, `validate`.
- **CD:** HCP Terraform workspace `procurement-ingest-prod` (VCS-bound to `main`, working directory `terraform/`, Manual apply until sealed). There is no push-to-main CDK/SAM deploy workflow. Post-apply smoke: `scripts/post-deploy-smoke.sh`.
- Plus dependency review and PR labeler workflows on every PR

Branch protection on `main` — all changes through PR.

## Setup

**Account prerequisites** — Terraform imports these by name/ARN; they must already exist in seahaven-prod (provisioned by seahaven-org-baseline and out-of-band scripts):

- SES receipt rule set `INBOUND_MAIL` + verified `int.seahaven.com` domain identity
- SNS topic `site-alerts` (with the `alias/seahaven-alarm-topics` CMK)
- SSM param `/seahaven/dynamodb/cmk-arn` → KMS `alias/seahaven-dynamodb`
- SSM param `/procurement-api/custom-domain/certificate-arn` (ACM in prod; DNS validation/alias for `procurement-api.seahaven.com` stays in mgmt via `scripts/setup_procurement_api_domain.sh`)
- Secrets Manager secret `procurement-ingest/web-ui-auth-token`
- HCP plan/apply roles `hcptf-procurement-ingest-plan` / `hcptf-procurement-ingest` (org-baseline terraform-substrate)

1. Set workspace variables in HCP (`procurement-ingest-prod`) from `terraform/terraform.tfvars.example` (secret ARNs only; never secret values).
2. Bedrock model access (account first-use). Anthropic Marketplace agreements and inference-profile reach are account-level; processor roles already carry `bedrock:InvokeModel` grants under `/tf-managed/`.
3. Create the web UI auth-gate shared secret if missing (imported by name, not Terraform-managed):
   ```bash
   aws secretsmanager create-secret --name procurement-ingest/web-ui-auth-token --secret-string "$(openssl rand -hex 32)"
   ```
4. Apply from the HCP workspace (Manual apply until the stack is sealed). Do not use local `terraform apply` against prod.
5. Post-apply smoke:
   ```bash
   AWS_PROFILE=seahaven-prod ./scripts/post-deploy-smoke.sh
   ```
6. Dashboards: `po-web-ui` and `workorder-web-ui` have no public endpoint (Function URLs removed 2026-06-08, INFRA-74). Fetch the token and forward it in the event's `headers`:
   ```bash
   TOKEN=$(aws secretsmanager get-secret-value --secret-id procurement-ingest/web-ui-auth-token --query SecretString --output text)
   aws lambda invoke --function-name po-web-ui --payload "{\"headers\":{\"x-auth-token\":\"$TOKEN\"}}" /tmp/out.json
   ```

> **Removed (PLAT-88):** `githubdeploy-procurement-ingest` OIDC deploy role and `infra/deploy-role/` artifacts deleted after the HCP hard-cut. Do not recreate a mgmt twin (deleted in PLAT-67).

## Tests

Offline unit tests (no AWS, no network) run via pytest from the repo root:

```bash
pip install -r tests/requirements.txt
pytest
```

**Test-root consolidation (Phase 8).** Both test roots are kept (`tests/` for genuinely cross-pipeline suites, plus each pipeline's own `lambdas/*/email_processor/tests/`), but loading is now single-sourced. A new repo-root `conftest.py` — not `tests/conftest.py`, which is never an ancestor of the pipeline test roots and so cannot load for a standalone `pytest lambdas/po/email_processor/tests` run — sets the dummy AWS env, imports `moto` before any handler import (registers moto's botocore stubber hook so boto3 sessions created afterwards are stubbable), and exposes the one `load_lambda_module(pipeline, name)` loader (implemented in `tests/support/loader.py`) that every handler exec in the suite goes through, including its `sys.modules` save/restore dance for the per-pipeline duplicated bare names (`template_parser`, `persistence`, etc. — `derived_fields`/`prompts`/`telemetry`/`extraction`/`enrichment` too). `tests/support/` is a small shared package (`FakeTable`, `FakeDynamoResource`, `FakeS3`, `load_email`, `load_golden`) used by both pipelines' `_po_parser_support.py`/`_wo_parser_support.py` shims — `load_golden` decodes JSON money with `parse_float=Decimal` unconditionally (load-bearing for PO's exact-money goldens; proven safe for WO, whose 55 golden fixtures contain zero float-typed JSON numbers). `test_local.py` (a root-level manual script that imported a handler at collection time, bypassing the loader gate, and globbed a nonexistent `samples/` dir) is deleted — the golden suites cover its role.

Coverage:
- `tests/` — shared handler + cross-pipeline tests: `parse_raw_email` (`test_parse_raw_email.py`), the fail-closed sender-authentication parser (`test_ses_auth.py`, INFRA-107), the packaging/handler-import consistency check (`test_bundle_consistency.py` — see [Deploy-Pipeline Guards](#deploy-pipeline-guards-phase-0)), the `scripts/reprocess.py` synthetic S3-event-shape contract (`test_reprocess_contract.py`, Phase 7), and, new in Phase 8: the handler-level SES-auth reject seam per pipeline (`test_handler_auth_seam.py` — no auth monkeypatch + an empty `ALLOWED_DKIM_DOMAINS` must yield zero Bedrock calls, zero writes, no raise), and both web_ui functions' first coverage — fail-closed auth (`test_web_ui_auth.py`) and the handlers themselves, including a 401-without-a-table-scan assertion and a hostile-field escaping regression lock (`test_web_ui_handlers.py`).
- `lambdas/wo/email_processor/tests/` — the deterministic WO parser suite: golden-file tests over 55 real scrubbed `.eml` fixtures (`test_parser.py`), fail-closed validation-gate rules and adversarial/injection cases (`test_validation_gate.py`), the issue #23 `comment_id` idempotency invariants (`test_comment_id.py`), the Bedrock-fallback dispatch/EMF-metric behavior with a mocked `invoke_model` (`test_bedrock_fallback.py`, which also carries the Phase 5 behavior pins — WO's mutually-exclusive emit and the `[0-9]+` key guard ahead of both saves), and the Phase 0 direct-invoke healthcheck contract (`test_healthcheck.py`). Golden JSON lives under `tests/fixtures/expected/`. New in Phase 8: `test_wo_bedrock_transport.py` (transport errors — throttle, missing `content`, empty content, non-JSON model text — plus the hand-computed no-double-count pin for the `handler.py` except-and-reraise metric wrap, and the multi-record failure-isolation pin) and `test_wo_merge.py` (a moto-backed mirror of `test_po_merge.py` against the `WorkOrders` table — null-status never clobbers `wo_status`, `created_at` immutable, `status`→`wo_status` mapping, `None` fields absent from `SET`, `record_type` only-when-present). After Phase 5 the suite patches accessors on the owning siblings (`extraction.bedrock`, `persistence.dynamodb`, `persistence.datetime`, `persistence.save_*`) rather than on `handler`.
- `lambdas/po/email_processor/tests/` — the deterministic PO parser suite: golden-file tests over real scrubbed `.eml` fixtures (17 single-line new-PO + 8 cancellations, exact `Decimal`-aware golden comparison via `parse_float=Decimal`), fail-closed validation-gate coverage for **every** gate reason code (fixture-driven for body-level triggers under `fixtures/adversarial/`, direct `validate()` unit tests for candidate-level mutations), real multi-line and comment/non-Coupa fallback fixtures under `fixtures/ai-fallback/`, dual line-ending (CRLF/LF) parse-identity, two-path `enrich_parsed`/`save_new_po` parity (the site-extractor stream-contract guard), fixture hygiene (`ses_auth` pass + scrub-marker leak sweep), the Bedrock-fallback dispatch/EMF-metric behavior plus the Phase 5 behavior pins (`test_po_bedrock_fallback.py` — the pre-Bedrock `ai_fallback` emit and the additive rejected double-count), the `_save_merge` collapse parity to both `save_new_po`/`save_revision` (`test_po_save_merge_parity.py`), the `ai_fallback`-only shadow telemetry after the `enrich_parsed` move (`test_po_derived_wiring.py`, which gains the PO-DC-02 64-char `po_number` clamp regression pin in Phase 8), and the Phase 0 direct-invoke healthcheck contract (`test_po_healthcheck.py`). New in Phase 8: `test_po_bedrock_transport.py` (the same four transport-error cases as WO, asserting the pre-call `ai_fallback` metric survives and no partial write occurs, plus the multi-record failure-isolation pin) and, moved here from `tests/`: `test_po_merge.py` (#97, moto-backed merge-write semantics) and `test_pad_zip.py` (zip-code padding). After Phase 5 the suite patches accessors/constants on the owning siblings (`po_extraction.bedrock`/`.BEDROCK_MODEL_ID`/`._EMAIL_TAG_RE`, `po_persistence.dynamodb`/`.PO_TABLE`, `po_enrichment.derive_all`/`.pad_zip`, `po_telemetry.DERIVED_METRIC_NAME`) rather than on `po_handler`; re-exported names it calls (`save_*`, `extract_with_claude`, `enrich_parsed`, `_emit_parse_method_metric`, `EXTRACTION_PROMPT`) stay on `po_handler`.

All three roots are discovered by `pytest.ini` (`testpaths`).

**Coverage floor (Phase 8).** `pytest.ini`'s `addopts` runs `pytest-cov` with an explicit `--cov` path per first-party package (`lambdas/po/email_processor`, `lambdas/wo/email_processor`, `lambdas/po/web_ui`, `lambdas/wo/web_ui`, `lambdas/po/site_extractor`, `lambdas/shared`) rather than relying on an `__init__.py` package marker — none of these dirs have one, and adding one would perturb flat packaging for zero runtime benefit; `pytest-cov`'s path form measures by source file regardless of package markers. `site_extractor` is deliberately included even though it measures 0% until Phase 6 lands — coverage honesty, not a silent skip. `.coveragerc` omits `*/tests/*` so the pipeline test dirs (which sit inside their own `--cov` path) don't dilute the number. `--cov-fail-under` is the new permanent CI floor (constraint 10 — never ratcheted down).

**Ruff C901/PLR floor (Phase 8).** `ruff.toml` adds `extend-select = ["C901", "PLR"]` with `max-complexity = 12`, making the tree's pre-existing `# noqa: PLR09xx` suppressions load-bearing instead of inert (no config previously enabled the rules they suppressed). `scripts/` is in the lint scope. Findings that could not be split or were out of this phase's file-ownership were resolved with a per-file `[lint.per-file-ignores]` entry carrying a written justification — most notably `derived_fields.py` (shadow-bake freeze: even an in-file `noqa` comment is a barred edit) and the other Phase 3/5/6/7 frozen modules (`enrichment.py`, WO `template_parser.py`, both `ses_auth.py`/`emf.py`, `site_extractor/handler.py`, `scripts/reprocess.py`). PO's `template_parser.py` — this phase's one in-scope split target — clears the ceiling by decomposing `extract_new_po` and `_validate_new_po_values` into per-rule helpers instead of an ignore.

## Scripts

**Reprocess emails** (re-invoke an email-processor with a synthetic S3 event). `reprocess.py` is now **pipeline-general**: `--pipeline po|wo` selects the function + raw-email bucket. **Targeted replay** (`--key` one object, `--prefix`, or `--since` a `LastModified` timestamp) is the default, preferred mode; the full-prefix sweep is demoted behind an explicit `--all`. Every mode is dry-run unless `--execute`. See the [DLQ recovery runbook](docs/runbook-dlq-recovery.md) for the targeted single-key re-invoke flow.
```bash
python scripts/reprocess.py --pipeline po --key inbound/2026/msg.eml            # targeted, dry-run
python scripts/reprocess.py --pipeline po --key inbound/2026/msg.eml --execute  # targeted, re-invoke one
python scripts/reprocess.py --pipeline wo --all --execute                       # demoted full-prefix sweep (see --all caveats)
```

**Backfill verified sites** (one-time scan of historical POs):
```bash
python scripts/backfill_sites.py
```

**Replay SHOC webhooks** (rebuild work-order webhook events from DynamoDB and re-POST them, marked `"replay": true`). `replay_shoc_webhooks.py` is the operator runbook for the `workorder-shoc-emitter-failures` / `-rejected` alarms: when deliveries were parked (SHOC down past the 24h retry window, or a contract-bug 4xx), replay re-sends the affected work orders from **current table state** — receivers dedupe on the deterministic `delivery_id`, so overlapping or repeated runs are harmless. Selection is exactly one of `--work-order-id` (repeatable, targeted) or `--since` (a full table Scan — a count banner prints per table); `--events` narrows to state rows, comments, or both. `--url` is required with no default — replay must be a deliberate act against a known receiver. Every mode is dry-run unless `--execute`.
```bash
python scripts/replay_shoc_webhooks.py --url https://... --work-order-id 11144580730            # targeted, dry-run first
python scripts/replay_shoc_webhooks.py --url https://... --work-order-id 11144580730 --execute  # then re-POST
python scripts/replay_shoc_webhooks.py --url https://... --since 2026-07-24T02:00:00Z --execute # everything touched since (full Scan)

**PLAT-11 WO table rename** (PascalCase → kebab, freeze cutover). See
[`docs/plat-11/cutover-runbook.md`](docs/plat-11/cutover-runbook.md). Dry-run copy:

```bash
python scripts/migrate_wo_tables.py copy
python scripts/migrate_wo_tables.py copy --execute
python scripts/migrate_wo_tables.py verify
```
```

## Directory Structure

```
terraform/             # Sole IaC path (HCP workspace procurement-ingest-prod)
  build_packages.sh    # Lambda zip packaging (copy_py_dir / copy_shared_all / copy_src_file)
  *.tf                 # PO / WO / API / SHOC resources imported under PLAT-86
lambdas/               # Runtime source; packaged by terraform/build_packages.sh
                        # email processors: copy_py_dir + copy_shared_all;
                        # web_ui: copy_py_dir + shared/web_ui_auth.py only;
                        # site_extractor / SHOC: copy_py_dir of their own dirs
  shared/              # Phase 3: single-sourced first-party modules, landed FLAT (no
                        # __init__.py -- bare-name imports) into each email-processor zip
    ses_auth.py          # fail-closed SES sender-auth (INFRA-107) -- one copy, one fix
    web_ui_auth.py       # fail-closed X-Auth-Token gate + token cache (INFRA-74; also gates the API /docs routes)
    email_parsing.py     # parse_raw_email (WO superset; returns cc unconditionally)
    emf.py               # generic CloudWatch EMF emitter (dimension-sets pinned once)
  api/                 # procurement-api Lambda (REST reads over both pipelines' tables)
    handler.py           # healthcheck early-return + docs-token gate + route dispatch + error mapping
    router.py            # single route table (spec-drift test pins it to openapi.json)
    pagination.py        # opaque base64(LastEvaluatedKey) cursor encode/validate (hostile cursor -> 400)
    serialization.py     # Decimal-safe JSON responses
    wo_repo.py           # WorkOrders/WorkOrderComments reads (paginated Scan / PK Query)
    po_repo.py           # purchase-orders/verified-sites reads (no VendorReplies -- dead table)
    openapi.json         # OpenAPI 3.1 source of truth (paths + outbound `webhooks` section)
    docs.html            # Redoc shell, SHOC design-system theme; handler inlines bundle/fonts/spec at request time (single token-gated request, no CDN)
    redoc.standalone.js  # vendored Redoc bundle (redoc 2.5.3, MIT), inlined into /docs; read-only docs, no try-it-out
    fonts.css            # SHOC fonts (DM Sans/Montserrat/JetBrains Mono, @fontsource latin subsets) as data URIs, inlined into /docs
  po/                  # PO pipeline Lambdas
    email_processor/     # Phase 5: God-handler decomposed into flat siblings (bare-name
                         # imports; copy_py_dir ships every new sibling automatically)
      handler.py         # event loop + fail-closed SES auth + email_type routing + {"healthcheck": true} early-return; lazy s3 accessor; re-exports EXTRACTION_PROMPT (4 tests deref handler.EXTRACTION_PROMPT)
      extraction.py      # extract_with_claude + _EMAIL_TAG_RE + EXTRACTION_PROMPT import; lazy bedrock accessor
      enrichment.py      # enrich_parsed + pad_zip (PO-only; no boto3); derived-field shadow block
      telemetry.py       # EMF ParseMethod + DerivedFieldAgreement emit wrappers (stdout EMF; no boto3)
      persistence.py     # _write_fields/_merge_update + save_cancellation; save_new_po/save_revision collapsed into one behavior-identical _save_merge (sticky-cancel guard intact); lazy dynamodb accessor
      prompts.py         # EXTRACTION_PROMPT (~181 lines); cross-ref header to derived_fields' rule tables
      template_parser.py # pure deterministic Coupa parser + fail-closed validation gate; Phase 8:
                         # extract_new_po/_validate_new_po_values split into C901/PLR-clean per-rule
                         # helpers (behavior byte-identical; see the ruff C901/PLR floor below)
      derived_fields.py  # deterministic site_code/trade/fiscal_year classifier (untouched by Phase 3/5/8 -- shadow-bake freeze; ruff per-file-ignore instead of an in-file noqa)
      tests/             # golden-file + validation-gate + fallback-dispatch + healthcheck + save-merge-parity + behavior-pin tests + fixtures
        conftest.py                  # fake_dynamo fixture, patches persistence.dynamodb
        _po_parser_support.py        # Phase 8: thin shim -- re-exports tests.support fakes/loader; PO module handles (template_parser, prompts, telemetry, extraction, enrichment, persistence)
        test_po_derived_fields.py    # derived_fields.derive_all() unit tests
        test_po_derived_wiring.py    # ai_fallback-only shadow-telemetry wiring; Phase 8 gains the PO-DC-02 64-char po_number clamp regression pin
        test_po_bedrock_transport.py # Phase 8: Bedrock transport errors (throttle/missing-content/empty-content/non-JSON) + pre-call metric survival + multi-record failure-isolation pin
        test_po_merge.py             # Phase 8: moved from tests/ -- PO merge-write semantics tests (#97), moto-backed
        test_pad_zip.py              # Phase 8: moved from tests/ -- PO zip-code padding tests
    site_extractor/
    web_ui/              # handler.py imports `from web_ui_auth import is_authenticated` (shared)
  wo/                  # WO pipeline Lambdas
    email_processor/     # Phase 5: decomposed into ~5 flat siblings (no enrichment stage);
                         # validate_ai_fallback + the [0-9]+ work_order_id key guard stay in
                         # the handler loop AHEAD of both saves
      handler.py         # event loop + fail-closed SES auth + [0-9]+ work_order_id key guard + {"healthcheck": true} early-return; lazy s3 accessor; re-exports EXTRACTION_PROMPT; Phase 8: the Bedrock call is wrapped in try/except so a transport error emits one ai_fallback/bedrock_error datapoint then re-raises (a gate rejection still emits only ai_fallback_rejected -- no double-count)
      extraction.py      # extract_with_bedrock + _EMAIL_TAG_RE + EXTRACTION_PROMPT import; lazy bedrock accessor
      telemetry.py       # emit_parse_metric EMF wrapper (stdout EMF; no boto3)
      persistence.py     # save_work_order + _header_date_iso + save_event (#23 comment_id determinism kept WITH persistence); lazy dynamodb accessor
      prompts.py         # EXTRACTION_PROMPT; cross-ref header to template_parser.CONTRACT_KEYS
      template_parser.py # pure deterministic parser + fail-closed validation gate; Phase 8: the
                         # template-path bad-status branch now returns "invalid_status" (was a
                         # copy-paste "malformed_site_code") -- matches validate_ai_fallback's code
                         # for the identical condition
      tests/             # golden-file + validation-gate + comment_id + fallback + healthcheck + behavior-pin tests
        conftest.py                  # fake_dynamo fixture, patches persistence.dynamodb (module-top `from _wo_parser_support import wo_persistence`, Phase 8)
        _wo_parser_support.py        # Phase 8: rewritten OFF the bare `import handler` strategy -- re-exports tests.support fakes/loader; WO module handles (template_parser, persistence, extraction, telemetry)
        test_wo_merge.py             # Phase 8: moto-backed WorkOrders save_work_order merge-semantics mirror of test_po_merge (null-status never clobbers wo_status, created_at immutable, status->wo_status mapping, None fields absent, record_type only-when-present)
        test_wo_bedrock_transport.py # Phase 8: Bedrock transport errors + the hand-computed emitted-series no-double-count pin for the handler.py except-and-reraise wrap + multi-record failure-isolation pin
        fixtures/
          ses-stamped/
            auth-pass-01.eml         # Phase 8: the one new fixture allowed this phase -- a synthesized Authentication-Results header block (from WO_SES_HEADER in test_ses_auth.py), not scraped mail
    web_ui/              # handler.py imports `from web_ui_auth import is_authenticated` (shared)
    shoc_emitter/        # SHOC webhook emitter (ACTIVE since 2026-07-30; see the SHOC
                         # webhook feed section). Flat siblings, bare-name imports
      handler.py         # thin per-record event loop + partial-batch failure report + rejected-queue parking
      envelope.py        # pure stream-record -> envelope mapping + event classification (incl. echo guard)
      delivery.py        # TTL-cached secret fetch + HMAC signing (sign_body) + POST + response classification
    shoc_hmac_rotator/
      handler.py         # 30-day Secrets Manager rotation (single-user: createSecret/finishSecret;
                         # dual-key overlap, kid YYYY-MM-DDTHH)
scripts/
  reprocess.py
  backfill_sites.py
  replay_shoc_webhooks.py # rebuild + re-POST webhook events ("replay": true) -- the
                          # -failures/-rejected alarm runbook; dry-run unless --execute
  post-deploy-smoke.sh   # CD gate: synchronous healthcheck invoke of the three smoke-gated functions, checks FunctionError
conftest.py             # Phase 8: THE repo-root session-invariant conftest. rootdir is pinned by
                        # pytest.ini at repo root, so this loads for every pytest invocation shape --
                        # including a standalone `pytest lambdas/po/email_processor/tests` -- before
                        # any collection import. Sets the dummy AWS env, imports moto BEFORE any
                        # handler import (registers moto's botocore stubber hook so boto3 sessions
                        # created afterwards are stubbable), and re-exports
                        # `tests.support.loader.load_lambda_module`. Owns the session fixtures
                        # (po_handler, wo_handler, email_handler, ses_auth, po_persistence,
                        # po_enrichment, wo_persistence). Supersedes the old tests/conftest.py
                        # (deleted -- it was never an ancestor of the pipeline test roots, so it
                        # could not carry these invariants for a standalone pipeline-root run).
tests/
  requirements.txt          # Test-only deps (moto, pytest-cov)
  support/                   # Phase 8: shared test package (tests/ itself has no __init__.py --
                             # PEP-420 namespace resolution via the root conftest's sys.path insert)
    __init__.py                # load_lambda_module + REPO_ROOT re-export, FIXTURE_DIRS, stems/
                                # load_raw/load_email/load_golden (parse_float=Decimal, pinned safe
                                # for both pipelines), and the superset FakeTable/FakeDynamoResource/FakeS3
    loader.py                   # the ONE load_lambda_module + sys.modules save/restore dance
                                # (dependency-topological sibling tuple + web_ui_auth); every
                                # pipeline handler exec in the suite goes through this
  test_parse_raw_email.py   # MIME parsing tests (PO + WO handlers) -- stays at root, cross-pipeline
  test_ses_auth.py          # Sender-authentication parser tests (INFRA-107) -- stays at root, cross-pipeline
  test_handler_auth_seam.py # Phase 8: handler-level SES-auth reject seam, per pipeline -- no auth
                             # monkeypatch + empty ALLOWED_DKIM_DOMAINS -> zero Bedrock calls, zero
                             # writes, no raise (closes the "delete the gate line, tests still pass" hole)
  test_web_ui_auth.py       # Phase 8: shared/web_ui_auth.py -- fail-closed on unset ARN / Secrets
                             # Manager exception, TTL cache refresh, header-matrix case-insensitivity,
                             # non-ASCII-token documenting pin
  test_web_ui_handlers.py   # Phase 8: both web_ui handlers (0% coverage before this phase) -- 401
                             # without a table scan, authenticated render path, hostile-field escaping
                             # regression lock
  test_bundle_consistency.py  # AST check: bundling command ships every handler.py sibling import
  test_reprocess_contract.py  # reprocess.py synthetic S3-event-shape contract (Phase 7)
```