procurement-ingest/README.md

Procurement Ingest

Unified email ingestion pipelines for Amazon procurement data. Two independent pipelines — purchase orders (Coupa) and work orders (APM/Hexagon EAM) — share a single repo and deploy to seahaven-prod via HCP Terraform workspace procurement-ingest-prod (PLAT-86).

Pipelines

Purchase Orders (po-ingest stack)

Coupa PO emails are received at amazon_po@int.seahaven.com, parsed deterministic-template-first with a Claude-Haiku-4.5-on-Bedrock fallback, and written to the purchase-orders DynamoDB table.

Flow:

1. Coupa sends a PO email (new, revision, or cancellation).
2. SES (INBOUND_MAIL rule set) drops the raw MIME into s3://po-ingest-emails-{AccountId}/inbound/.
3. S3 ObjectCreated triggers the po-email-processor Lambda.
4. Fail-closed sender authentication (INFRA-107): the SES-stamped Authentication-Results header must show dkim=pass for amazon.coupahost.com (see Sender authentication); otherwise the email is logged and dropped.
5. Parse: a pure, offline template parser (template_parser.py) tries the two known Coupa templates first — coupa_new_po (95.5% of traffic) and coupa_cancellation (2.9%) — behind a strict fail-closed validation gate. Only on a miss/invalid result does the Lambda fall back to the Claude-on-Bedrock AI extractor (InvokeModel), which extracts the identical structured-JSON contract (PO number, status, supplier, nested ship-to, line items). Numeric amounts are Decimal on both paths (the AI decode uses parse_float=Decimal; DynamoDB rejects floats). The derived classification fields (site_code, trade, fiscal_year) are computed by a deterministic Python classifier in the shared enrich_parsed() post-stage (zip padding, ship_to_raw/state promotion, metadata) which runs identically on both paths — Python-authoritative on the template path and an LLM-authoritative-with-Python-shadow bake on the AI-fallback path (see the Derived fields note below).
6. Merge write to DynamoDB:
   • new_po — merge insert. Creates the PO, or backfills data into a pre-existing Cancelled skeleton left by an out-of-order cancellation (preserving the Cancelled status). No longer silently dropped when a record already exists.
   • revision — field-level merge (update_item SETs only the fields present in the revision). A revision that omits line_items/supplier no longer deletes them. Will not un-cancel a Cancelled PO.
   • cancellation — marks the row Cancelled (creating a minimal skeleton if the cancellation arrives before the new_po).
7. DynamoDB Streams feeds the site extractor (po-ingest-site-extractor) — real-time site address extraction into the verified-sites table. (LedgerFlow / seahaven-slack-bot/po-sync, the former second stream consumer, was decommissioned 2026-07-23.)

Deterministic template parser. template_parser.try_deterministic_parse() classifies by exact subject regex, extracts the nested contract (supplier{}, ship_to{} — 8 keys, line_items[] — 10 keys per item), and returns a parsed result only if it passes a fail-closed validation gate: recursive exact key-set at every nesting level; email_type emitted only on the exact new-PO subject and a confirmed-safe Status (never defaulted — a cancellation misrouted as new_po would defeat the sticky-Cancelled guard); po_number shape + byte-equality with the subject, the body PO ID, the Amazon Purchase Order # heading, and the orders/<id> URL; duplicate-label anchor integrity (Supplier/Shipping/Total each appear twice, the first Shipping must be the literal None placeholder); money fidelity (every Decimal re-serializes byte-identically to its source token with a digit/comma border check — the thousands-separator-truncation kill switch — plus sum(line amounts) == total, proven against both Total blocks); bullet-metadata label discipline (closed label set, assigned by leading label, never ordinal — immune to the optional Part Number segment); USPS address shape on the raw pre-enrichment value; and rejection of any unparseable sentinel or residual \r/\xa0 artifact. Multi-line-item (0.18%) and non-USD (0 observed) new-POs, comment emails, and anything else falls back to the AI extractor. The AI path is gated too: the untrusted email reaches Bedrock inside a neutralized <email> data block (forged tag lookalikes in the body are defanged) with temperature=0, and the raw model output must pass the fail-closed validate_ai_fallback() gate — the PO-specific nested contract (exact key-set at every level, with missing keys normalized rather than rejected), a po_number shape check hardened against fullwidth-digit and trailing-artifact injection (the same regex family protecting the DynamoDB partition key the handler builds from it), an email_type allow-list enforced before dispatch so a miss can never fall into the new_po default branch, and Decimal/int/None money typing (PO decodes with parse_float=Decimal) — before any DynamoDB write. The template path is gate-enforced end-to-end; the AI-fallback path is validated and fail-closed — output that fails the gate is skipped, never written (see ai_fallback_rejected below), so a malformed or injected email raises the fallback rate rather than corrupting a record. Every record emits one CloudWatch EMF metric (see below).

Derived fields (site_code, trade, fiscal_year). These three are classified, not verbatim-extracted, so neither parse path emits them directly — instead a deterministic Python classifier (derived_fields.derive_all()) computes them inside the shared enrich_parsed() post-stage, which runs identically on both paths. derived_fields.py is byte-frozen this phase (shadow-bake freeze, per the constraint-9 module-diff gate); note for accuracy that its in-file docstring's "FAITHFUL v1 port" self-description is aspirational, not descriptive — the module is a deliberate spec-superset of the original EXTRACTION_PROMPT rule text (its own inline backtest-tuning comments diverge from those three prompt sections), with the LLM staying runtime-authoritative on the AI-fallback path during the bake regardless. The docstring reword is deferred to the first post-bake PR, since even a text-only edit is barred by this phase's machine-checked empty-diff gate on the file. trade is a closed label set (the same one the AI prompt enumerates): Plumbing - PM, Plumbing - Reactive, Electrical, HVAC, Dock Doors, Doors, Signage, Carpentry, Fencing/Gates, Conveyance/MHE, Painting, Flooring, Janitorial, Fire/Life Safety, Landscaping/Yard, Roofing, Security/Locksmith, Snow Removal, PO Uplift, General Building - Emergency, General Building - Handyman, General Building - Project, General Building. Authority differs by path:

• Template path — the parser leaves all three null, so Python is authoritative: it fills them.
• AI-fallback path — the LLM value stays authoritative during the bake (Python fills only a gap the LLM left null, never overwrites), and Python additionally runs in shadow mode: enrich_parsed() emits one DerivedFieldAgreement EMF record per field comparing the two.

DerivedFieldAgreement metric — namespace Seahaven/PoIngest, metric name DerivedFieldAgreement (Unit Count, value 1), dimensioned only by Field × Agreement (cardinality fixed at 3 × 4). Agreement ∈ {agree (both non-null, equal after str-strip), disagree (both non-null, different), llm_null_python_filled (LLM null, Python supplied a value), python_null (LLM non-null, Python null)}; the record is skipped entirely when both are null. po_number, PythonValue, and LlmValue ride along as non-dimensioned Logs-Insights properties so a disagreement can be reviewed by example without inflating cardinality. Only the AI-fallback path emits these — the template path has no LLM value to shadow.

Review disagreements during the bake with this CloudWatch Logs Insights query over /aws/lambda/po-email-processor:

fields @timestamp, Field, Agreement, LlmValue, PythonValue, po_number
| filter ispresent(DerivedFieldAgreement)
| filter Agreement = "disagree"
| sort @timestamp desc
| limit 200

Or aggregate overall agreement per field: | filter ispresent(DerivedFieldAgreement) | stats count(*) by Field, Agreement.

Post-bake follow-up: once agreement is acceptable, make Python authoritative on the AI-fallback path too (stop keeping the LLM value) and drop the site_code/trade/fiscal_year rule sections from EXTRACTION_PROMPT — the LLM stays authoritative on the fallback path only until then.

Lambdas (lambdas/po/):

Function	Trigger	Purpose
po-email-processor	S3 ObjectCreated	Claude extraction + DynamoDB write
po-ingest-site-extractor	DynamoDB Streams	Site code/address extraction -> verified-sites
po-web-ui	Manual invoke (authenticated — see Setup §6)	HTML dashboard (public Function URL removed 2026-06-08, INFRA-74)

Tables:

• purchase-orders (PK: po_number, Streams: NEW_IMAGE) — read by procurement-api (see Shared Resources; the former seahaven-slack-bot reader was decommissioned 2026-07-23)
• verified-sites (PK: siteCode) — ~1,100 unique Amazon facility sites (by-state GSI removed 2026-06-03, audit M-20)
• pending-site-review (PK: po_number) — unresolvable POs for manual Payee Central verification

Work Orders (WorkorderIngestStack stack)

Amazon APM work order emails (from Hexagon EAM / HxGN SmartCloud) are received at apm@int.seahaven.com, parsed deterministic-template-first with a Claude-on-Bedrock fallback, and written to the WorkOrders DynamoDB table.

Flow:

1. Hexagon EAM sends email notifications (new assignments, comments, updates, cancellations) to amazon@seahavenind.com.
2. Gmail filter forwards APM emails to apm@int.seahaven.com (SES).
3. SES drops the raw MIME into s3://workorder-ingest-emails-{AccountId}/inbound/.
4. S3 triggers the workorder-email-processor Lambda.
5. Fail-closed sender authentication (INFRA-107): the SES-stamped Authentication-Results header must show dkim=pass for the domain that re-signs the forward (currently allowlisted as seahaven.com — see the validation caveat under Sender authentication); otherwise the email is logged and dropped.
6. Parse: a pure, offline template parser (template_parser.py) tries the two known Hexagon templates first, behind a strict fail-closed validation gate. Only on a miss/invalid result does the Lambda fall back to the Claude-on-Bedrock AI extractor. Both paths emit the identical structured-JSON contract (work order ID, site code, severity, priority, dates, assigned technician).
7. Work order upserted to WorkOrders, event/comment appended to WorkOrderComments.

Deterministic template parser. ~93.6% of WO traffic is the plain-text "AMAZON UPDATE WO DETAILS <id>" comment template (T1) and ~6.4% is the HTML "AMAZON assign Work Order <id> on building <SITE>" assignment template (T2). template_parser.try_deterministic_parse() classifies by subject, extracts the contract fields, and returns a parsed result only if it passes a strict validation gate (exact contract-key set; work_order_id matches the subject and is all-digits; the T1 Work Order: <id> double-space is literally present; email_type matches the template; site_code shape; per-type required fields; and a label-bleed guard so a value that over-ran into another field fails). Anything that fails — the rare update/cancellation shapes, Hexagon template drift, or an extractor exception — falls back to the AI extractor. The AI path is gated too: the untrusted email reaches Bedrock inside a neutralized <email> data block (tag lookalikes in the body are defanged), and the raw model output must pass the fail-closed validate_ai_fallback() schema/enum/date gate before any DynamoDB write — output that fails is dropped and paged (see the ai-fallback-rejected alarm below), a prompt-injection defence for DKIM-passing but attacker-influenced mail. Data is never corrupted; only the fallback rate rises. Every record emits one CloudWatch EMF metric (see below).

Lambdas (lambdas/wo/):

Function	Trigger	Purpose
workorder-email-processor	S3 ObjectCreated	Claude extraction + DynamoDB write
workorder-web-ui	Manual invoke (authenticated — see Setup §6)	HTML dashboard (public Function URL removed 2026-06-08, INFRA-74)
workorder-shoc-emitter	DynamoDB Streams, both WO tables (ESMs ship disabled — see SHOC webhook feed)	HMAC-signed webhook push of every WO mutation to the SHOC backend
workorder-shoc-hmac-rotator	Secrets Manager rotation schedule (30 days)	Rotates the webhook HMAC signing keys (dual-key overlap)

Tables:

• WorkOrders (PK: work_order_id, Streams: NEW_AND_OLD_IMAGES) — site-code-index and status-index GSIs removed 2026-06-03 (audit M-20)
• WorkOrderComments (PK: work_order_id, SK: comment_id, Streams: NEW_AND_OLD_IMAGES) — see the comment_id format note below

SHOC webhook feed (workorder-shoc-emitter)

Flow: WorkOrders / WorkOrderComments DynamoDB Streams (NEW_AND_OLD_IMAGES — the OLD image is what lets the emitter detect the wo_status → cancelled transition) → workorder-shoc-emitter → HMAC-signed HTTPS POST → SHOC backend. Every WO mutation becomes one webhook event (work_order.created / .updated / .cancelled / .comment_added) within seconds of the DynamoDB commit; DynamoDB stays the source of truth.

• Contract: docs/shoc-webhook-contract.md (Rev 2026-07-23) is the producer/consumer contract SHOC builds its receiver against, and docs/shoc-webhook-test-vectors.json is the shared receiver-verification vector set — both sides pin their HMAC implementation against the same vectors (producer-side via the golden-vector tests over delivery.sign_body).
• ACTIVE since 2026-07-30. Both event-source mappings run with enabled=True, flipped after the SHOC receiver on api.dev.seahaven.com passed the shared test vectors live (valid current-kid signature accepted, duplicate delivery_id deduplicated, tampered/stale/unknown-kid all rejected 401). The stack originally shipped dark (enabled=False) so it could deploy and be tested with zero deliveries while SHOC had no receiver. The ESMs start at LATEST — no historical flood; SHOC backfills history through the procurement-api read API, not the stream.
• Ordering/retry semantics. parallelization_factor=1, bisect_batch_on_error=False, retry_attempts=-1, maximum_record_age=24h: a retryable failure (429/5xx/timeout/connection error) blocks the shard and retries from the failed record — per-work-order commit order is the guarantee, and blocking is the intended behavior when SHOC is down. report_batch_item_failures keeps earlier in-batch successes from being re-delivered. Records that exhaust the 24h age are parked as ESM failure metadata (not full records) on workorder-shoc-emitter-failures (on_failure destination); a non-retryable 4xx (a contract bug, never worth blocking the shard for 24h) parks the full {envelope, response_status} payload on workorder-shoc-emitter-rejected and the loop continues. Both queues: 14-day retention, SSL-enforced, alarmed (see CloudWatch alarms); recovery is scripts/replay_shoc_webhooks.py (see Scripts).
• Secret + KMS. The HMAC signing keys live in Secrets Manager secret workorder-ingest/shoc-webhook-hmac (value {"keys": [{"kid", "secret"}, ...]}, newest first, max 2), encrypted with the dedicated CMK workorder-ingest-shoc-webhook-kms — deliberately not alias/seahaven-dynamodb, so the SHOC cross-account grant's decrypt reach covers exactly this one secret and nothing else. The secret's removal policy is DESTROY, deliberately not RETAIN: the value is machine-generated HMAC material, fully regenerable by a single rotation, so RETAIN buys nothing and would expose the fixed-name RETAIN-orphan deadlock (a failed create orphans an empty shell holding the global name; every later create fails AlreadyExists). Accidental-deletion recovery runbook: redeploy to recreate the secret, force a rotation (aws secretsmanager rotate-secret --secret-id workorder-ingest/shoc-webhook-hmac), notify the SHOC team — receivers re-fetch within their ≤5-minute cache TTL, so no coordination window is needed — then watch the -failures queue and replay the gap with the replay script.
• Rotation. workorder-shoc-hmac-rotator runs on a 30-day schedule: it prepends a fresh 64-hex-char key as keys[0] and truncates the list to 2 entries (one overlap cycle). kid format is YYYY-MM-DDTHH. The emitter always signs with keys[0] behind a 5-minute TTL cache; the receiver accepts any listed kid and re-fetches on an unknown one — there is no delivery window in which signatures can't verify.
• Cross-account grants (exact ARN only): arn:aws:iam::396287094661:role/shoc-backend-dev is granted secretsmanager:GetSecretValue on the secret's resource policy and kms:Decrypt on the CMK's key policy — both halves are required; either one alone fails silently at the receiver. Future staging/prod receiver roles are each a deliberate, individually-reviewed policy addition — no wildcard/prefix trust.

Procurement API (procurement-api stack)

A read-only REST API (API Gateway + the procurement-api Lambda, lambdas/api/) over both pipelines' tables, plus a token-gated OpenAPI docs page. Primary consumer: the SHOC backend, for which this replaces the retired SyncController cross-account DynamoDB scan as the reconciliation/backfill path (and the initial-history load for the outbound work-order webhook, which starts at LATEST).

Method + path	Auth	Backing read
GET /work-orders · /purchase-orders · /verified-sites	IAM SigV4	unordered paginated Scan (limit 1–500, opaque cursor = base64 LastEvaluatedKey; malformed cursor → 400)
GET /work-orders/{id} · /purchase-orders/{id} · /verified-sites/{siteCode}	IAM SigV4	GetItem (404 on miss)
GET /work-orders/{id}/comments	IAM SigV4	Query on the partition key, paginated
POST /work-orders/{id}/comments, PATCH /work-orders/{id}	—	phase-2 planned (x-planned in the spec); handler answers 501
GET /docs, GET /openapi.json	shared docs token (X-Auth-Token header or ?token= in a browser)	Redoc reference docs (vendored offline, no CDN) with the spec inlined / the committed spec

• Spec is source of truth: lambdas/api/openapi.json (OpenAPI 3.1). Its top-level webhooks section documents the outbound SHOC work-order feed, so one page describes both directions (call + be-called). tests/test_api_spec_drift.py pins the spec's paths to the router table, so spec and implementation cannot drift.
• Auth: data routes use API Gateway AWS_IAM (SigV4) plus a resource policy allowing exactly arn:aws:iam::396287094661:role/shoc-backend-dev on GET/*; same-account admin callers authorize via identity policy (Postman signs SigV4 natively). Docs routes are auth NONE at the gateway (resource-policy carve-out for exactly those two GETs) but the handler fails closed on the shared token (lambdas/shared/web_ui_auth.py, secret procurement-ingest/web-ui-auth-token) — not an unauthenticated data path (INFRA-74 posture).
• Custom domain: https://procurement-api.seahaven.com (REGIONAL API Gateway domain, TLS 1.2, empty base-path mapping to the prod stage, so callers hit /work-orders with no /prod segment). The stable SHOC-facing endpoint; the raw *.execute-api.us-east-1.amazonaws.com/prod URL still works. Cross-account DNS: the seahaven.com public zone is in the mgmt account (328440206208), so the ACM cert's validation record and the A-alias are added there out of band — scripts/setup_procurement_api_domain.sh cert issues the cert (DNS-validated against the mgmt zone) and writes its ARN to prod SSM /procurement-api/custom-domain/certificate-arn, which Terraform reads (data.aws_ssm_parameter in terraform/api.tf); after an HCP apply that owns the API Gateway DomainName, scripts/setup_procurement_api_domain.sh alias adds the A-alias from API Gateway get-domain-name (regionalDomainName / regionalHostedZoneId). SigV4 is unaffected (same underlying API id + resource policy); the docs page injects whichever host served the request into servers[0].url.
• KMS: purchase-orders is CMK-encrypted; the imported-by-name table doesn't carry the key association, so the stack grants kms:Decrypt/DescribeKey on the CMK from SSM /seahaven/dynamodb/cmk-arn explicitly (the INFRA-104 failure class).
• No access logging in v1 (keeps the ?token= shim out of any log and avoids the account-level API Gateway CloudWatch role); rotate the docs token before ever enabling it. No CORS (server-to-server + Postman callers).
• Alarms: procurement-api-errors/-throttles/-duration (p99 ≥ 22.5 s) + gateway procurement-api-5xx, all ALARM-only → site-alerts. No 4XX alarm (401/403/404 are expected traffic).
• Docs page: /docs serves Redoc (read-only reference docs; live calls go through Postman since data routes need SigV4) as ONE token-gated response: the handler inlines the vendored redoc.standalone.js, the design-system fonts (fonts.css, data-URI @font-face — nothing may fetch from Google Fonts, test-pinned), and the spec into docs.html, with </script>/</style> breakout guards on every blob. Theme = SHOC tokens (Montserrat/DM Sans/JetBrains Mono, primary #1c75bc, 64px gradient topbar). The right-panel gradient and the topbar's Hide/Show-samples toggle target styled-components class names that are deterministic for the pinned Redoc bundle but change on any bump — re-derive them then (headless probe: elements whose computed background equals the rightPanel color); stale selectors degrade to a solid panel / inert toggle, cosmetic only. Tooling: npm run lint:api lints the spec against redocly.yaml (CI job spec-lint; deliberate exceptions live in .redocly.lint-ignore.yaml), npm run docs:preview renders the real handler output locally.

Architecture

IaC: Terraform under terraform/ (HCP remote apply, Manual until sealed). Workspace procurement-ingest-prod in project seahaven-prod is the sole mutate path for live resources in seahaven-prod 011934824531 / us-east-1 (PLAT-86 import-in-place). Former CDK stacks (po-ingest, WorkorderIngestStack, procurement-api) are historical and removed from the tree (PLAT-89). Never apply against mgmt 328440206208 — PLAT-67 left cold-archive RETAIN leftovers there only.

All Lambdas: Python 3.12, ARM64, 60-day log retention.

LLM provider — Amazon Bedrock. Both email processors call Claude Haiku 4.5 through the Bedrock inference profile us.anthropic.claude-haiku-4-5-20251001-v1:0 (bedrock-runtime InvokeModel), configured via the BEDROCK_MODEL_ID env var. There is no Anthropic API key and no Secrets Manager secret any more. Each processor role is granted bedrock:InvokeModel + bedrock:InvokeModelWithResponseStream on both the inference-profile ARN and the per-region foundation-model ARNs for us-east-1 / us-east-2 / us-west-2 (empty-account foundation-model ARNs) — the us.* profile can route cross-region, so a profile-only grant would AccessDenied at runtime under load.

Retired secrets (manual cleanup outstanding): the former secrets po-ingest/anthropic-api-key and workorder-ingest/anthropic-api-key were retained when removed from IaC, so they were orphaned rather than deleted. Delete both by hand and revoke the stored keys at the provider. The processors no longer read any ANTHROPIC_API_KEY_SECRET_ARN — authentication to Bedrock is via the Lambda execution-role IAM grant, so there is no provider API key or Secrets Manager fetch on the parse path.

SES: Both pipelines add rules to the shared INBOUND_MAIL receipt rule set on int.seahaven.com.

Shared alarms and packaging helpers

DynamoDB alarms, the sender-auth-rejected metric filter + alarm, the standard per-Lambda alarm set, the Bedrock InvokeModel grant, the raw-email bucket, the async-invoke DLQ, and the template-fallback-rate math alarm live under terraform/ (po_*.tf, wo_*.tf, api*.tf). Lambda zip packaging is terraform/build_packages.sh (non-recursive copy_py_dir + copy_shared_all / selective copy_src_file), gated by tests/test_bundle_consistency.py.

Per-function alarm variance is preserved: PO's email-processor duration alarm uses p99, WO's uses p95; po-web-ui gets throttles + duration only (no errors, no DLQ); po-ingest-site-extractor has no DLQ alarm (it's a DynamoDB-stream consumer, not async-invoked); workorder-web-ui gets zero alarms.

Sender authentication (INFRA-107)

The From header and any Authentication-Results header inside the raw MIME are attacker-forgeable, so neither is trusted. Instead, both email processors authenticate the sender against the verdicts SES itself stamps at delivery time, failing closed. The authenticator is single-sourced at lambdas/shared/ses_auth.py (Phase 3) — previously duplicated byte-for-byte in each pipeline's email_processor/ dir and kept in sync by a fixture-hygiene test; now one copy, so a future hardening fix to the fail-closed logic lands once instead of needing two identical edits. Packaging copies it flat beside each handler (copy_shared_all in terraform/build_packages.sh) so the handlers' unchanged from ses_auth import authenticate_inbound_email still resolves at runtime (see lambdas/shared/). The gate:

1. Take only the topmost Authentication-Results header (SES prepends its trace headers; any lower copies arrived inside the message and are ignored).
2. Require its authserv-id to be amazonses.com.
3. Require a dkim=pass clause whose header.d=/header.i= domain is in the pipeline's allowlist.

The allowlist is the ALLOWED_DKIM_DOMAINS Lambda environment variable (comma-separated, set per pipeline in Terraform — no code change needed to adjust):

Pipeline	ALLOWED_DKIM_DOMAINS	Why
Work orders	seahaven.com	APM mail reaches apm@int.seahaven.com via a forward off amazon@seahavenind.com; the allowlist trusts the domain that re-signs DKIM on that forward (the original hxgnsmartcloud.com signature does not survive it). Validated against live SES-stamped headers (2026-07-16) — real APM deliveries carry dkim=pass header.i=@seahaven.com. Note a plain Gmail auto-forward re-signs under the sending Workspace domain (seahavenind.com / a *.gappssmtp.com key), not seahaven.com; only a Google Group (or Workspace routing) with "sign as seahaven.com" produces dkim=pass header.i=@seahaven.com. If the observed re-signing domain differs, update this value (do not widen it to a shared key like *.gappssmtp.com, which any Google customer's mail would pass). The workorder-email-processor-sender-auth-rejected alarm pages if this assumption is wrong instead of silently dropping every work order.
Purchase orders	amazon.coupahost.com	Coupa signs as amazon.coupahost.com. amazonses.com also passes but is deliberately not allowlisted — every SES customer's mail passes for it

Clause-injection hardening: SES echoes attacker-controlled SMTP-session tokens (envelope-from, helo, header.from) into its own Authentication-Results value, and an RFC 5321 quoted-local-part MAIL FROM may legally contain ; and spaces. The parser therefore tokenises comment- and quoted-string-aware (RFC 8601 / RFC 5322): CFWS comments (...) are stripped and clauses are split only on semicolons outside a quoted string, so a ; inside a quoted envelope-from= value can never be torn into a forged dkim=pass clause. The DKIM signer domain is read from header.d= when present (falling back to header.i=, taking the domain after the AUID's last top-level @ so a quoted local-part cannot smuggle an allowlisted domain). Two latent comment-parsing edge cases (early comment-close, no-separator-on-strip) are tracked as hardening follow-ups — see the SES-AR-01/02 issue; neither is reachable through SES's real header encoding today.

Risk acceptance — forwarder-domain binding (INFRA-107, accepted 2026-07-16): for work orders this control authenticates the domain that re-signs the apm@ forward (seahaven.com), not the Hexagon originator (hxgnsmartcloud.com, whose signature does not survive the forward). Its strength therefore rests on the apm@ Google Group's posting policy being restricted to trusted internal senders — that restriction is the load-bearing control and is accepted as documented risk. If the apm@ group is ever opened to external posting, this finding escalates to HIGH (anyone able to post to the group could inject a forged work order) and the correct fix is to bind acceptance to the originator via DMARC alignment rather than the forwarder's re-signature. The PO pipeline is unaffected — amazon.coupahost.com is an external domain an attacker cannot get SES to sign.

On any failure (env var unset, header missing/unparseable, verdict fail, unaligned domain) the processor logs a structured sender_auth_rejected warning with the reason and S3 key, skips the email, and returns normally — rejected mail never triggers Lambda retries or DLQ messages, but the <fn>-sender-auth-rejected CloudWatch alarm (see CloudWatch alarms) pages on a rejection spike so a drift-induced outage is not silent. Unit tests live in tests/test_ses_auth.py.

Failure handling (INFRA-41): Each email-processor is async-invoked (S3 → Lambda). Both have a Terraform-managed SQS dead-letter queue (14-day retention, SSL-enforced) so a failed parse is captured rather than silently dropped after Lambda's retries.

CloudWatch alarms

Every alarm is ALARM-only (no OK action), sends to the shared site-alerts SNS topic, and uses treat-missing-data not-breaching.

Lambda alarms (AWS/Lambda, FunctionName dimension):

Alarm	Functions	Metric / config
<fn>-errors	po-email-processor, po-ingest-site-extractor, workorder-email-processor, workorder-shoc-emitter, workorder-shoc-hmac-rotator	Errors Sum, 5 min, > 0, eval 1
<fn>-throttles	po-email-processor, po-ingest-site-extractor, po-web-ui, workorder-email-processor, workorder-shoc-emitter, workorder-shoc-hmac-rotator	Throttles Sum, 5 min, > 0, eval 1
<fn>-duration	po-email-processor, po-ingest-site-extractor, po-web-ui, workorder-shoc-emitter, workorder-shoc-hmac-rotator (p99); workorder-email-processor (p95)	Duration percentile, 5 min, >= 45000 ms (75% of the 60s timeout), eval 3 / datapoints 2
<fn>-sender-auth-rejected	po-email-processor, workorder-email-processor	Log-metric-filter count (namespace Seahaven/ProcurementIngest, default_value=0) on sender_auth_rejected warnings, Sum 5 min, >= 1, eval 3 / datapoints 2

The <fn>-sender-auth-rejected alarm closes the silent-drop gap in INFRA-107: a rejected email returns normally (no error, no retry, no DLQ message), so without a log-metric filter a signing-domain drift or a wrong allowlist would discard 100% of legitimate mail while every other alarm stayed green. It counts sender_auth_rejected warnings per 5-minute period (default_value=0 keeps the series continuous) and pages when 2 of the last 3 periods each see at least one rejection — a lone stray spoof probe to the internal ingest address self-clears, but a sustained false-reject storm pages within ~10–15 minutes even at low mail volume; the config is easy to tune in Terraform. (A residual gap remains for a very sparse total-reject outage — see the SES-AR-01/02 hardening issue.)

The <fn>-duration and <fn>-throttles alarms for po-email-processor and workorder-email-processor supersede the orphaned, CLI-created Lambda-Duration-* / Lambda-Throttles-* alarms (deleted post-deploy).

DLQ alarms (AWS/SQS): po-email-processor-dlq-messages and workorder-email-processor-dlq-messages fire when any message is visible on an email-processor DLQ (ApproximateNumberOfMessagesVisible Maximum, 5 min, > 0, eval 1) — a message there means an email was dropped after Lambda exhausted its async retries. Recovery from a DLQ message (no console redrive) is documented in the DLQ recovery runbook.

SHOC emitter alarms (bespoke — these don't fit the standard-Lambda-alarm helper's shape): workorder-shoc-emitter-iterator-age (IteratorAge Maximum, 5 min, >= 600000 ms, eval 3 / datapoints 2) fires when the stream lags ≥ 10 minutes — SHOC is likely down and the shard is blocking on retries, which is exactly the ordered-backpressure design working, but an operator should know. workorder-shoc-emitter-failures-messages and workorder-shoc-emitter-rejected-messages (ApproximateNumberOfMessagesVisible Maximum, 5 min, > 0, eval 1) page the replay runbook: the -failures queue receives ESM failure metadata for retry-exhausted records, the -rejected queue receives the full parked payloads of non-retryable 4xx deliveries (see the SHOC webhook feed section and scripts/replay_shoc_webhooks.py). All three: ALARM-only → site-alerts, NOT_BREACHING, per the house rules above.

Parse-outcome metric + fallback-rate alarm (workorder-ingest): the WO processor writes one CloudWatch EMF line per email to namespace Seahaven/WorkorderIngest, metric ParseOutcome (Unit Count, value 1), dimensioned by ParseMethod (template | ai_fallback | ai_fallback_rejected) and TemplateId (update_plaintext | assign_html | unknown). ai_fallback_rejected counts AI-fallback output that failed the fail-closed validate_ai_fallback() gate (schema/enum/date contract on raw Bedrock output — prompt-injection defence) and was dropped without a DynamoDB write. Non-dimension EMF properties ReasonCode and work_order_id are queryable in Logs Insights but not promoted to metrics (kept low-cardinality). EMF is used instead of PutMetricData so there is no extra sync call / latency / IAM grant on the async hot path (the role already has logs:PutLogEvents). The alarm workorder-email-processor-template-fallback-rate fires when the AI-fallback share of parses — rejected fallback parses included, so a drift outage whose AI output also fails the gate cannot lower the observed rate while dropping mail — exceeds 15% sustained (a MathExpression with FILL(...,0) and a ≥10-sample volume floor over 15-minute periods, eval 3 / datapoints 2) — catching Hexagon template-drift coverage collapse while the volume floor + FILL prevent low-volume false pages / INSUFFICIENT_DATA. ALARM-only SnsAction to site-alerts, no OK action, NOT_BREACHING. The 15-minute period is a deliberate deviation from the 5-minute house style to accumulate a stable denominator at the low ~760/day volume. A second alarm, workorder-email-processor-ai-fallback-rejected, pages on the rejected series itself (≥1 rejection per 5-min period, 2 of the last 6 periods — the sender-auth-rejected sparse-arrival idiom) because a gate rejection drops mail without error/retry/DLQ and would otherwise be silent.

WO Bedrock transport-error metric (Phase 8). A Bedrock-side transport error (throttling, malformed response, non-JSON model text) during the AI-fallback attempt previously emitted zero ParseOutcome datapoints — the only emit sites were post-gate. handler.py now wraps the extract_with_bedrock call in a try/except that emits exactly one ParseMethod=ai_fallback / ReasonCode=bedrock_error datapoint and then re-raises (the exception still propagates into the errors alarm / DLQ path unchanged). This is an except-and-reraise, not a reorder: a gate-rejected email (Bedrock returns successfully, validate_ai_fallback() then rejects it) still emits only the single ai_fallback_rejected datapoint and nothing else — the except branch never fires because Bedrock did not raise — so the workorder-email-processor-ai-fallback-rejected "a rejected email emits nothing else" alarm contract holds with no double-count.

Parse-outcome metric + fallback-rate alarm (po-ingest): the PO processor emits the same EMF shape to namespace Seahaven/PoIngest, metric ParseOutcome, dimensioned by ParseMethod (template | ai_fallback | ai_fallback_rejected) and TemplateId (coupa_new_po | coupa_cancellation | unknown), with ReasonCode (the fail-closed gate reason) and po_number as Logs-Insights ride-alongs. ai_fallback_rejected counts AI-fallback output that failed the fail-closed validate_ai_fallback() gate (nested key-set contract, po_number shape, email_type allow-list, Decimal money typing) and was dropped without a DynamoDB write. The ai_fallback_rejected emission's po_number ride-along is clamped to 64 chars (handler.py:118, pinned by the PO-DC-02 regression test) — telemetry.py's DerivedFieldAgreement PythonValue/LlmValue properties clamp the same way.

Deliberate double-count: unlike WO, PO emits ParseMethod=ai_fallback before the Bedrock call (so a Bedrock-side error still records the outcome) — a rejected email therefore always emits both an ai_fallback datapoint (pre-call) and an ai_fallback_rejected datapoint (post-gate), never just the latter. This is intentional and load-bearing, not a bug; the fallback-rate math below treats fb as already inclusive of every rejection.

The alarm po-email-processor-template-fallback-rate is deliberately retuned for PO volume — do NOT copy the WO numbers: at ~57 emails/day a 15-minute period holds ~0.6 emails, so the WO ≥10-sample floor would never be met and the alarm would be structurally dead. Instead: 6-hour periods (~14.25 expected emails each), an IF((fb+tmpl)>=8, …) volume floor (at the floor a single fallback email is 12.5% < the threshold, so one email can never breach a datapoint; a breach needs ≥2 fallbacks in one window, or ≥3 at typical volume), threshold >20% (expected baseline fallback ≈1%: comments 0.55% + multi-line 0.18% + non-USD 0), eval 4 / datapoints 2 (a 24h span — isolated noise self-clears while total template drift at 100% fallback pages within ~12h). Sparse overnight/weekend windows below the floor evaluate to 0 (non-breaching by design; accepted trade: a Friday-evening drift may not page until weekend volume accrues). The ai_fallback_rejected series (rej) is deliberately excluded from this expression's numerator, denominator, and volume floor: because the pre-call emit already counts every rejected email once inside fb, folding WO's fb+rej math in verbatim would double-count each rejection in both terms and inflate the observed rate toward 100% — fb/(fb+tmpl) alone is already exact for PO. Same idiom otherwise: ALARM-only SnsAction to site-alerts, NOT_BREACHING, and no element-wise MAX in the math expression (the post-#102 rule — the IF floor guarantees the non-zero denominator).

A second alarm, po-email-processor-ai-fallback-rejected, monitors the rejected series on its own — retuned for ~57 emails/day, not WO's 5-minute sparse idiom (which needs two rejections inside one 30-minute window and would be structurally dead at PO volume). It uses the same 6h/IF-floor/eval-4/datapoints-2 idiom as the fallback-rate alarm above, but as a plain count-floor on the rejected series itself (IF(FILL(rej,0)>=1, …), no denominator so no divide guard is needed): threshold ≥1, over 6-hour periods, eval 4 / datapoints 2 — a lone stray rejection self-clears, while ≥2 rejections landing in ≥2 distinct 6h windows within 24h (sustained prompt-injection probing, or template drift whose AI output also fails the gate) pages within ~12–24h. ALARM-only SnsAction to site-alerts, NOT_BREACHING. Accepted residual: a single isolated rejected email never pages this alarm by itself — it is still visible as an ai_fallback_rejected datapoint and in the ReasonCode log line, and it has already raised the fallback-rate numerator above via its pre-call ai_fallback emit.

DynamoDB alarms (AWS/DynamoDB): each owned table gets <table>-throttles (ThrottledRequests) and <table>-system-errors (SystemErrors). These metrics emit only at the TableName + Operation dimension set, so each alarm is a Sum math expression across the operations the table uses (Get/BatchGet/Query/Scan/Put/Update/Delete/BatchWrite). Tables covered: purchase-orders, verified-sites, pending-site-review (po-ingest); WorkOrders, WorkOrderComments (workorder-ingest).

Deploy-Pipeline Guards (Phase 0)

Goal: a broken Lambda bundle fails the deploy job, not Monday's first email.

Healthcheck direct-invoke contract. Both po-email-processor and workorder-email-processor recognize a top-level direct-invoke probe payload {"healthcheck": true}. In each handler(event, context), the very first statements — before any S3 fetch, before ses_auth, before iterating event["Records"] — are:

if isinstance(event, dict) and event.get("healthcheck") is True:
    return {"healthcheck": "ok"}

This placement is deliberate, not incidental: real mail always arrives as an S3 ObjectCreated event whose top-level keys (Records) AWS controls, so email content can never set a top-level healthcheck key — the branch creates no accept path for forged mail. It also emits no EMF metric and no log line, so it can never match the sender_auth_rejected log-metric-filter pattern that feeds the <fn>-sender-auth-rejected alarm (see CloudWatch alarms) — that alarm pages at ≥1 match in its window, so repeated healthcheck invokes across deploys (two deploys in ~30 min is routine) must never contribute to it. Unit coverage: lambdas/po/email_processor/tests/test_po_healthcheck.py and lambdas/wo/email_processor/tests/test_healthcheck.py.

Post-apply smoke gate. scripts/post-deploy-smoke.sh runs after HCP applies (manual or gated) against seahaven-prod. It invokes po-email-processor, workorder-email-processor, and procurement-api with aws lambda invoke --invocation-type RequestResponse --payload '{"healthcheck": true}' (region us-east-1) and asserts, per function:

1. The invoke response's FunctionError field is absent — this is the load-bearing check. A broken bundle (e.g. an ImportError at module init from a missing sibling module) still returns HTTP 200 from the Lambda Invoke API with FunctionError=Unhandled; a bare exit-code check on aws lambda invoke would false-pass on exactly the failure this gate exists to catch.
2. The returned payload is exactly {"healthcheck": "ok"}.

The script runs set -euo pipefail and exits non-zero on any invoke failure, any FunctionError, or a payload mismatch, failing the apply gate.

Lambda packaging (terraform/build_packages.sh). HCP plan/apply packages each function via non-recursive copy_py_dir (top-level *.py only, so tests/ is excluded) plus copy_shared_all for the email processors and selective copy_src_file for web UI auth / API static assets. That eliminates the failure mode that shipped a broken bundle twice (PR #105 omitted template_parser.py; PR #2 nearly omitted derived_fields.py): a new sibling module the handler imports ships automatically when added under the pipeline dir, and tests/test_bundle_consistency.py pins the exact-set + ships-all contracts against the packaging script (no Terraform plan required). Pip installs for third-party deps still use --platform manylinux2014_aarch64 --only-binary=:all: — removing that pin once shipped x86 wheels into the ARM64 function and caused a total outage (PR #34).

Phase 3: shared module extraction (lambdas/shared/)

Four first-party modules that were previously duplicated per pipeline (or inlined in each handler) are now single-sourced under lambdas/shared/, following the handbook's lambdas/shared/ convention:

Module	What it is	Imported by
ses_auth.py	fail-closed SES sender-authentication (INFRA-107)	both email processors
web_ui_auth.py	fail-closed X-Auth-Token gate + token cache (INFRA-74)	both web_ui handlers
email_parsing.py	parse_raw_email (the WO superset that returns cc unconditionally; PO simply ignores cc)	both email processors
emf.py	generic CloudWatch EMF emitter (emit_metric, emit_parse_outcome) parameterized by namespace / dimension-sets / properties	both email processors (PO also uses emit_metric for DerivedFieldAgreement)

Flat-landing import rule. The shared dir has no __init__.py — the modules are consumed by bare name (from ses_auth import ..., from emf import emit_parse_outcome), exactly as when they were siblings. This works because packaging lands them flat in the zip root beside handler.py, so at runtime each shared module sits on the function's own sys.path under its bare name — the handler import lines are unchanged, which is what keeps the byte-identical fail-closed ses_auth behavior through the move. The load-bearing emf dimension-set list [["ParseMethod"], ["ParseMethod", "TemplateId"]] is now pinned once in emf.py (one-sided dimension drift between the two pipelines becomes structurally impossible), while the deliberate per-pipeline emission-ordering differences stay in the handlers (PO emits ai_fallback before the Bedrock call with an intentional double-count; WO emits mutually-exclusive ai_fallback/ai_fallback_rejected after its gate).

Packaging — email processors. Both email-processor packages run copy_py_dir for the pipeline dir plus copy_shared_all. This ships all four shared modules flat into each email-processor zip. web_ui_auth.py therefore rides along into both email-processor bundles even though the email handlers never import it — a harmless, deliberate consequence of copying all of shared/ (PO_EXPECTED_TOP_LEVEL_MODULES and the bundle-parity expectations account for it).

Packaging — web UIs. Both po-web-ui and workorder-web-ui copy their own dir plus only shared/web_ui_auth.py (not all of shared/) — the web UIs need only the auth module. site_extractor packages its own dir only.

tests/test_bundle_consistency.py is updated in lockstep without losing teeth: _first_party_sibling_imports resolves shared-sourced imports under lambdas/shared/; packaging recipes are parsed from terraform/build_packages.sh; ships-all accumulates across copy_py_dir + copy_shared_all / copy_src_file; and pins require shared / web_ui_auth copies to be actually present (a commented-out or removed copy fails CI).

Phase 5: handler decomposition + lazy boto3 clients

Both email-processor God-handlers are decomposed along the seams that already work into flat sibling modules in the same directory (bare-name imports, exactly like the ses_auth/template_parser/derived_fields pattern), so the packaging copy_py_dir ships every new sibling automatically — no packaging change beyond the exact-set pin. Every move is a pure delete-here/add-there; derived_fields.py, both template_parser.py, the validate_ai_fallback gates, and lambdas/shared/ are byte-untouched.

PO (handler.py → 5 siblings + prompts.py): handler.py keeps the event loop, fail-closed SES auth, and email_type routing; extraction.py owns extract_with_claude + _EMAIL_TAG_RE; enrichment.py owns enrich_parsed + pad_zip (PO-only — WO has no enrichment stage) as a byte-identical move including the derived-field shadow block; telemetry.py owns the EMF ParseMethod/DerivedFieldAgreement emit wrappers; persistence.py owns _write_fields/_merge_update/save_* — with the two byte-identical save_new_po/save_revision collapsed into one _save_merge plus two thin wrappers differing only in the log verb (behavior-identical to both originals, sticky-cancel ConditionExpression guard intact; save_cancellation stays its own function). WO splits into ~5 concerns (no enrichment), keeping validate_ai_fallback and the re.fullmatch(r"[0-9]+", work_order_id) key guard in the handler loop AHEAD of both save_work_order and save_event (it protects the partition key and the #-delimited comment_id range-key segment), and keeps _header_date_iso/comment_id determinism together with save_event in persistence.py.

EXTRACTION_PROMPT (the ~181-line prompt) moves to prompts.py with a cross-reference header to the second authoritative copy of the trade/site/fiscal rule tables in derived_fields.py; handler.py keeps a from prompts import EXTRACTION_PROMPT re-export so handler.EXTRACTION_PROMPT still resolves for the tests that dereference it.

Lazy cached boto3 clients. Each I/O module initializes its client cache to None and populates it through a private _get_<client>() accessor on first call (extraction.bedrock, persistence.dynamodb, handler.s3); pure modules (enrichment, telemetry, prompts) import no boto3. The cache attribute keeps its original public name, so a test patches the same attribute — only the owning module moved (e.g. setattr(persistence, "dynamodb", fake)). Building the client at first call (deep inside a test) rather than at import also strengthens the moto-before-handler invariant.

Behavior preserved (pinned by new tests). PO emits ParseMethod=ai_fallback before the Bedrock call (a throttle that raises still leaves the pre-call datapoint), and a gate rejection is an additive second ai_fallback_rejected datapoint (PO's deliberate double-count); WO emits after its gate, mutually exclusive; the DerivedFieldAgreement shadow telemetry stays ai_fallback-only; _save_merge issues byte-identical update_item calls for both new_po and revision; and the WO key guard fires before either save.

Security

Web UI auth (defense-in-depth). The po-web-ui / workorder-web-ui handlers refuse unauthenticated requests even though their public Function URLs were removed (INFRA-74). Each requires a shared secret in the X-Auth-Token header (or Authorization: Bearer <token>), compared in constant time against the configured token. The handler fails closed if the token is unset or unreadable (denies all). The token lives in the Secrets Manager secret procurement-ingest/web-ui-auth-token; only its ARN is passed to the Lambda (WEB_UI_AUTH_TOKEN_SECRET_ARN), and the value is fetched at runtime — never embedded in the CloudFormation template or Lambda env vars. The fetched value is cached in the warm container with a short TTL (5 min) so a rotated secret propagates without waiting for the execution environment to recycle. This is a defense-in-depth floor for a detached URL, not primary auth.

Output escaping. All caller-influenced values (including prompt-injectable strings Claude may return for total_amount/line-item amounts) are HTML-escaped before interpolation to prevent stored XSS.

Shared Resources

purchase-orders table (owned here)

The purchase-orders DynamoDB table is owned by this repo's Terraform config (defined in terraform/po_ddb.tf with retain lifecycle, StreamViewType.NEW_IMAGE, and SSE-KMS encryption with the shared customer-managed CMK alias/seahaven-dynamodb, INFRA-95 / M-3). The po-email-processor Lambda is the authoritative writer — it performs the merge inserts, field-level revision merges, and cancellation updates described above.

Consumers (read-only):

Consumer	How it reads	Purpose
procurement-api (this repo)	Table.from_table_name + grant_read_data	REST reads (GET /purchase-orders*)

seahaven-slack-bot, the former external consumer, was decommissioned 2026-07-23 and its cross-account grants removed. External consumers now read through the procurement-api REST contract (lambdas/api/openapi.json), never by importing the table directly.

Schema-coordination rule: Any change to the purchase-orders schema (partition key, item shape, attribute names, streams view type) must be reflected in the API's OpenAPI schemas in the same PR (the spec-drift test pins paths, not item shapes — the schema fields are maintained by hand). Treat schema changes as a contract migration, not a local edit.

Known exception (INFRA-51): amazon-po-parser currently writes directly to purchase-orders outside this stack (backfill/enrichment scripts). This second writer is being folded into the po-ingest pipeline so this stack is the sole writer; until INFRA-51 closes, coordinate any schema change with amazon-po-parser as well.

WorkOrders and WorkOrderComments tables (owned here)

PLAT-11 in progress. Kebab physical names work-orders / work-order-comments are created empty under Terraform; writers/ESMs stay on PascalCase until the freeze cutover in docs/plat-11/cutover-runbook.md. After cutover, this section will name the kebab tables.

Both tables are owned by this repo's Terraform config (terraform/wo_ddb.tf, retain lifecycle):

• WorkOrders — PK work_order_id (S).
• WorkOrderComments — PK work_order_id (S), SK comment_id (S).

comment_id format change (issue #23). The WorkOrderComments range key is now work_order_id#<comment_time|nocomment>#<sha256(s3_object_key)[:12]> (e.g. 11144580730#2026-04-27T23:51:48#a1b2c3d4e5f6, or …#nocomment#… when the source email carries no comment time). Previously it was work_order_id#<timestamp>, where two emails on the same WO with an identical/absent comment time collided and overwrote each other. The 12-hex suffix is derived from the S3 object key alone — deterministic, so a Lambda async retry of the same object produces a byte-identical key (idempotent, no duplicate row), while two distinct emails on the same WO get distinct keys. Wall-clock now() is kept out of the key. Consumers that split on # and read index [0]/[1] are unaffected; anything that treated "everything after the first #" as a bare timestamp now also captures the hash segment.

Timestamp format shift. All stored ISO timestamps (created_at, updated_at, ingested_at on WO; processed_at, cancelled_at on PO) moved from naive datetime.utcnow().isoformat() to timezone-aware datetime.now(timezone.utc).isoformat(), so they now carry a +00:00 suffix (e.g. 2026-07-15T12:00:00+00:00). Downstream parsers that assumed a naive/no-offset string must accept the offset.

Both currently use default DynamoDB encryption — they are not yet on the shared customer-managed CMK (alias/seahaven-dynamodb, INFRA-95 / M-3); that migration is tracked in INFRA-6. The workorder-email-processor role no longer holds a pre-emptive encrypt/decrypt grant on that CMK (removed in the 2026-06-17 security sweep — it was unused while the tables are unencrypted and extended the role's decrypt reach to the CMK protecting purchase-orders). Re-add the grant as part of the INFRA-6 migration, at which point grant_read_write_data on the then-encrypted tables propagates the needed key permissions automatically.

Consumers (read-only) — data contract: seahaven-slack-bot (the former external reader) was decommissioned 2026-07-23; its grants are gone. Current consumers: the procurement-api Lambda (this repo, Table.from_table_name + grant_read_data, serving GET /work-orders*), and the workorder-shoc-emitter stream consumer (this repo) — both tables now stream NEW_AND_OLD_IMAGES, and the emitter is an active consumer of those streams (ESMs enabled 2026-07-30). External readers (SHOC) consume through the REST contract (lambdas/api/openapi.json) and the webhook contract (docs/shoc-webhook-contract.md); both documents' field lists mirror lambdas/wo/email_processor/persistence.py. Any change to table name, key schema, attribute names, or encryption configuration (e.g. the INFRA-6 CMK migration) must update those two contracts in the same PR — the tables are imported by name, so there is no compile-time link and breakage surfaces at runtime.

Stream field contract (site_code). Three definitions of "is this a valid site code" have existed in this repo at once. The canonical shape is derived_fields._STRICT_CODE_RE ([A-Z][A-Z0-9]{2,4}, fullmatch) plus its skip-list semantics (a code-shaped token is only a real site code if it is not skip-listed, e.g. LLC/INC/CORP/LTD/ATTN), used by derive_site_code() (the enrich_parsed() classifier, see the Derived fields note in the Purchase Orders flow above). Honestly noted, not papered over: lambdas/po/site_extractor/handler.py's own SITE_CODE_PATTERN ([A-Z]{2,4}\d{1,2}, prefix-anchored .match, digit-requiring) still diverges from the canonical shape as of this phase — it rejects valid all-letter codes like KLAL (which surface as permanent pending-site-review rows) and accepts overlong junk like DLI6X/SNY55 that the canonical fullmatch would not. Reconciling po-ingest-site-extractor's direct-field validation onto the canonical derived_fields shape is Phase 6 scope, tracked separately — this paragraph will be updated when it lands.

verified-sites table (owned here)

Owned by this repo's Terraform config (terraform/po_ddb.tf). PK siteCode (S); default DynamoDB encryption (NOT the shared CMK).

Consumer (read-only) — data contract: seahaven-slack-bot (former reader, incl. the INFRA-138/INFRA-180 by-state GSI drift saga) was decommissioned 2026-07-23 — the GSI-drift issue died with it. Current consumer: the procurement-api Lambda (GET /verified-sites*). The API's VerifiedSite schema deliberately promises only pipeline-written fields (siteCode, address, city, state, zip, fullAddress, locationCode, poCount, sourcePOs) — latitude/longitude/notes were manually curated legacy attributes the extractor never writes, and are not part of the contract.

Documentation

The canonical map of Sea Haven's AWS infrastructure lives in Confluence. This project's live path is HCP Terraform workspace procurement-ingest-prod (PLAT-86).

• AWS Architecture Map (Confluence, IT space, page 1540098)
• Import / dispose notes: docs/plat-86/

CI/CD

GitHub Actions with reusable workflows from Sea-Haven-Industries/.github (all pinned to a commit SHA of main):

• CI (ci.yaml, PR to main): linting + pytest via ci-python-sam.yaml (run-cdk-synth: false; HCP Terraform is the deploy path).
• Terraform CI (ci-terraform.yaml, PR to main when terraform/** changes): fmt -check, init -backend=false, validate.
• CD: HCP Terraform workspace procurement-ingest-prod (VCS-bound to main, working directory terraform/, Manual apply until sealed). There is no push-to-main CDK/SAM deploy workflow. Post-apply smoke: scripts/post-deploy-smoke.sh.
• Plus dependency review and PR labeler workflows on every PR

Branch protection on main — all changes through PR.

Setup

Account prerequisites — Terraform imports these by name/ARN; they must already exist in seahaven-prod (provisioned by seahaven-org-baseline and out-of-band scripts):

• SES receipt rule set INBOUND_MAIL + verified int.seahaven.com domain identity
• SNS topic site-alerts (with the alias/seahaven-alarm-topics CMK)
• SSM param /seahaven/dynamodb/cmk-arn → KMS alias/seahaven-dynamodb
• SSM param /procurement-api/custom-domain/certificate-arn (ACM in prod; DNS validation/alias for procurement-api.seahaven.com stays in mgmt via scripts/setup_procurement_api_domain.sh)
• Secrets Manager secret procurement-ingest/web-ui-auth-token
• HCP plan/apply roles hcptf-procurement-ingest-plan / hcptf-procurement-ingest (org-baseline terraform-substrate)

1. Set workspace variables in HCP (procurement-ingest-prod) from terraform/terraform.tfvars.example (secret ARNs only; never secret values).
2. Bedrock model access (account first-use). Anthropic Marketplace agreements and inference-profile reach are account-level; processor roles already carry bedrock:InvokeModel grants under /tf-managed/.
3. Create the web UI auth-gate shared secret if missing (imported by name, not Terraform-managed):

   aws secretsmanager create-secret --name procurement-ingest/web-ui-auth-token --secret-string "$(openssl rand -hex 32)"
   

4. Apply from the HCP workspace (Manual apply until the stack is sealed). Do not use local terraform apply against prod.
5. Post-apply smoke:

   AWS_PROFILE=seahaven-prod ./scripts/post-deploy-smoke.sh
   

6. Dashboards: po-web-ui and workorder-web-ui have no public endpoint (Function URLs removed 2026-06-08, INFRA-74). Fetch the token and forward it in the event's headers:

   TOKEN=$(aws secretsmanager get-secret-value --secret-id procurement-ingest/web-ui-auth-token --query SecretString --output text)
   aws lambda invoke --function-name po-web-ui --payload "{\"headers\":{\"x-auth-token\":\"$TOKEN\"}}" /tmp/out.json
   

Removed (PLAT-88): githubdeploy-procurement-ingest OIDC deploy role and infra/deploy-role/ artifacts deleted after the HCP hard-cut. Do not recreate a mgmt twin (deleted in PLAT-67).

Tests

Offline unit tests (no AWS, no network) run via pytest from the repo root:

pip install -r tests/requirements.txt
pytest

Test-root consolidation (Phase 8). Both test roots are kept (tests/ for genuinely cross-pipeline suites, plus each pipeline's own lambdas/*/email_processor/tests/), but loading is now single-sourced. A new repo-root conftest.py — not tests/conftest.py, which is never an ancestor of the pipeline test roots and so cannot load for a standalone pytest lambdas/po/email_processor/tests run — sets the dummy AWS env, imports moto before any handler import (registers moto's botocore stubber hook so boto3 sessions created afterwards are stubbable), and exposes the one load_lambda_module(pipeline, name) loader (implemented in tests/support/loader.py) that every handler exec in the suite goes through, including its sys.modules save/restore dance for the per-pipeline duplicated bare names (template_parser, persistence, etc. — derived_fields/prompts/telemetry/extraction/enrichment too). tests/support/ is a small shared package (FakeTable, FakeDynamoResource, FakeS3, load_email, load_golden) used by both pipelines' _po_parser_support.py/_wo_parser_support.py shims — load_golden decodes JSON money with parse_float=Decimal unconditionally (load-bearing for PO's exact-money goldens; proven safe for WO, whose 55 golden fixtures contain zero float-typed JSON numbers). test_local.py (a root-level manual script that imported a handler at collection time, bypassing the loader gate, and globbed a nonexistent samples/ dir) is deleted — the golden suites cover its role.

Coverage:

• tests/ — shared handler + cross-pipeline tests: parse_raw_email (test_parse_raw_email.py), the fail-closed sender-authentication parser (test_ses_auth.py, INFRA-107), the packaging/handler-import consistency check (test_bundle_consistency.py — see Deploy-Pipeline Guards), the scripts/reprocess.py synthetic S3-event-shape contract (test_reprocess_contract.py, Phase 7), and, new in Phase 8: the handler-level SES-auth reject seam per pipeline (test_handler_auth_seam.py — no auth monkeypatch + an empty ALLOWED_DKIM_DOMAINS must yield zero Bedrock calls, zero writes, no raise), and both web_ui functions' first coverage — fail-closed auth (test_web_ui_auth.py) and the handlers themselves, including a 401-without-a-table-scan assertion and a hostile-field escaping regression lock (test_web_ui_handlers.py).
• lambdas/wo/email_processor/tests/ — the deterministic WO parser suite: golden-file tests over 55 real scrubbed .eml fixtures (test_parser.py), fail-closed validation-gate rules and adversarial/injection cases (test_validation_gate.py), the issue #23 comment_id idempotency invariants (test_comment_id.py), the Bedrock-fallback dispatch/EMF-metric behavior with a mocked invoke_model (test_bedrock_fallback.py, which also carries the Phase 5 behavior pins — WO's mutually-exclusive emit and the [0-9]+ key guard ahead of both saves), and the Phase 0 direct-invoke healthcheck contract (test_healthcheck.py). Golden JSON lives under tests/fixtures/expected/. New in Phase 8: test_wo_bedrock_transport.py (transport errors — throttle, missing content, empty content, non-JSON model text — plus the hand-computed no-double-count pin for the handler.py except-and-reraise metric wrap, and the multi-record failure-isolation pin) and test_wo_merge.py (a moto-backed mirror of test_po_merge.py against the WorkOrders table — null-status never clobbers wo_status, created_at immutable, status→wo_status mapping, None fields absent from SET, record_type only-when-present). After Phase 5 the suite patches accessors on the owning siblings (extraction.bedrock, persistence.dynamodb, persistence.datetime, persistence.save_*) rather than on handler.
• lambdas/po/email_processor/tests/ — the deterministic PO parser suite: golden-file tests over real scrubbed .eml fixtures (17 single-line new-PO + 8 cancellations, exact Decimal-aware golden comparison via parse_float=Decimal), fail-closed validation-gate coverage for every gate reason code (fixture-driven for body-level triggers under fixtures/adversarial/, direct validate() unit tests for candidate-level mutations), real multi-line and comment/non-Coupa fallback fixtures under fixtures/ai-fallback/, dual line-ending (CRLF/LF) parse-identity, two-path enrich_parsed/save_new_po parity (the site-extractor stream-contract guard), fixture hygiene (ses_auth pass + scrub-marker leak sweep), the Bedrock-fallback dispatch/EMF-metric behavior plus the Phase 5 behavior pins (test_po_bedrock_fallback.py — the pre-Bedrock ai_fallback emit and the additive rejected double-count), the _save_merge collapse parity to both save_new_po/save_revision (test_po_save_merge_parity.py), the ai_fallback-only shadow telemetry after the enrich_parsed move (test_po_derived_wiring.py, which gains the PO-DC-02 64-char po_number clamp regression pin in Phase 8), and the Phase 0 direct-invoke healthcheck contract (test_po_healthcheck.py). New in Phase 8: test_po_bedrock_transport.py (the same four transport-error cases as WO, asserting the pre-call ai_fallback metric survives and no partial write occurs, plus the multi-record failure-isolation pin) and, moved here from tests/: test_po_merge.py (#97, moto-backed merge-write semantics) and test_pad_zip.py (zip-code padding). After Phase 5 the suite patches accessors/constants on the owning siblings (po_extraction.bedrock/.BEDROCK_MODEL_ID/._EMAIL_TAG_RE, po_persistence.dynamodb/.PO_TABLE, po_enrichment.derive_all/.pad_zip, po_telemetry.DERIVED_METRIC_NAME) rather than on po_handler; re-exported names it calls (save_*, extract_with_claude, enrich_parsed, _emit_parse_method_metric, EXTRACTION_PROMPT) stay on po_handler.

All three roots are discovered by pytest.ini (testpaths).

Coverage floor (Phase 8). pytest.ini's addopts runs pytest-cov with an explicit --cov path per first-party package (lambdas/po/email_processor, lambdas/wo/email_processor, lambdas/po/web_ui, lambdas/wo/web_ui, lambdas/po/site_extractor, lambdas/shared) rather than relying on an __init__.py package marker — none of these dirs have one, and adding one would perturb flat packaging for zero runtime benefit; pytest-cov's path form measures by source file regardless of package markers. site_extractor is deliberately included even though it measures 0% until Phase 6 lands — coverage honesty, not a silent skip. .coveragerc omits */tests/* so the pipeline test dirs (which sit inside their own --cov path) don't dilute the number. --cov-fail-under is the new permanent CI floor (constraint 10 — never ratcheted down).

Ruff C901/PLR floor (Phase 8). ruff.toml adds extend-select = ["C901", "PLR"] with max-complexity = 12, making the tree's pre-existing # noqa: PLR09xx suppressions load-bearing instead of inert (no config previously enabled the rules they suppressed). scripts/ is in the lint scope. Findings that could not be split or were out of this phase's file-ownership were resolved with a per-file [lint.per-file-ignores] entry carrying a written justification — most notably derived_fields.py (shadow-bake freeze: even an in-file noqa comment is a barred edit) and the other Phase 3/5/6/7 frozen modules (enrichment.py, WO template_parser.py, both ses_auth.py/emf.py, site_extractor/handler.py, scripts/reprocess.py). PO's template_parser.py — this phase's one in-scope split target — clears the ceiling by decomposing extract_new_po and _validate_new_po_values into per-rule helpers instead of an ignore.

Scripts

Reprocess emails (re-invoke an email-processor with a synthetic S3 event). reprocess.py is now pipeline-general: --pipeline po|wo selects the function + raw-email bucket. Targeted replay (--key one object, --prefix, or --since a LastModified timestamp) is the default, preferred mode; the full-prefix sweep is demoted behind an explicit --all. Every mode is dry-run unless --execute. See the DLQ recovery runbook for the targeted single-key re-invoke flow.

python scripts/reprocess.py --pipeline po --key inbound/2026/msg.eml            # targeted, dry-run
python scripts/reprocess.py --pipeline po --key inbound/2026/msg.eml --execute  # targeted, re-invoke one
python scripts/reprocess.py --pipeline wo --all --execute                       # demoted full-prefix sweep (see --all caveats)

Backfill verified sites (one-time scan of historical POs):

python scripts/backfill_sites.py

Replay SHOC webhooks (rebuild work-order webhook events from DynamoDB and re-POST them, marked "replay": true). replay_shoc_webhooks.py is the operator runbook for the workorder-shoc-emitter-failures / -rejected alarms: when deliveries were parked (SHOC down past the 24h retry window, or a contract-bug 4xx), replay re-sends the affected work orders from current table state — receivers dedupe on the deterministic delivery_id, so overlapping or repeated runs are harmless. Selection is exactly one of --work-order-id (repeatable, targeted) or --since (a full table Scan — a count banner prints per table); --events narrows to state rows, comments, or both. --url is required with no default — replay must be a deliberate act against a known receiver. Every mode is dry-run unless --execute.

python scripts/replay_shoc_webhooks.py --url https://... --work-order-id 11144580730            # targeted, dry-run first
python scripts/replay_shoc_webhooks.py --url https://... --work-order-id 11144580730 --execute  # then re-POST
python scripts/replay_shoc_webhooks.py --url https://... --since 2026-07-24T02:00:00Z --execute # everything touched since (full Scan)

**PLAT-11 WO table rename** (PascalCase → kebab, freeze cutover). See
[`docs/plat-11/cutover-runbook.md`](docs/plat-11/cutover-runbook.md). Dry-run copy:

```bash
python scripts/migrate_wo_tables.py copy
python scripts/migrate_wo_tables.py copy --execute
python scripts/migrate_wo_tables.py verify

## Directory Structure

terraform/ # Sole IaC path (HCP workspace procurement-ingest-prod) build_packages.sh # Lambda zip packaging (copy_py_dir / copy_shared_all / copy_src_file) *.tf # PO / WO / API / SHOC resources imported under PLAT-86 lambdas/ # Runtime source; packaged by terraform/build_packages.sh # email processors: copy_py_dir + copy_shared_all; # web_ui: copy_py_dir + shared/web_ui_auth.py only; # site_extractor / SHOC: copy_py_dir of their own dirs shared/ # Phase 3: single-sourced first-party modules, landed FLAT (no # init.py -- bare-name imports) into each email-processor zip ses_auth.py # fail-closed SES sender-auth (INFRA-107) -- one copy, one fix web_ui_auth.py # fail-closed X-Auth-Token gate + token cache (INFRA-74; also gates the API /docs routes) email_parsing.py # parse_raw_email (WO superset; returns cc unconditionally) emf.py # generic CloudWatch EMF emitter (dimension-sets pinned once) api/ # procurement-api Lambda (REST reads over both pipelines' tables) handler.py # healthcheck early-return + docs-token gate + route dispatch + error mapping router.py # single route table (spec-drift test pins it to openapi.json) pagination.py # opaque base64(LastEvaluatedKey) cursor encode/validate (hostile cursor -> 400) serialization.py # Decimal-safe JSON responses wo_repo.py # WorkOrders/WorkOrderComments reads (paginated Scan / PK Query) po_repo.py # purchase-orders/verified-sites reads (no VendorReplies -- dead table) openapi.json # OpenAPI 3.1 source of truth (paths + outbound webhooks section) docs.html # Redoc shell, SHOC design-system theme; handler inlines bundle/fonts/spec at request time (single token-gated request, no CDN) redoc.standalone.js # vendored Redoc bundle (redoc 2.5.3, MIT), inlined into /docs; read-only docs, no try-it-out fonts.css # SHOC fonts (DM Sans/Montserrat/JetBrains Mono, @fontsource latin subsets) as data URIs, inlined into /docs po/ # PO pipeline Lambdas email_processor/ # Phase 5: God-handler decomposed into flat siblings (bare-name # imports; copy_py_dir ships every new sibling automatically) handler.py # event loop + fail-closed SES auth + email_type routing + {"healthcheck": true} early-return; lazy s3 accessor; re-exports EXTRACTION_PROMPT (4 tests deref handler.EXTRACTION_PROMPT) extraction.py # extract_with_claude + _EMAIL_TAG_RE + EXTRACTION_PROMPT import; lazy bedrock accessor enrichment.py # enrich_parsed + pad_zip (PO-only; no boto3); derived-field shadow block telemetry.py # EMF ParseMethod + DerivedFieldAgreement emit wrappers (stdout EMF; no boto3) persistence.py # _write_fields/_merge_update + save_cancellation; save_new_po/save_revision collapsed into one behavior-identical _save_merge (sticky-cancel guard intact); lazy dynamodb accessor prompts.py # EXTRACTION_PROMPT (~181 lines); cross-ref header to derived_fields' rule tables template_parser.py # pure deterministic Coupa parser + fail-closed validation gate; Phase 8: # extract_new_po/_validate_new_po_values split into C901/PLR-clean per-rule # helpers (behavior byte-identical; see the ruff C901/PLR floor below) derived_fields.py # deterministic site_code/trade/fiscal_year classifier (untouched by Phase 3/5/8 -- shadow-bake freeze; ruff per-file-ignore instead of an in-file noqa) tests/ # golden-file + validation-gate + fallback-dispatch + healthcheck + save-merge-parity + behavior-pin tests + fixtures conftest.py # fake_dynamo fixture, patches persistence.dynamodb _po_parser_support.py # Phase 8: thin shim -- re-exports tests.support fakes/loader; PO module handles (template_parser, prompts, telemetry, extraction, enrichment, persistence) test_po_derived_fields.py # derived_fields.derive_all() unit tests test_po_derived_wiring.py # ai_fallback-only shadow-telemetry wiring; Phase 8 gains the PO-DC-02 64-char po_number clamp regression pin test_po_bedrock_transport.py # Phase 8: Bedrock transport errors (throttle/missing-content/empty-content/non-JSON) + pre-call metric survival + multi-record failure-isolation pin test_po_merge.py # Phase 8: moved from tests/ -- PO merge-write semantics tests (#97), moto-backed test_pad_zip.py # Phase 8: moved from tests/ -- PO zip-code padding tests site_extractor/ web_ui/ # handler.py imports from web_ui_auth import is_authenticated (shared) wo/ # WO pipeline Lambdas email_processor/ # Phase 5: decomposed into ~5 flat siblings (no enrichment stage); # validate_ai_fallback + the [0-9]+ work_order_id key guard stay in # the handler loop AHEAD of both saves handler.py # event loop + fail-closed SES auth + [0-9]+ work_order_id key guard + {"healthcheck": true} early-return; lazy s3 accessor; re-exports EXTRACTION_PROMPT; Phase 8: the Bedrock call is wrapped in try/except so a transport error emits one ai_fallback/bedrock_error datapoint then re-raises (a gate rejection still emits only ai_fallback_rejected -- no double-count) extraction.py # extract_with_bedrock + _EMAIL_TAG_RE + EXTRACTION_PROMPT import; lazy bedrock accessor telemetry.py # emit_parse_metric EMF wrapper (stdout EMF; no boto3) persistence.py # save_work_order + _header_date_iso + save_event (#23 comment_id determinism kept WITH persistence); lazy dynamodb accessor prompts.py # EXTRACTION_PROMPT; cross-ref header to template_parser.CONTRACT_KEYS template_parser.py # pure deterministic parser + fail-closed validation gate; Phase 8: the # template-path bad-status branch now returns "invalid_status" (was a # copy-paste "malformed_site_code") -- matches validate_ai_fallback's code # for the identical condition tests/ # golden-file + validation-gate + comment_id + fallback + healthcheck + behavior-pin tests conftest.py # fake_dynamo fixture, patches persistence.dynamodb (module-top from _wo_parser_support import wo_persistence, Phase 8) _wo_parser_support.py # Phase 8: rewritten OFF the bare import handler strategy -- re-exports tests.support fakes/loader; WO module handles (template_parser, persistence, extraction, telemetry) test_wo_merge.py # Phase 8: moto-backed WorkOrders save_work_order merge-semantics mirror of test_po_merge (null-status never clobbers wo_status, created_at immutable, status->wo_status mapping, None fields absent, record_type only-when-present) test_wo_bedrock_transport.py # Phase 8: Bedrock transport errors + the hand-computed emitted-series no-double-count pin for the handler.py except-and-reraise wrap + multi-record failure-isolation pin fixtures/ ses-stamped/ auth-pass-01.eml # Phase 8: the one new fixture allowed this phase -- a synthesized Authentication-Results header block (from WO_SES_HEADER in test_ses_auth.py), not scraped mail web_ui/ # handler.py imports from web_ui_auth import is_authenticated (shared) shoc_emitter/ # SHOC webhook emitter (ACTIVE since 2026-07-30; see the SHOC # webhook feed section). Flat siblings, bare-name imports handler.py # thin per-record event loop + partial-batch failure report + rejected-queue parking envelope.py # pure stream-record -> envelope mapping + event classification (incl. echo guard) delivery.py # TTL-cached secret fetch + HMAC signing (sign_body) + POST + response classification shoc_hmac_rotator/ handler.py # 30-day Secrets Manager rotation (single-user: createSecret/finishSecret; # dual-key overlap, kid YYYY-MM-DDTHH) scripts/ reprocess.py backfill_sites.py replay_shoc_webhooks.py # rebuild + re-POST webhook events ("replay": true) -- the # -failures/-rejected alarm runbook; dry-run unless --execute post-deploy-smoke.sh # CD gate: synchronous healthcheck invoke of the three smoke-gated functions, checks FunctionError conftest.py # Phase 8: THE repo-root session-invariant conftest. rootdir is pinned by # pytest.ini at repo root, so this loads for every pytest invocation shape -- # including a standalone pytest lambdas/po/email_processor/tests -- before # any collection import. Sets the dummy AWS env, imports moto BEFORE any # handler import (registers moto's botocore stubber hook so boto3 sessions # created afterwards are stubbable), and re-exports # tests.support.loader.load_lambda_module. Owns the session fixtures # (po_handler, wo_handler, email_handler, ses_auth, po_persistence, # po_enrichment, wo_persistence). Supersedes the old tests/conftest.py # (deleted -- it was never an ancestor of the pipeline test roots, so it # could not carry these invariants for a standalone pipeline-root run). tests/ requirements.txt # Test-only deps (moto, pytest-cov) support/ # Phase 8: shared test package (tests/ itself has no init.py -- # PEP-420 namespace resolution via the root conftest's sys.path insert) init.py # load_lambda_module + REPO_ROOT re-export, FIXTURE_DIRS, stems/ # load_raw/load_email/load_golden (parse_float=Decimal, pinned safe # for both pipelines), and the superset FakeTable/FakeDynamoResource/FakeS3 loader.py # the ONE load_lambda_module + sys.modules save/restore dance # (dependency-topological sibling tuple + web_ui_auth); every # pipeline handler exec in the suite goes through this test_parse_raw_email.py # MIME parsing tests (PO + WO handlers) -- stays at root, cross-pipeline test_ses_auth.py # Sender-authentication parser tests (INFRA-107) -- stays at root, cross-pipeline test_handler_auth_seam.py # Phase 8: handler-level SES-auth reject seam, per pipeline -- no auth # monkeypatch + empty ALLOWED_DKIM_DOMAINS -> zero Bedrock calls, zero # writes, no raise (closes the "delete the gate line, tests still pass" hole) test_web_ui_auth.py # Phase 8: shared/web_ui_auth.py -- fail-closed on unset ARN / Secrets # Manager exception, TTL cache refresh, header-matrix case-insensitivity, # non-ASCII-token documenting pin test_web_ui_handlers.py # Phase 8: both web_ui handlers (0% coverage before this phase) -- 401 # without a table scan, authenticated render path, hostile-field escaping # regression lock test_bundle_consistency.py # AST check: bundling command ships every handler.py sibling import test_reprocess_contract.py # reprocess.py synthetic S3-event-shape contract (Phase 7)