mirror of
https://github.com/Sea-Haven-Industries/procurement-ingest.git
synced 2026-10-07 16:08:54 +00:00
Document PO template-first parser and retuned alarm
README: PO flow is now template-first with Bedrock fallback; parser/gate section mirroring the WO writeup; Seahaven/PoIngest ParseOutcome namespace and the fallback-rate alarm numbers with their volume justification (deliberately not WO's settings); test-suite and repo-layout updates. Design doc: mark PR #1 complete in progress/checklist sections; document the six value-level gate reason codes and the scaffold guard removal; correct the stale data-access note (default CLI session is 328440206208) and note the ~90-day S3 lifecycle aging of the corpus; record the 2.3 layout addendum (leading Supplier bullet segment, EA evidence lines, summary unit-price tokens, decode-path line endings), the fixture-build pins (address join convention, quantity/unit/price source), the V10 sweep outcome, and resolutions for open questions Q3/Q6. Cross-family review and the Confluence architecture-map update are flagged outstanding for merge. Refs: #99
This commit is contained in:
parent
14fad0c317
commit
5d100eba9b
2 changed files with 62 additions and 26 deletions
16
README.md
16
README.md
|
|
@ -10,14 +10,14 @@ Unified email ingestion pipelines for Amazon procurement data. Two independent p
|
||||||
|
|
||||||
### Purchase Orders (`po-ingest` stack)
|
### Purchase Orders (`po-ingest` stack)
|
||||||
|
|
||||||
Coupa PO emails are received at `amazon_po@int.seahaven.com`, parsed by Claude Haiku 4.5 **on Amazon Bedrock**, and written to the `purchase-orders` DynamoDB table. PO parsing is fully AI (no deterministic parser).
|
Coupa PO emails are received at `amazon_po@int.seahaven.com`, parsed **deterministic-template-first with a Claude-Haiku-4.5-on-Bedrock fallback**, and written to the `purchase-orders` DynamoDB table.
|
||||||
|
|
||||||
**Flow:**
|
**Flow:**
|
||||||
1. Coupa sends a PO email (new, revision, or cancellation).
|
1. Coupa sends a PO email (new, revision, or cancellation).
|
||||||
2. SES (`INBOUND_MAIL` rule set) drops the raw MIME into `s3://po-ingest-emails-{AccountId}/inbound/`.
|
2. SES (`INBOUND_MAIL` rule set) drops the raw MIME into `s3://po-ingest-emails-{AccountId}/inbound/`.
|
||||||
3. S3 `ObjectCreated` triggers the `po-email-processor` Lambda.
|
3. S3 `ObjectCreated` triggers the `po-email-processor` Lambda.
|
||||||
4. Fail-closed sender authentication (INFRA-107): the SES-stamped `Authentication-Results` header must show `dkim=pass` for `amazon.coupahost.com` (see [Sender authentication](#sender-authentication-infra-107)); otherwise the email is logged and dropped.
|
4. Fail-closed sender authentication (INFRA-107): the SES-stamped `Authentication-Results` header must show `dkim=pass` for `amazon.coupahost.com` (see [Sender authentication](#sender-authentication-infra-107)); otherwise the email is logged and dropped.
|
||||||
5. Claude (via Bedrock `InvokeModel`) extracts structured JSON (PO number, status, supplier, site code, trade classification, line items, fiscal year). Numeric amounts are decoded with `parse_float=Decimal` (DynamoDB rejects floats).
|
5. **Parse:** a pure, offline template parser (`template_parser.py`) tries the two known Coupa templates first — `coupa_new_po` (95.5% of traffic) and `coupa_cancellation` (2.9%) — behind a strict fail-closed validation gate. Only on a miss/invalid result does the Lambda fall back to the Claude-on-Bedrock AI extractor (`InvokeModel`), which extracts the identical structured-JSON contract (PO number, status, supplier, nested ship-to, line items). Numeric amounts are `Decimal` on both paths (the AI decode uses `parse_float=Decimal`; DynamoDB rejects floats). The derived classification fields (`site_code`, `trade`, `fiscal_year`) stay LLM-computed for now: the template parser leaves them `null`, and the shared `enrich_parsed()` post-stage (zip padding, `ship_to_raw`/`state` promotion, metadata) runs identically on both paths so downstream shapes are byte-identical.
|
||||||
6. Merge write to DynamoDB:
|
6. Merge write to DynamoDB:
|
||||||
- `new_po` — merge insert. Creates the PO, or backfills data into a pre-existing `Cancelled` skeleton left by an out-of-order cancellation (preserving the `Cancelled` status). No longer silently dropped when a record already exists.
|
- `new_po` — merge insert. Creates the PO, or backfills data into a pre-existing `Cancelled` skeleton left by an out-of-order cancellation (preserving the `Cancelled` status). No longer silently dropped when a record already exists.
|
||||||
- `revision` — field-level merge (`update_item` SETs only the fields present in the revision). A revision that omits `line_items`/`supplier` no longer deletes them. Will not un-cancel a `Cancelled` PO.
|
- `revision` — field-level merge (`update_item` SETs only the fields present in the revision). A revision that omits `line_items`/`supplier` no longer deletes them. Will not un-cancel a `Cancelled` PO.
|
||||||
|
|
@ -26,6 +26,8 @@ Coupa PO emails are received at `amazon_po@int.seahaven.com`, parsed by Claude H
|
||||||
- **LedgerFlow** (`seahaven-slack-bot/po-sync`) — daily KB sync
|
- **LedgerFlow** (`seahaven-slack-bot/po-sync`) — daily KB sync
|
||||||
- **Site extractor** (`po-ingest-site-extractor`) — real-time site address extraction into `verified-sites` table
|
- **Site extractor** (`po-ingest-site-extractor`) — real-time site address extraction into `verified-sites` table
|
||||||
|
|
||||||
|
**Deterministic template parser.** `template_parser.try_deterministic_parse()` classifies by exact subject regex, extracts the nested contract (`supplier{}`, `ship_to{}` — 8 keys, `line_items[]` — 10 keys per item), and returns a parsed result **only if** it passes a fail-closed validation gate: recursive exact key-set at every nesting level; `email_type` emitted **only** on the exact new-PO subject *and* a confirmed-safe `Status` (never defaulted — a cancellation misrouted as `new_po` would defeat the sticky-`Cancelled` guard); `po_number` shape + byte-equality with the subject, the body `PO ID`, the `Amazon Purchase Order #` heading, and the `orders/<id>` URL; duplicate-label anchor integrity (`Supplier`/`Shipping`/`Total` each appear twice, the first `Shipping` must be the literal `None` placeholder); money fidelity (every `Decimal` re-serializes byte-identically to its source token with a digit/comma border check — the thousands-separator-truncation kill switch — plus `sum(line amounts) == total`, proven against **both** `Total` blocks); bullet-metadata label discipline (closed label set, assigned by leading label, never ordinal — immune to the optional `Part Number` segment); USPS address shape on the raw pre-enrichment value; and rejection of any unparseable sentinel or residual `\r`/`\xa0` artifact. Multi-line-item (0.18%) and non-USD (0 observed) new-POs, comment emails, and anything else falls back to the AI extractor. **Data is never corrupted; only the fallback rate rises.** Every record emits one CloudWatch EMF metric (see below).
|
||||||
|
|
||||||
**Lambdas** (`lambdas/po/`):
|
**Lambdas** (`lambdas/po/`):
|
||||||
| Function | Trigger | Purpose |
|
| Function | Trigger | Purpose |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
|
|
@ -119,6 +121,8 @@ The `<fn>-duration` and `<fn>-throttles` alarms for `po-email-processor` and `wo
|
||||||
|
|
||||||
**Parse-outcome metric + fallback-rate alarm (workorder-ingest):** the WO processor writes one CloudWatch **EMF** line per email to namespace `Seahaven/WorkorderIngest`, metric `ParseOutcome` (Unit Count, value 1), dimensioned by `ParseMethod` (`template` | `ai_fallback`) and `TemplateId` (`update_plaintext` | `assign_html` | `unknown`). Non-dimension EMF properties `ReasonCode` and `work_order_id` are queryable in Logs Insights but not promoted to metrics (kept low-cardinality). EMF is used instead of `PutMetricData` so there is no extra sync call / latency / IAM grant on the async hot path (the role already has `logs:PutLogEvents`). The alarm `workorder-email-processor-template-fallback-rate` fires when the AI-fallback share of parses exceeds **15%** sustained (a `MathExpression` with `FILL(...,0)` and a ≥10-sample volume floor over 15-minute periods, eval 3 / datapoints 2) — catching Hexagon template-drift coverage collapse while the volume floor + `FILL` prevent low-volume false pages / `INSUFFICIENT_DATA`. ALARM-only `SnsAction` to `site-alerts`, no OK action, `NOT_BREACHING`. The 15-minute period is a deliberate deviation from the 5-minute house style to accumulate a stable denominator at the low ~760/day volume.
|
**Parse-outcome metric + fallback-rate alarm (workorder-ingest):** the WO processor writes one CloudWatch **EMF** line per email to namespace `Seahaven/WorkorderIngest`, metric `ParseOutcome` (Unit Count, value 1), dimensioned by `ParseMethod` (`template` | `ai_fallback`) and `TemplateId` (`update_plaintext` | `assign_html` | `unknown`). Non-dimension EMF properties `ReasonCode` and `work_order_id` are queryable in Logs Insights but not promoted to metrics (kept low-cardinality). EMF is used instead of `PutMetricData` so there is no extra sync call / latency / IAM grant on the async hot path (the role already has `logs:PutLogEvents`). The alarm `workorder-email-processor-template-fallback-rate` fires when the AI-fallback share of parses exceeds **15%** sustained (a `MathExpression` with `FILL(...,0)` and a ≥10-sample volume floor over 15-minute periods, eval 3 / datapoints 2) — catching Hexagon template-drift coverage collapse while the volume floor + `FILL` prevent low-volume false pages / `INSUFFICIENT_DATA`. ALARM-only `SnsAction` to `site-alerts`, no OK action, `NOT_BREACHING`. The 15-minute period is a deliberate deviation from the 5-minute house style to accumulate a stable denominator at the low ~760/day volume.
|
||||||
|
|
||||||
|
**Parse-outcome metric + fallback-rate alarm (po-ingest):** the PO processor emits the same EMF shape to namespace `Seahaven/PoIngest`, metric `ParseOutcome`, dimensioned by `ParseMethod` (`template` | `ai_fallback`) and `TemplateId` (`coupa_new_po` | `coupa_cancellation` | `unknown`), with `ReasonCode` (the fail-closed gate reason) and `po_number` as Logs-Insights ride-alongs. The alarm `po-email-processor-template-fallback-rate` is **deliberately retuned for PO volume — do NOT copy the WO numbers**: at ~57 emails/day a 15-minute period holds ~0.6 emails, so the WO ≥10-sample floor would never be met and the alarm would be structurally dead. Instead: **6-hour periods** (~14.25 expected emails each), an `IF((fb+tmpl)>=8, …)` volume floor (at the floor a single fallback email is 12.5% < the threshold, so one email can never breach a datapoint; a breach needs ≥2 fallbacks in one window, or ≥3 at typical volume), threshold **>20%** (expected baseline fallback ≈1%: comments 0.55% + multi-line 0.18% + non-USD 0), **eval 4 / datapoints 2** (a 24h span — isolated noise self-clears while total template drift at 100% fallback pages within ~12h). Sparse overnight/weekend windows below the floor evaluate to 0 (non-breaching by design; accepted trade: a Friday-evening drift may not page until weekend volume accrues). Same idiom otherwise: ALARM-only `SnsAction` to `site-alerts`, `NOT_BREACHING`, and no element-wise `MAX` in the math expression (the post-#102 rule — the `IF` floor guarantees the non-zero denominator).
|
||||||
|
|
||||||
**DynamoDB alarms** (`AWS/DynamoDB`): each owned table gets `<table>-throttles` (`ThrottledRequests`) and `<table>-system-errors` (`SystemErrors`). These metrics emit only at the `TableName` + `Operation` dimension set, so each alarm is a `Sum` math expression across the operations the table uses (Get/BatchGet/Query/Scan/Put/Update/Delete/BatchWrite). Tables covered: `purchase-orders`, `verified-sites`, `pending-site-review` (po-ingest); `WorkOrders`, `WorkOrderComments` (workorder-ingest).
|
**DynamoDB alarms** (`AWS/DynamoDB`): each owned table gets `<table>-throttles` (`ThrottledRequests`) and `<table>-system-errors` (`SystemErrors`). These metrics emit only at the `TableName` + `Operation` dimension set, so each alarm is a `Sum` math expression across the operations the table uses (Get/BatchGet/Query/Scan/Put/Update/Delete/BatchWrite). Tables covered: `purchase-orders`, `verified-sites`, `pending-site-review` (po-ingest); `WorkOrders`, `WorkOrderComments` (workorder-ingest).
|
||||||
|
|
||||||
## Security
|
## Security
|
||||||
|
|
@ -231,8 +235,9 @@ pytest
|
||||||
Coverage:
|
Coverage:
|
||||||
- `tests/` — shared handler + cross-pipeline tests: `parse_raw_email` and `pad_zip`, the PO merge-write semantics (`test_po_merge.py`, #97, moto-backed), and the fail-closed sender-authentication parser (`test_ses_auth.py`, INFRA-107).
|
- `tests/` — shared handler + cross-pipeline tests: `parse_raw_email` and `pad_zip`, the PO merge-write semantics (`test_po_merge.py`, #97, moto-backed), and the fail-closed sender-authentication parser (`test_ses_auth.py`, INFRA-107).
|
||||||
- `lambdas/wo/email_processor/tests/` — the deterministic WO parser suite: golden-file tests over 55 real scrubbed `.eml` fixtures (`test_parser.py`), fail-closed validation-gate rules and adversarial/injection cases (`test_validation_gate.py`), the issue #23 `comment_id` idempotency invariants (`test_comment_id.py`), and the Bedrock-fallback dispatch/EMF-metric behavior with a mocked `invoke_model` (`test_bedrock_fallback.py`). Golden JSON lives under `tests/fixtures/expected/`.
|
- `lambdas/wo/email_processor/tests/` — the deterministic WO parser suite: golden-file tests over 55 real scrubbed `.eml` fixtures (`test_parser.py`), fail-closed validation-gate rules and adversarial/injection cases (`test_validation_gate.py`), the issue #23 `comment_id` idempotency invariants (`test_comment_id.py`), and the Bedrock-fallback dispatch/EMF-metric behavior with a mocked `invoke_model` (`test_bedrock_fallback.py`). Golden JSON lives under `tests/fixtures/expected/`.
|
||||||
|
- `lambdas/po/email_processor/tests/` — the deterministic PO parser suite: golden-file tests over real scrubbed `.eml` fixtures (17 single-line new-PO + 8 cancellations, exact `Decimal`-aware golden comparison via `parse_float=Decimal`), fail-closed validation-gate coverage for **every** gate reason code (fixture-driven for body-level triggers under `fixtures/adversarial/`, direct `validate()` unit tests for candidate-level mutations), real multi-line and comment/non-Coupa fallback fixtures under `fixtures/ai-fallback/`, dual line-ending (CRLF/LF) parse-identity, two-path `enrich_parsed`/`save_new_po` parity (the site-extractor stream-contract guard), fixture hygiene (`ses_auth` pass + scrub-marker leak sweep), and the Bedrock-fallback dispatch/EMF-metric behavior (`test_po_bedrock_fallback.py`).
|
||||||
|
|
||||||
Both roots are discovered by `pytest.ini` (`testpaths`).
|
All three roots are discovered by `pytest.ini` (`testpaths`).
|
||||||
|
|
||||||
## Scripts
|
## Scripts
|
||||||
|
|
||||||
|
|
@ -256,7 +261,10 @@ cdk/
|
||||||
wo_stack.py # Work order pipeline resources
|
wo_stack.py # Work order pipeline resources
|
||||||
lambdas/
|
lambdas/
|
||||||
po/ # PO pipeline Lambdas
|
po/ # PO pipeline Lambdas
|
||||||
email_processor/ # handler.py (Bedrock, fully AI)
|
email_processor/
|
||||||
|
handler.py # template-first + Bedrock fallback, EMF metric, merge writes
|
||||||
|
template_parser.py # pure deterministic Coupa parser + fail-closed validation gate
|
||||||
|
tests/ # golden-file + validation-gate + fallback-dispatch tests + fixtures
|
||||||
site_extractor/
|
site_extractor/
|
||||||
web_ui/
|
web_ui/
|
||||||
wo/ # WO pipeline Lambdas
|
wo/ # WO pipeline Lambdas
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,6 @@
|
||||||
# PO Template-First Parser — Design, Investigation & Progress
|
# PO Template-First Parser — Design, Investigation & Progress
|
||||||
|
|
||||||
> **Status:** in progress · **Branch:** `feat/po-template-parser` (stacked on PR #99) · **Last updated:** 2026-07-16
|
> **Status:** PR #1 implemented (extraction + gate + wiring + alarm + tests); PR #2 (derived-classifier factoring) not started · **Branch:** `feat/po-template-parser` (stacked on PR #99) · **Last updated:** 2026-07-16
|
||||||
>
|
>
|
||||||
> Living document for making the Coupa **purchase-order** email parser *template-first with LLM fallback*, mirroring the work-order (WO) processor's PR #99. Captures the investigation, the data we gathered, the decisions made, the current scaffold, and everything still to do.
|
> Living document for making the Coupa **purchase-order** email parser *template-first with LLM fallback*, mirroring the work-order (WO) processor's PR #99. Captures the investigation, the data we gathered, the decisions made, the current scaffold, and everything still to do.
|
||||||
|
|
||||||
|
|
@ -30,7 +30,8 @@ Two phases: a read-only multi-agent (ultracode) feasibility study over a 120-ema
|
||||||
|
|
||||||
### 2.1 Data access
|
### 2.1 Data access
|
||||||
- PO email bucket: `s3://po-ingest-emails-328440206208/inbound/` (AWS account **328440206208**, us-east-1).
|
- PO email bucket: `s3://po-ingest-emails-328440206208/inbound/` (AWS account **328440206208**, us-east-1).
|
||||||
- Reached via AWS profile **`amoussa-mgmt`** (SSO). The default CLI creds are the personal account `681986854588`, which is **denied** on the ingest buckets — a recurring gotcha.
|
- ~~Reached via AWS profile `amoussa-mgmt` (SSO); default CLI creds are the personal account `681986854588`~~ **Stale (corrected 2026-07-16 during the fixture harvest):** the default CLI session is now authenticated to **328440206208** directly — no `--profile` flag needed.
|
||||||
|
- ⚠️ **Corpus is aging out:** `inbound/` objects carry an S3 lifecycle expiration (~90-day rolling window; oldest object 2026-04-17 at harvest time, 3,422 objects vs 3,448 at triage). Any further harvesting should not be deferred long.
|
||||||
|
|
||||||
### 2.2 Full-bucket triage results (all 3,448 emails, 2026-07-16)
|
### 2.2 Full-bucket triage results (all 3,448 emails, 2026-07-16)
|
||||||
|
|
||||||
|
|
@ -65,6 +66,12 @@ Within **new_po**:
|
||||||
- Every captured value carries a trailing `\r`; a lone `\xa0` (nbsp) line sits before `Total`.
|
- Every captured value carries a trailing `\r`; a lone `\xa0` (nbsp) line sits before `Total`.
|
||||||
- Real amounts include thousands separators (`18,624.05`, `32,405.00`).
|
- Real amounts include thousands separators (`18,624.05`, `32,405.00`).
|
||||||
|
|
||||||
|
**Addendum (2026-07-16 fixture harvest):**
|
||||||
|
- Every per-line bullet run BEGINS with a `Supplier <name>` segment before `Need By` (the gate proves it byte-equals `supplier.name` on every item).
|
||||||
|
- Multi-item Lines blocks carry a `<qty> EA` evidence line before each description; the single-item Items summary carries an optional `<qty> <UNIT> x <price>` line (absent on some emails, e.g. new-po-14) which is the **source** for `quantity`/`unit`/`price` (the Lines-block `EA` line is a gate cross-check only).
|
||||||
|
- The Items summary contains **unit-price money tokens distinct from line amounts** (e.g. `50.0 EACH x 66.00` with line amount `8,356.00`) — money must be anchored on the `for <amt> <CCY>` / Total-block / summary-`x` contexts, never free money-shaped scanning; and line `amount ≠ qty×price` on partial quantities, so only `sum(lines) == total` is load-bearing.
|
||||||
|
- Body line endings are **decode-path dependent**: Coupa QP encodes `=0D`, so `policy.default` `get_content()` yields `\r\n` even after transport normalization, while other decode paths yield bare LF — the parser matches only `_clean`-ed lines and is tested against both representations.
|
||||||
|
|
||||||
### 2.4 cancellation layout
|
### 2.4 cancellation layout
|
||||||
|
|
||||||
Subject: `<SiteName> Purchase Order #<PO#> has been cancelled`. Body is a short dashes-delimited notice — **no `Status` label**, no line items. The only field the handler's `save_cancellation()` needs is `po_number`.
|
Subject: `<SiteName> Purchase Order #<PO#> has been cancelled`. Body is a short dashes-delimited notice — **no `Status` label**, no line items. The only field the handler's `save_cancellation()` needs is `po_number`.
|
||||||
|
|
@ -129,15 +136,17 @@ File: `lambdas/po/email_processor/template_parser.py` — pure module (no boto3/
|
||||||
- `validate(candidate, template_id, email_data) -> (bool, reason)` — fail closed.
|
- `validate(candidate, template_id, email_data) -> (bool, reason)` — fail closed.
|
||||||
- `try_deterministic_parse(email_data) -> (parsed|None, method, template_id, reason)` — entry point; `try/except` fails closed on any error.
|
- `try_deterministic_parse(email_data) -> (parsed|None, method, template_id, reason)` — entry point; `try/except` fails closed on any error.
|
||||||
|
|
||||||
**Handler integration (not yet wired):** after `authenticate_inbound_email()` + `parse_raw_email()`, call `try_deterministic_parse`; if `None`, `extract_with_claude`; then run the shared `enrich_parsed()`+classify stage on `parsed` regardless of path, then the existing `save_cancellation` / `save_revision` / `save_new_po` routing. Emit a `ParseMethod` EMF metric like WO.
|
**Handler integration (✅ wired in PR #1):** after `authenticate_inbound_email()` + `parse_raw_email()`, `try_deterministic_parse` runs first; if `None`, `extract_with_claude`; the shared `enrich_parsed()` stage runs on `parsed` regardless of path, then the existing `save_cancellation` / `save_revision` / `save_new_po` routing (untouched). The `ParseMethod` EMF metric is emitted immediately after the deterministic attempt — before any Bedrock call — so a Bedrock-side error still records the `ai_fallback` outcome.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 6. Fail-Closed Gate Rules
|
## 6. Fail-Closed Gate Rules
|
||||||
|
|
||||||
Reason codes currently emitted by `validate()` / `try_deterministic_parse()`:
|
Reason codes emitted by `validate()` / `try_deterministic_parse()` (closed set):
|
||||||
|
|
||||||
`subject_no_match` · `key_set_mismatch` · `derived_field_set` · `email_type_mismatch` · `missing_required_field` · `po_id_mismatch` · `unrecognized_status` · `multiline_unsupported` · `non_usd` · `new_po_not_implemented` · `extractor_raised` · (`ok` / `template` on success).
|
`subject_no_match` · `key_set_mismatch` · `derived_field_set` · `email_type_mismatch` · `missing_required_field` · `po_id_mismatch` · `unrecognized_status` · `multiline_unsupported` · `non_usd` · `amount_mismatch` · `anchor_violation` · `address_shape_invalid` · `bullet_label_unrecognized` · `unparseable_value` · `residual_artifact` · `extractor_raised` · (`ok` / `template` on success).
|
||||||
|
|
||||||
|
(`new_po_not_implemented` disappeared with the scaffold guard — both copies removed; a test asserts the string no longer exists in the module.)
|
||||||
|
|
||||||
**Implemented (both templates):**
|
**Implemented (both templates):**
|
||||||
- Known template only (`subject_no_match` otherwise).
|
- Known template only (`subject_no_match` otherwise).
|
||||||
|
|
@ -148,7 +157,14 @@ Reason codes currently emitted by `validate()` / `try_deterministic_parse()`:
|
||||||
|
|
||||||
**Implemented (new_po):** safe-status enum (`unrecognized_status`), single-line only (`multiline_unsupported`), USD only (`non_usd`).
|
**Implemented (new_po):** safe-status enum (`unrecognized_status`), single-line only (`multiline_unsupported`), USD only (`non_usd`).
|
||||||
|
|
||||||
**TODO (new_po value-level — see §8):** amount byte-equality + `sum(lines)==total`; duplicate-label anchoring (reject `Shipping=='None'`, require `Location Code:`, `supplier.name` contains `SEA HAVEN`, `ship_to.name != supplier.name`); bullet-metadata split by leading label; address shape (`state` null-or-USPS, `zip` null-or `^\d{5}(-\d{4})?$`); `_UNPARSEABLE` rejection; residual `\r`/`\xa0` rejection.
|
**Implemented (new_po value-level rules V1–V13 — ✅ done, one reason code each):** the gate **re-derives every byte proof from `email_data["body"]`** (never trusting extractor-carried state), with the six value-level reason codes:
|
||||||
|
- `amount_mismatch` — money fidelity (each Decimal re-serializes byte-identically to its re-derived source token, non-digit/comma border — the `18,624.05`→`624.05` kill switch), `sum(lines) == total` (exact Decimal; deliberately **no** `qty×price == amount` rule — corpus shows partial quantities), dual-`Total` byte-identity, and quantity/unit/price coherence vs the summary and `EA` evidence lines.
|
||||||
|
- `anchor_violation` — exactly 2× `Supplier`/`Shipping`/`Total`, first `Shipping` is the literal `None` placeholder, section ordering, `supplier.name` contains `SEA HAVEN` and byte-equals the detail-block name + every item's `Supplier` bullet segment, `ship_to.name != supplier.name`.
|
||||||
|
- `address_shape_invalid` — city line fullmatches `^City, ST ZIP$` immediately before `United States`, state in the frozen USPS set, zip `^\d{5}(-\d{4})?$` on the RAW pre-enrichment value (a short zip fails closed to the LLM path where the shared `pad_zip` repairs it — the gate never predicts enrichment).
|
||||||
|
- `bullet_label_unrecognized` — every U+2022 segment carries a recognized leading label from the closed set; core labels exactly once, `Part Number` at most once.
|
||||||
|
- `unparseable_value` — recursive walk: no `_UNPARSEABLE` sentinel leaf.
|
||||||
|
- `residual_artifact` — recursive walk: no `\r`/`\xa0` in any string leaf.
|
||||||
|
Plus `missing_required_field` for the required labeled fields (per-item metadata, `Location Code:` sourcing, no invented `Attn:`) and `po_id_mismatch` for the body `PO ID` / heading / `orders/<id>`-URL identity proofs.
|
||||||
|
|
||||||
**Precondition (handled a layer earlier):** `authenticate_inbound_email()` (SES `dkim=pass` for `amazon.coupahost.com`) must pass before the parser runs — the 35 non-Coupa emails are rejected there.
|
**Precondition (handled a layer earlier):** `authenticate_inbound_email()` (SES `dkim=pass` for `amazon.coupahost.com`) must pass before the parser runs — the 35 non-Coupa emails are rejected there.
|
||||||
|
|
||||||
|
|
@ -179,26 +195,33 @@ Reason codes currently emitted by `validate()` / `try_deterministic_parse()`:
|
||||||
- [x] `classify_template` for both templates (real subject regexes).
|
- [x] `classify_template` for both templates (real subject regexes).
|
||||||
- [x] `coupa_cancellation` extract + gate — **implemented & smoke-tested** (parses a real cancellation → `po_number`, `email_type=cancellation`).
|
- [x] `coupa_cancellation` extract + gate — **implemented & smoke-tested** (parses a real cancellation → `po_number`, `email_type=cancellation`).
|
||||||
- [x] `coupa_new_po` classify + structural/status/single-line/USD gate.
|
- [x] `coupa_new_po` classify + structural/status/single-line/USD gate.
|
||||||
- [x] Scaffold guard: `extract_new_po` is a documented TODO → new_po returns `new_po_not_implemented` and falls back to the LLM (safe).
|
- [x] ~~Scaffold guard~~ — removed (both copies) with the PR #1 implementation below.
|
||||||
- [x] `ruff check` + `ruff format --check` pass.
|
- [x] `ruff check` + `ruff format --check` pass.
|
||||||
- Smoke test (real corpus): cancellation → `template`; new_po → `ai_fallback/new_po_not_implemented`; comment & non-Coupa → `ai_fallback/subject_no_match`.
|
- [x] **PR #1 implementation (2026-07-16):**
|
||||||
|
- [x] `extract_new_po` — section-windowed, anchor-disciplined, label-keyed bullet split, `Decimal` money, representation-agnostic line handling. All 17 harvested single-line new-POs parse to `template/ok`; the 3 real multi-line ones extract faithfully (`sum==total`) then gate-reject with `multiline_unsupported`.
|
||||||
> Not yet committed; working-tree only.
|
- [x] Value-level gate rules V1–V13 (§6) — every byte proof re-derived from the body.
|
||||||
|
- [x] Handler wiring: `try_deterministic_parse` first, Bedrock fallback on `None`, shared `enrich_parsed()` on both paths, `save_*` routing untouched.
|
||||||
|
- [x] `ParseMethod` EMF metric set (`Seahaven/PoIngest`/`ParseOutcome`, dims `[["ParseMethod"],["ParseMethod","TemplateId"]]`, `ReasonCode`/`po_number` ride-alongs), emitted before the Bedrock call.
|
||||||
|
- [x] Fallback-rate alarm in `cdk/po_stack.py`, retuned for ~57/day (6h periods · ≥8 volume floor · >20% · 2-of-4) — `npx cdk synth po-ingest` green.
|
||||||
|
- [x] `cdk/po_stack.py` bundling `cp` now includes `template_parser.py` (was a deploy-time ImportError waiting to happen).
|
||||||
|
- [x] Offline test suite `lambdas/po/email_processor/tests/` (132 tests): goldens for all 25 positive fixtures, every §6 reason code covered, dual line-ending identity, two-path enrich/save parity, fixture hygiene (ses_auth + leak-sweep markers), Bedrock dispatch/EMF. Full-root pytest: 354 green.
|
||||||
|
- [x] Committed sanitized `.eml` fixture corpus (header + body scrub, per-file digit cipher, narrow `.gitignore` exception scoped to `lambdas/po/email_processor/tests/fixtures/`).
|
||||||
|
- [x] README updated (PO flow, parser section, alarm numbers + justification, tests, repo layout).
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 9. TODO / Path Forward
|
## 9. TODO / Path Forward
|
||||||
|
|
||||||
**PR #1 — extraction template + wiring**
|
**PR #1 — extraction template + wiring**
|
||||||
- [ ] Implement `extract_new_po`: labeled single-occurrence fields; duplicate-label anchoring (`Supplier`/`Shipping`/`Total`); `ship_to` by sentinel anchors; `line_items` split on `•` by leading label; `Decimal` amounts; `coupa_category` verbatim.
|
- [x] Implement `extract_new_po`: labeled single-occurrence fields; duplicate-label anchoring (`Supplier`/`Shipping`/`Total`); `ship_to` by sentinel anchors; `line_items` split on `•` by leading label; `Decimal` amounts; `coupa_category` verbatim.
|
||||||
- [ ] Implement the new_po value-level gate rules (§6 TODO) and remove the scaffold guard.
|
- [x] Implement the new_po value-level gate rules (§6 V1–V13) and remove the scaffold guard (both copies).
|
||||||
- [ ] Handler integration: `try_deterministic_parse` first, Bedrock fallback on `None`; route through existing `save_*` branches.
|
- [x] Handler integration: `try_deterministic_parse` first, Bedrock fallback on `None`; route through existing `save_*` branches.
|
||||||
- [ ] Emit `ParseMethod`-only EMF set (namespace `Seahaven/PoIngest`, dims `[["ParseMethod"], ["ParseMethod","TemplateId"]]`).
|
- [x] Emit `ParseMethod`-only EMF set (namespace `Seahaven/PoIngest`, dims `[["ParseMethod"], ["ParseMethod","TemplateId"]]`).
|
||||||
- [ ] Fallback-rate CloudWatch alarm in `cdk/po_stack.py` — **retune thresholds for ~57/day volume** (PO is far lower than WO's ~760/day; the WO 15-min/sample-floor settings won't fit).
|
- [x] Fallback-rate CloudWatch alarm in `cdk/po_stack.py` — retuned for ~57/day: 6h periods, `IF((fb+tmpl)>=8, …)` volume floor, >20% threshold, eval 4 / datapoints 2 (see the po_stack comment + README for the arithmetic; `npx cdk synth po-ingest` gated).
|
||||||
- [ ] Offline tests + fixtures: good cancellation + good new_po; one adversarial per gate rule (thousands-sep, dup-label swap, Part-Number bullet shift, multiline, non-USD, bad status); comment + non-Coupa assertions; contract⊆prompt parity test.
|
- [x] Offline tests + fixtures: goldens for every positive fixture; one adversarial per gate rule (thousands-sep, dup-label swap, Part-Number bullet shift pair, multiline, non-USD, bad status, PO-id/URL mismatch, unlabeled/duplicate bullets, prompt injection); comment + non-Coupa assertions; contract⊆prompt parity test.
|
||||||
- [ ] README + Confluence "AWS Architecture Map" updates.
|
- [x] README updated. Confluence "AWS Architecture Map" update — ⚠️ outstanding (must land with the merge; PO subgraph gains the template-parser stage + fallback alarm).
|
||||||
- [ ] `cross_reviewer` (GPT-4.1) pass — handler contract change (Sea Haven policy).
|
- [ ] `cross_review.py` (GPT-4.1) pass — handler contract change (Sea Haven policy). ⚠️ Outstanding until run against the final PR diff — do not merge without it.
|
||||||
- [ ] `ruff check` + `ruff format` + `pytest` green in CI.
|
- [x] `ruff check` + `ruff format` + `pytest` green locally (354 tests; CI must confirm).
|
||||||
|
|
||||||
**PR #2 — derived-classifier factoring (follow-up)**
|
**PR #2 — derived-classifier factoring (follow-up)**
|
||||||
- [ ] Move `trade`/`site_code`/`fiscal_year` into shared `enrich_parsed()`.
|
- [ ] Move `trade`/`site_code`/`fiscal_year` into shared `enrich_parsed()`.
|
||||||
|
|
@ -212,10 +235,15 @@ Reason codes currently emitted by `validate()` / `try_deterministic_parse()`:
|
||||||
|
|
||||||
1. **Goal — cost, latency, or determinism?** Determinism (same input → same output) is the strongest justification for the derived-classifier port regardless of per-email cost.
|
1. **Goal — cost, latency, or determinism?** Determinism (same input → same output) is the strongest justification for the derived-classifier port regardless of per-email cost.
|
||||||
2. **Classifier factoring now or deferred?** Recommendation: extraction template first (PR #1), factoring as shadow-logged PR #2.
|
2. **Classifier factoring now or deferred?** Recommendation: extraction template first (PR #1), factoring as shadow-logged PR #2.
|
||||||
3. **Strict single-line/USD-only v1 acceptable?** Triage says it covers ~99.8% of new_po, so yes — but confirm.
|
3. **Strict single-line/USD-only v1 acceptable?** ✅ **Resolved (PR #1): yes.** Covers ~99.8% of new_po; multi-line and non-USD fail closed with honest reason codes.
|
||||||
4. **Confirm the real cancellation `Status` spelling** — `handler.py` hardcodes `CANCELLED_STATUS='Cancelled'`; the cancellation body doesn't restate it. Verify against a known-cancelled PO in the table.
|
4. **Confirm the real cancellation `Status` spelling** — `handler.py` hardcodes `CANCELLED_STATUS='Cancelled'`; the cancellation body doesn't restate it. Verify against a known-cancelled PO in the table. (Doesn't affect parser output — `save_cancellation` hardcodes the string.)
|
||||||
5. **Comments** — should `New Comment on Purchase Order` emails create/update PO records at all, or be dropped? (Out of scope for the parser, but a real data-quality decision.)
|
5. **Comments** — should `New Comment on Purchase Order` emails create/update PO records at all, or be dropped? (Out of scope for the parser, but a real data-quality decision.)
|
||||||
6. **Repo fixtures** — commit a sanitized PO corpus (like WO's checked-in `.eml` fixtures), or keep it out of the repo?
|
6. **Repo fixtures** — ✅ **Resolved (PR #1): committed sanitized corpus** under `lambdas/po/email_processor/tests/fixtures/` (35 scrubbed `.eml`: 17 single-line new-PO, 8 cancellations, 4 comments, 3 non-Coupa, 3 multi-line; plus 17 synthetic adversarial mutations). Transport/auth headers scrubbed to same-shape placeholders (structure preserved, `ses_auth` still passes), per-file digit cipher on identifiers, amounts remapped with `sum==total` re-established, and a narrow `.gitignore` exception scoped to the PO fixtures path (never blanket `*.eml`). Source S3 keys deliberately unrecorded (they ARE the SES receipt tokens the scrub replaced).
|
||||||
|
|
||||||
|
**Fixture-build pins recorded during PR #1 (locked by goldens):**
|
||||||
|
- **`ship_to.street`/`ship_to.address` join convention:** multi-line segments joined with `'\n'`; `address` = the lines from `ship_to.name` through `United States` inclusive (this is what `enrich_parsed` promotes to `ship_to_raw` on the site-extractor stream).
|
||||||
|
- **`quantity`/`unit`/`price` source:** the Items-summary `<qty> <UNIT> x <price>` line (e.g. `1.0 EACH x 55,206.00`); the Lines-block `<qty> EA` line is gate evidence only (V13 numeric cross-check). Emails without a summary line (e.g. new-po-14) leave all three `null` — nullable by contract.
|
||||||
|
- **V10 URL-id sweep outcome:** all 20 harvested new_po emails satisfy `orders/<id> == po_number` digit suffix — the strict V10 check stays live (no demotion needed); a corpus-sweep test locks it.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
|
||||||
Loading…
Add table
Reference in a new issue