mirror of
https://github.com/Sea-Haven-Industries/procurement-ingest.git
synced 2026-10-07 16:08:54 +00:00
feature: Add PO template parser scaffold and design doc
Mirror WO PR #99's template-first approach for the Coupa PO processor. Two templates identified from a full 3,448-email triage: - coupa_new_po (95.5%): scaffolded; fails closed to the LLM until extract_new_po lands. - coupa_cancellation (2.9%): implemented. Nested contract with recursive validation, Decimal money, and a fail-closed gate. Derived fields (site_code/trade/fiscal_year) are deferred to a shared post-stage. Comments, revisions, multi-line, and non-USD emails fall back to Bedrock. docs/po-template-parser.md records the investigation, decisions, and remaining work. Signed-off-by: Adam Moussa <166072409+amoussa1229@users.noreply.github.com>
This commit is contained in:
Adam Moussa
parent
042a03b3a4
commit
fc5b1bc0ba
3 changed files with 636 additions and 0 deletions
2
.gitignore
vendored
2
.gitignore
vendored
|
|
@ -14,3 +14,5 @@ cdk.out/
|
|||
# scratch, not the vetted fixture corpus).
|
||||
!lambdas/wo/email_processor/tests/fixtures/**/*.eml
|
||||
lambdas/*/*/package/
|
||||
.idea/
|
||||
# JetBrains user settings directory
|
||||
|
|
|
|||
240
docs/po-template-parser.md
Normal file
240
docs/po-template-parser.md
Normal file
|
|
@ -0,0 +1,240 @@
|
|||
# PO Template-First Parser — Design, Investigation & Progress
|
||||
|
||||
> **Status:** in progress · **Branch:** `feat/po-template-parser` (stacked on PR #99) · **Last updated:** 2026-07-16
|
||||
>
|
||||
> Living document for making the Coupa **purchase-order** email parser *template-first with LLM fallback*, mirroring the work-order (WO) processor's PR #99. Captures the investigation, the data we gathered, the decisions made, the current scaffold, and everything still to do.
|
||||
|
||||
---
|
||||
|
||||
## 1. Context & Goal
|
||||
|
||||
The PO email processor (`lambdas/po/email_processor/handler.py`) sends **every** inbound Coupa/Amazon purchase-order email to an LLM (Claude Haiku 4.5 on Bedrock) for structured extraction. WO PR #99 established the pattern we want to replicate here:
|
||||
|
||||
> Run a **deterministic template parser first**. It returns a result only when the email provably conforms to a known template and passes a **fail-closed validation gate**; otherwise the handler falls back to the Bedrock LLM. This removes the LLM from the hot path for routine traffic while keeping full AI coverage for anything unexpected.
|
||||
|
||||
**Goal:** apply the same template-first + fail-closed-fallback approach to the PO processor, without regressing data quality or the downstream `po-ingest-site-extractor` pipeline (which reads `site_code`/`ship_to` off the `purchase-orders` DynamoDB stream).
|
||||
|
||||
**Reference implementation:** `lambdas/wo/email_processor/template_parser.py` (PR #99).
|
||||
|
||||
### Why this is harder than WO
|
||||
- The PO contract is ~40 fields with **nested** objects (`supplier{}`, `ship_to{}`, `line_items[]`) vs WO's flat 16.
|
||||
- Much of the PO extraction prompt is **derived classification** (`trade`, `site_code`, `fiscal_year`) already expressed as deterministic English rules.
|
||||
- Money fields require `Decimal` (DynamoDB rejects floats).
|
||||
- `email_type` is **not** in the subject (unlike WO's two subject templates).
|
||||
|
||||
---
|
||||
|
||||
## 2. Investigation
|
||||
|
||||
Two phases: a read-only multi-agent (ultracode) feasibility study over a 120-email sample, then a full-bucket triage over **all 3,448** inbound emails to get real distribution numbers.
|
||||
|
||||
### 2.1 Data access
|
||||
- PO email bucket: `s3://po-ingest-emails-328440206208/inbound/` (AWS account **328440206208**, us-east-1).
|
||||
- Reached via AWS profile **`amoussa-mgmt`** (SSO). The default CLI creds are the personal account `681986854588`, which is **denied** on the ingest buckets — a recurring gotcha.
|
||||
|
||||
### 2.2 Full-bucket triage results (all 3,448 emails, 2026-07-16)
|
||||
|
||||
Method: parallel ranged-GET (`bytes=0-12000`) of every object, then local classification (quoted-printable bodies, decodable offline).
|
||||
|
||||
| Email kind | Count | % | Notes |
|
||||
|---|---:|---:|---|
|
||||
| **new_po** — `***Copy for Reference*** New Purchase Order <PO#> has been issued` | 3,294 | **95.5%** | Single uniform Coupa layout |
|
||||
| **cancellation** — `<Site> Purchase Order #<PO#> has been cancelled` | 100 | **2.9%** | Distinct subject + minimal body |
|
||||
| **comment** — `New Comment on Purchase Order for Amazon` | 19 | 0.55% | Type NOT in handler enum (see §7) |
|
||||
| **non-Coupa senders** (human replies, WO mail, an AWS SES setup notice) | 35 | 1.0% | Rejected at ses_auth before parse |
|
||||
| **revision** | **0** | 0% | No distinct revision emails exist |
|
||||
|
||||
Within **new_po**:
|
||||
- **Status** values: `Issued - Created` (2,223) and `Issued - Scheduled for email` (1,071) — only these two.
|
||||
- **Multi-line-item:** 6 / 3,294 (**0.18%**), all ≤3 items.
|
||||
- **Non-USD:** 0 (after excluding unit-token false positives like `EACH`/`HOUR`).
|
||||
- So the single-line / USD scope covers **~99.8%** of new_po traffic.
|
||||
|
||||
### 2.3 new_po layout (the one template that matters)
|
||||
|
||||
`multipart/alternative`; the **text/plain** part is a stable, label-delimited flattening:
|
||||
- Top: `Amazon Purchase Order #<PO#>`, then `Submitted By`, `On Behalf Of`, `Supplier`, `Total`, `Items`.
|
||||
- `More Detail` block: `PO ID`, `Department`, `Status`, `Last Opened`, `Order Date`, `Acknowledged At`, `Revision Date`, `Payment Term`, `Req #`, `Shipping`.
|
||||
- `Supplier` detail block (address).
|
||||
- `Shipping` detail block (ship-to address, `Location Code:`, `Attn:`).
|
||||
- `Lines` section: one block per line item; per-line metadata is **U+2022 (`•`) delimited**: `Need By` · `Category` · `Account` · `Period` [· optional `Part Number`].
|
||||
|
||||
**Structural hazards found in real data:**
|
||||
- `Supplier`, `Shipping`, `Total` labels **each appear twice** (summary placeholder + detail block); the *first* `Shipping` value is literally `None`.
|
||||
- Optional `Part Number` bullet segment appears in a minority of emails, inserted between `Category` and `Account` → shifts any positional splitter.
|
||||
- Every captured value carries a trailing `\r`; a lone `\xa0` (nbsp) line sits before `Total`.
|
||||
- Real amounts include thousands separators (`18,624.05`, `32,405.00`).
|
||||
|
||||
### 2.4 cancellation layout
|
||||
|
||||
Subject: `<SiteName> Purchase Order #<PO#> has been cancelled`. Body is a short dashes-delimited notice — **no `Status` label**, no line items. The only field the handler's `save_cancellation()` needs is `po_number`.
|
||||
|
||||
---
|
||||
|
||||
## 3. Key Decisions
|
||||
|
||||
1. **Build two templates**, not one and not three:
|
||||
- `coupa_new_po` (95.5%) — the main workload.
|
||||
- `coupa_cancellation` (2.9%) — trivial and low-risk (just `po_number`); we have 100 real examples.
|
||||
- Together ≈ **98.4%** of all mail deterministically handled/routed.
|
||||
2. **No revision template** — 0 distinct revision emails in 3,448. The handler's `revision` type essentially never arrives as its own email.
|
||||
3. **`email_type` never defaults to new_po.** It is emitted only when the exact new_po subject regex matches **and** `po_status` ∈ the two confirmed-safe strings. Anything else → LLM. (Misrouting a cancellation to new_po would silently defeat the sticky-`Cancelled` guard.)
|
||||
4. **Fail closed, always.** Any miss / invalid / exception returns `None` and falls back to the LLM. A failure is never a parsed result. (Copies WO's `try/except` posture.)
|
||||
5. **`Decimal` for all money**, with thousands-separator stripping, to match the LLM path's `parse_float=Decimal`.
|
||||
6. **Derived fields (`site_code`, `trade`, `fiscal_year`) are NOT computed in the parser.** They are left `None`; a shared post-stage (`enrich_parsed()`, the `pad_zip` precedent) fills them **identically** on both the template and LLM paths, so the gate judges *extraction fidelity only* and both paths write byte-identical shapes downstream. `coupa_category` is a **verbatim** label capture, not a classifier, and IS extracted.
|
||||
7. **Structural nested contract** with recursive key-set validation (unlike WO's flat tuple).
|
||||
8. **Two-PR split:** (1) extraction template + gate + handler wiring; (2) derived-classifier factoring, shadow-logged before it becomes authoritative.
|
||||
|
||||
---
|
||||
|
||||
## 4. Contract
|
||||
|
||||
Top-level `CONTRACT_KEYS` (23) plus three nested sub-tuples. Taken from the `EXTRACTION_PROMPT` in `handler.py`.
|
||||
|
||||
```
|
||||
CONTRACT_KEYS = (
|
||||
email_type, po_number, po_status, source_system, submitted_by, on_behalf_of,
|
||||
order_date, revision_date, last_opened, acknowledged_at, payment_terms,
|
||||
requisition_number, department, view_order_url, supplier, site_code, ship_to,
|
||||
total_amount, currency, fiscal_year, trade, coupa_category, line_items,
|
||||
)
|
||||
SUPPLIER_KEYS = (name,)
|
||||
SHIP_TO_KEYS = (name, address, street, city, state, zip, location_code, attn)
|
||||
LINE_ITEM_KEYS = (description, amount, currency, need_by, category,
|
||||
account_code, period, quantity, unit, price)
|
||||
```
|
||||
|
||||
| Class | Keys |
|
||||
|---|---|
|
||||
| **Verbatim / labeled extract** | po_number, po_status, submitted_by, on_behalf_of, order_date, revision_date, last_opened, acknowledged_at, payment_terms, requisition_number, department, view_order_url, supplier.name, ship_to.location_code, ship_to.attn, coupa_category, all `line_items[*]` |
|
||||
| **Extract with care** | total_amount, currency, ship_to.name/street/city/state/zip (`Decimal`; sentinel-anchored address) |
|
||||
| **Constant** | source_system = `"coupa"` |
|
||||
| **Derived — post-stage, NOT parser, NOT gate-validated** | site_code, trade, fiscal_year |
|
||||
|
||||
Handler enrichment metadata (`raw_s3_key`, `processed_at`, `data_source`, `email_subject`, `ship_to_raw`, top-level `state`) is added by `enrich_parsed()` and is **not** part of the parser contract.
|
||||
|
||||
A parity test (mirroring WO's `test_contract_keys_match_extraction_prompt`) should assert the flattened contract key set is a subset of the keys quoted in `EXTRACTION_PROMPT`.
|
||||
|
||||
---
|
||||
|
||||
## 5. Parser Design
|
||||
|
||||
File: `lambdas/po/email_processor/template_parser.py` — pure module (no boto3/network), mirroring WO idioms.
|
||||
|
||||
- `classify_template(email_data) -> (template_id, reason)` — exact subject regex → `coupa_new_po` | `coupa_cancellation` | `unknown`.
|
||||
- `extract_new_po` / `extract_cancellation` — build the nested candidate; leave derived fields `None`.
|
||||
- `_empty_candidate()` / `_normalize()` — build & normalize the **nested** skeleton (`supplier={}`, `ship_to={}`, `line_items=[one item]`).
|
||||
- `_clean()` — strip trailing `\r` + `\xa0`, map `"None"`/empty → `None`.
|
||||
- `_to_decimal()` — thousands-separator-safe `Decimal`, `_UNPARSEABLE` sentinel on failure.
|
||||
- `validate(candidate, template_id, email_data) -> (bool, reason)` — fail closed.
|
||||
- `try_deterministic_parse(email_data) -> (parsed|None, method, template_id, reason)` — entry point; `try/except` fails closed on any error.
|
||||
|
||||
**Handler integration (not yet wired):** after `authenticate_inbound_email()` + `parse_raw_email()`, call `try_deterministic_parse`; if `None`, `extract_with_claude`; then run the shared `enrich_parsed()`+classify stage on `parsed` regardless of path, then the existing `save_cancellation` / `save_revision` / `save_new_po` routing. Emit a `ParseMethod` EMF metric like WO.
|
||||
|
||||
---
|
||||
|
||||
## 6. Fail-Closed Gate Rules
|
||||
|
||||
Reason codes currently emitted by `validate()` / `try_deterministic_parse()`:
|
||||
|
||||
`subject_no_match` · `key_set_mismatch` · `derived_field_set` · `email_type_mismatch` · `missing_required_field` · `po_id_mismatch` · `unrecognized_status` · `multiline_unsupported` · `non_usd` · `new_po_not_implemented` · `extractor_raised` · (`ok` / `template` on success).
|
||||
|
||||
**Implemented (both templates):**
|
||||
- Known template only (`subject_no_match` otherwise).
|
||||
- Exact nested key-set at every level (`key_set_mismatch`).
|
||||
- Derived fields must be unset by the parser (`derived_field_set`).
|
||||
- `email_type` in enum and == template's expected type (`email_type_mismatch`).
|
||||
- `po_number` valid shape `^[A-Z0-9]{1,6}-\d+$` **and** byte-equals the subject id (`po_id_mismatch` / `missing_required_field`).
|
||||
|
||||
**Implemented (new_po):** safe-status enum (`unrecognized_status`), single-line only (`multiline_unsupported`), USD only (`non_usd`).
|
||||
|
||||
**TODO (new_po value-level — see §8):** amount byte-equality + `sum(lines)==total`; duplicate-label anchoring (reject `Shipping=='None'`, require `Location Code:`, `supplier.name` contains `SEA HAVEN`, `ship_to.name != supplier.name`); bullet-metadata split by leading label; address shape (`state` null-or-USPS, `zip` null-or `^\d{5}(-\d{4})?$`); `_UNPARSEABLE` rejection; residual `\r`/`\xa0` rejection.
|
||||
|
||||
**Precondition (handled a layer earlier):** `authenticate_inbound_email()` (SES `dkim=pass` for `amazon.coupahost.com`) must pass before the parser runs — the 35 non-Coupa emails are rejected there.
|
||||
|
||||
---
|
||||
|
||||
## 7. Scope — In / Out
|
||||
|
||||
**IN (template-parsed):**
|
||||
- Single-line-item, USD, `new_po` with the exact subject and a safe `Status`.
|
||||
- `cancellation` (subject match → `po_number`).
|
||||
|
||||
**OUT (always LLM fallback / rejected):**
|
||||
- **Comments** (19) — `New Comment on Purchase Order` is an `email_type` the handler enum (`new_po`/`revision`/`cancellation`) does not model. The LLM currently shoehorns these. ⚠️ **Pre-existing data-quality gap, independent of this work** — decide separately whether comments should create/update PO records at all.
|
||||
- **Revisions** — no template (0 emails).
|
||||
- **Multi-line-item new_po** (0.18%) — `Lines`-array structure unobserved; `multiline_unsupported`.
|
||||
- **Non-USD new_po** (0 observed) — path unexercised; `non_usd`.
|
||||
- **Non-Coupa senders** (1.0%) — rejected at ses_auth.
|
||||
|
||||
---
|
||||
|
||||
## 8. Progress
|
||||
|
||||
- [x] Read-only ultracode feasibility investigation (8 agents, GO-with-conditions).
|
||||
- [x] Full-bucket triage over all 3,448 emails → distribution + scope numbers (§2.2).
|
||||
- [x] Confirmed cancellation & comment body layouts against real emails.
|
||||
- [x] **Scaffold** `template_parser.py` on `feat/po-template-parser`:
|
||||
- [x] Nested contract + recursive `_empty_candidate`/`_normalize`.
|
||||
- [x] `classify_template` for both templates (real subject regexes).
|
||||
- [x] `coupa_cancellation` extract + gate — **implemented & smoke-tested** (parses a real cancellation → `po_number`, `email_type=cancellation`).
|
||||
- [x] `coupa_new_po` classify + structural/status/single-line/USD gate.
|
||||
- [x] Scaffold guard: `extract_new_po` is a documented TODO → new_po returns `new_po_not_implemented` and falls back to the LLM (safe).
|
||||
- [x] `ruff check` + `ruff format --check` pass.
|
||||
- Smoke test (real corpus): cancellation → `template`; new_po → `ai_fallback/new_po_not_implemented`; comment & non-Coupa → `ai_fallback/subject_no_match`.
|
||||
|
||||
> Not yet committed; working-tree only.
|
||||
|
||||
---
|
||||
|
||||
## 9. TODO / Path Forward
|
||||
|
||||
**PR #1 — extraction template + wiring**
|
||||
- [ ] Implement `extract_new_po`: labeled single-occurrence fields; duplicate-label anchoring (`Supplier`/`Shipping`/`Total`); `ship_to` by sentinel anchors; `line_items` split on `•` by leading label; `Decimal` amounts; `coupa_category` verbatim.
|
||||
- [ ] Implement the new_po value-level gate rules (§6 TODO) and remove the scaffold guard.
|
||||
- [ ] Handler integration: `try_deterministic_parse` first, Bedrock fallback on `None`; route through existing `save_*` branches.
|
||||
- [ ] Emit `ParseMethod`-only EMF set (namespace `Seahaven/PoIngest`, dims `[["ParseMethod"], ["ParseMethod","TemplateId"]]`).
|
||||
- [ ] Fallback-rate CloudWatch alarm in `cdk/po_stack.py` — **retune thresholds for ~57/day volume** (PO is far lower than WO's ~760/day; the WO 15-min/sample-floor settings won't fit).
|
||||
- [ ] Offline tests + fixtures: good cancellation + good new_po; one adversarial per gate rule (thousands-sep, dup-label swap, Part-Number bullet shift, multiline, non-USD, bad status); comment + non-Coupa assertions; contract⊆prompt parity test.
|
||||
- [ ] README + Confluence "AWS Architecture Map" updates.
|
||||
- [ ] `cross_reviewer` (GPT-4.1) pass — handler contract change (Sea Haven policy).
|
||||
- [ ] `ruff check` + `ruff format` + `pytest` green in CI.
|
||||
|
||||
**PR #2 — derived-classifier factoring (follow-up)**
|
||||
- [ ] Move `trade`/`site_code`/`fiscal_year` into shared `enrich_parsed()`.
|
||||
- [ ] Run in **shadow mode**: compute Python values, log Python-vs-LLM disagreement via EMF, keep LLM authoritative during a bake period.
|
||||
- [ ] Harden `site_code` (7-shape + skip-list, incl. multi-hop ATTN like `CBRE - RME - DLI6`) and `trade` against the full sample with per-shape fixtures.
|
||||
- [ ] Only after acceptable agreement: make Python authoritative and drop the fields from `EXTRACTION_PROMPT`.
|
||||
|
||||
---
|
||||
|
||||
## 10. Open Questions
|
||||
|
||||
1. **Goal — cost, latency, or determinism?** Determinism (same input → same output) is the strongest justification for the derived-classifier port regardless of per-email cost.
|
||||
2. **Classifier factoring now or deferred?** Recommendation: extraction template first (PR #1), factoring as shadow-logged PR #2.
|
||||
3. **Strict single-line/USD-only v1 acceptable?** Triage says it covers ~99.8% of new_po, so yes — but confirm.
|
||||
4. **Confirm the real cancellation `Status` spelling** — `handler.py` hardcodes `CANCELLED_STATUS='Cancelled'`; the cancellation body doesn't restate it. Verify against a known-cancelled PO in the table.
|
||||
5. **Comments** — should `New Comment on Purchase Order` emails create/update PO records at all, or be dropped? (Out of scope for the parser, but a real data-quality decision.)
|
||||
6. **Repo fixtures** — commit a sanitized PO corpus (like WO's checked-in `.eml` fixtures), or keep it out of the repo?
|
||||
|
||||
---
|
||||
|
||||
## 11. Risk Register (high-severity)
|
||||
|
||||
| Risk | Mitigation |
|
||||
|---|---|
|
||||
| **Cancellation misrouted to new_po** → sticky-`Cancelled` guard defeated, PO stays active | `email_type` closed-set gate; never default-to-new_po; only emit on exact subject + safe status |
|
||||
| **Thousands-separator truncation** (`18,624.05`→`624.05`, ~30× too small, passes naive checks) | Amount byte-equality: re-serialize `Decimal` to source grouped format; reject if token bordered by digit/comma; `sum(lines)==total` |
|
||||
| **Multi-line structure unobserved** | Hard-fail `len(line_items)!=1` → LLM; profile real multi-line before supporting |
|
||||
| **Derived-field silent divergence** (wrong `trade`/`site_code` still well-formed; `site_code` rules already incomplete) | Factor into shared post-stage; keep OFF the gate; shadow-log before authoritative; per-shape fixtures |
|
||||
| **Duplicate-label first-match** (`Supplier`/`Shipping`/`Total` twice; first `Shipping`=`None`) | Occurrence/sentinel anchoring; gate rejects `Shipping=='None'`, requires `Location Code:`, `supplier.name` has `SEA HAVEN`, `ship_to.name != supplier.name` |
|
||||
|
||||
---
|
||||
|
||||
## Appendix — References
|
||||
|
||||
- WO reference parser: `lambdas/wo/email_processor/template_parser.py` (PR #99).
|
||||
- PO handler + `EXTRACTION_PROMPT`: `lambdas/po/email_processor/handler.py`.
|
||||
- PO stack (alarms, IAM, Bedrock): `cdk/po_stack.py`.
|
||||
- Bedrock model: inference profile `us.anthropic.claude-haiku-4-5-20251001-v1:0`.
|
||||
- PO email bucket: `s3://po-ingest-emails-328440206208/inbound/` (account 328440206208, profile `amoussa-mgmt`).
|
||||
394
lambdas/po/email_processor/template_parser.py
Normal file
394
lambdas/po/email_processor/template_parser.py
Normal file
|
|
@ -0,0 +1,394 @@
|
|||
"""Deterministic template parser for Coupa purchase order emails.
|
||||
|
||||
Pure module: no boto3, no network. Runs ahead of the AI extraction path in the
|
||||
purchase-order email processor. Only returns a parsed result when it is proven
|
||||
conformant to one of the known Coupa templates; otherwise it fails closed and
|
||||
signals the caller to fall back to the Bedrock AI extractor.
|
||||
|
||||
Mirrors the WO parser idioms (lambdas/wo/email_processor/template_parser.py):
|
||||
classify_template -> extract -> validate (FAIL CLOSED) -> try_deterministic_parse
|
||||
returning (parsed|None, method, template_id, reason). A failure is NEVER a
|
||||
parsed result.
|
||||
|
||||
Full-bucket triage of all 3,448 inbound emails (2026-07-16) fixed the scope:
|
||||
|
||||
T1 coupa_new_po -- 3,294 / 3,448 (95.5%). Subject
|
||||
"***Copy for Reference*** New Purchase Order <PO#> has been issued".
|
||||
multipart/alternative; the text/plain part is a stable label-delimited
|
||||
layout (PO ID / Status / Order Date / Revision Date / Payment Term / Req # /
|
||||
Submitted By / On Behalf Of / Supplier block / Shipping block w/ Location
|
||||
Code + Attn / a Lines section whose per-line metadata is U+2022-delimited:
|
||||
Need By / Category / Account / Period [/ optional Part Number]).
|
||||
Maps to email_type "new_po". 99.8% single-line-item, 100% USD.
|
||||
|
||||
T2 coupa_cancellation -- 100 / 3,448 (2.9%). Subject
|
||||
"<SiteName> Purchase Order #<PO#> has been cancelled". Minimal body; the
|
||||
only field the handler's save_cancellation() needs is po_number.
|
||||
Maps to email_type "cancellation".
|
||||
|
||||
DELIBERATELY OUT OF SCOPE -> always AI fallback (never template-parsed):
|
||||
* "New Comment on Purchase Order for Amazon" (19 / 3,448) -- an email_type the
|
||||
handler enum does not model; do not fabricate a PO record deterministically.
|
||||
* revision -- 0 distinct emails in 3,448; no template to build.
|
||||
* any multi-line-item new_po (6 / 3,448) -- Lines-array structure unobserved.
|
||||
* any non-USD new_po (0 observed) -- non-USD path entirely unexercised.
|
||||
* non-Coupa senders (35 / 3,448) -- already rejected at the ses_auth layer.
|
||||
|
||||
Derived fields (site_code, trade, fiscal_year) are NOT computed here: the parser
|
||||
leaves them None and a shared post-stage (handler enrich_parsed(), the pad_zip
|
||||
precedent) fills them identically on both the template and the LLM path, so the
|
||||
gate judges extraction fidelity only. coupa_category is a verbatim label capture,
|
||||
not a classifier, and IS extracted here.
|
||||
|
||||
Entry point: try_deterministic_parse(email_data) -> (parsed|None, method,
|
||||
template_id, reason_code).
|
||||
"""
|
||||
|
||||
import re
|
||||
from decimal import Decimal, InvalidOperation
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# IMPLEMENTATION STATUS (scaffold)
|
||||
# [x] contract + recursive skeleton/normalize
|
||||
# [x] classify_template (both templates)
|
||||
# [x] coupa_cancellation extract + gate (simple, safe)
|
||||
# [ ] coupa_new_po extract -- labeled fields, duplicate-label anchoring,
|
||||
# U+2022 line split, Decimal amounts <-- TODO, needs fixtures
|
||||
# [ ] coupa_new_po value-level gate rules <-- TODO, currently FAILS CLOSED
|
||||
# Until the new_po TODOs land, a new_po email deterministically returns
|
||||
# reason 'new_po_not_implemented' and falls back to the LLM -- safe by design.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# The contract keys (23), EXACTLY -- mirrors the AI EXTRACTION_PROMPT fields.
|
||||
CONTRACT_KEYS = (
|
||||
"email_type",
|
||||
"po_number",
|
||||
"po_status",
|
||||
"source_system",
|
||||
"submitted_by",
|
||||
"on_behalf_of",
|
||||
"order_date",
|
||||
"revision_date",
|
||||
"last_opened",
|
||||
"acknowledged_at",
|
||||
"payment_terms",
|
||||
"requisition_number",
|
||||
"department",
|
||||
"view_order_url",
|
||||
"supplier",
|
||||
"site_code",
|
||||
"ship_to",
|
||||
"total_amount",
|
||||
"currency",
|
||||
"fiscal_year",
|
||||
"trade",
|
||||
"coupa_category",
|
||||
"line_items",
|
||||
)
|
||||
|
||||
SUPPLIER_KEYS = ("name",)
|
||||
|
||||
SHIP_TO_KEYS = (
|
||||
"name",
|
||||
"address",
|
||||
"street",
|
||||
"city",
|
||||
"state",
|
||||
"zip",
|
||||
"location_code",
|
||||
"attn",
|
||||
)
|
||||
|
||||
LINE_ITEM_KEYS = (
|
||||
"description",
|
||||
"amount",
|
||||
"currency",
|
||||
"need_by",
|
||||
"category",
|
||||
"account_code",
|
||||
"period",
|
||||
"quantity",
|
||||
"unit",
|
||||
"price",
|
||||
)
|
||||
|
||||
# Derived fields the parser must leave None (filled by the shared post-stage).
|
||||
DERIVED_KEYS = ("site_code", "trade", "fiscal_year")
|
||||
|
||||
SOURCE_SYSTEM = "coupa"
|
||||
|
||||
# email_type per template.
|
||||
_TEMPLATE_EMAIL_TYPE = {
|
||||
"coupa_new_po": "new_po",
|
||||
"coupa_cancellation": "cancellation",
|
||||
}
|
||||
VALID_EMAIL_TYPES = {"new_po", "revision", "cancellation"}
|
||||
|
||||
# Only these Status strings were observed on new_po (2,223 + 1,071 of 3,294).
|
||||
# Anything else fails closed to the LLM -- NEVER default-to-new_po.
|
||||
NEW_PO_SAFE_STATUSES = {"Issued - Created", "Issued - Scheduled for email"}
|
||||
|
||||
# Subject classifiers (Python unfolds header continuation lines before we see them).
|
||||
_NEW_PO_SUBJECT = re.compile(
|
||||
r"^\*\*\*Copy for Reference\*\*\* New Purchase Order\s+(?P<po>\S+)\s+has been issued$"
|
||||
)
|
||||
_CANCELLATION_SUBJECT = re.compile(
|
||||
r"Purchase Order\s+#(?P<po>[A-Z0-9-]+)\s+has been cancelled\s*$"
|
||||
)
|
||||
|
||||
# PO number shape, e.g. 2D-21456967, FK-21920384, B187-17955555.
|
||||
_PO_ID_RE = re.compile(r"^[A-Z0-9]{1,6}-\d+$")
|
||||
|
||||
# U+2022 bullet delimiting per-line metadata in the Lines section.
|
||||
_BULLET = "•"
|
||||
|
||||
# Sentinel: a label was present but its value did not parse. Must FAIL the gate
|
||||
# (present-but-unparseable), distinct from an absent value (None).
|
||||
_UNPARSEABLE = "__UNPARSEABLE__"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Skeleton helpers (recursive -- unlike WO's flat contract)
|
||||
# ---------------------------------------------------------------------------
|
||||
def _empty_line_item():
|
||||
return {k: None for k in LINE_ITEM_KEYS}
|
||||
|
||||
|
||||
def _empty_candidate():
|
||||
"""Full nested skeleton: every contract key present, None where absent."""
|
||||
cand = {k: None for k in CONTRACT_KEYS}
|
||||
cand["supplier"] = {k: None for k in SUPPLIER_KEYS}
|
||||
cand["ship_to"] = {k: None for k in SHIP_TO_KEYS}
|
||||
cand["line_items"] = [_empty_line_item()]
|
||||
cand["source_system"] = SOURCE_SYSTEM
|
||||
return cand
|
||||
|
||||
|
||||
def _normalize(candidate):
|
||||
"""Guarantee exact nested key presence before returning."""
|
||||
out = _empty_candidate()
|
||||
for k in CONTRACT_KEYS:
|
||||
if k in candidate and k not in ("supplier", "ship_to", "line_items"):
|
||||
out[k] = candidate[k]
|
||||
supplier = candidate.get("supplier") or {}
|
||||
out["supplier"] = {k: supplier.get(k) for k in SUPPLIER_KEYS}
|
||||
ship_to = candidate.get("ship_to") or {}
|
||||
out["ship_to"] = {k: ship_to.get(k) for k in SHIP_TO_KEYS}
|
||||
items = candidate.get("line_items") or [{}]
|
||||
out["line_items"] = [{k: (it or {}).get(k) for k in LINE_ITEM_KEYS} for it in items]
|
||||
return out
|
||||
|
||||
|
||||
def _clean(value):
|
||||
"""Strip trailing CR and U+00A0 nbsp that every captured Coupa value carries;
|
||||
collapse nothing else. Returns None for empty/placeholder 'None'."""
|
||||
if value is None:
|
||||
return None
|
||||
v = value.replace("\r", "").replace(" ", " ").strip()
|
||||
if v == "" or v == "None":
|
||||
return None
|
||||
return v
|
||||
|
||||
|
||||
def _plain_lines(body):
|
||||
return body.replace("\r\n", "\n").replace("\r", "\n").split("\n")
|
||||
|
||||
|
||||
def _to_decimal(raw):
|
||||
"""Parse a Coupa money token ('18,624.05') to Decimal, stripping thousands
|
||||
separators. Returns _UNPARSEABLE if it does not parse (gate must reject)."""
|
||||
if raw is None:
|
||||
return None
|
||||
token = raw.replace(",", "").strip()
|
||||
try:
|
||||
return Decimal(token)
|
||||
except (InvalidOperation, ValueError):
|
||||
return _UNPARSEABLE
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Subject helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
def classify_template(email_data):
|
||||
"""Return (template_id, reason). template_id in
|
||||
{coupa_new_po, coupa_cancellation, unknown}."""
|
||||
subject = _clean(email_data.get("subject")) or ""
|
||||
if _NEW_PO_SUBJECT.match(subject):
|
||||
return "coupa_new_po", "ok"
|
||||
if _CANCELLATION_SUBJECT.search(subject):
|
||||
return "coupa_cancellation", "ok"
|
||||
return "unknown", "subject_no_match"
|
||||
|
||||
|
||||
def _subject_po_id(email_data):
|
||||
"""PO number parsed from the subject, or None."""
|
||||
subject = _clean(email_data.get("subject")) or ""
|
||||
m = _NEW_PO_SUBJECT.match(subject)
|
||||
if m:
|
||||
return m.group("po")
|
||||
m = _CANCELLATION_SUBJECT.search(subject)
|
||||
if m:
|
||||
return m.group("po")
|
||||
return None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# T2: coupa_cancellation -> cancellation (simple + safe: po_number only)
|
||||
# ---------------------------------------------------------------------------
|
||||
def extract_cancellation(email_data):
|
||||
"""Cancellation carries no PO detail we trust beyond the id; the handler's
|
||||
save_cancellation() only needs po_number. Everything else stays None."""
|
||||
candidate = _empty_candidate()
|
||||
candidate["email_type"] = "cancellation"
|
||||
candidate["po_number"] = _subject_po_id(email_data)
|
||||
return _normalize(candidate)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# T1: coupa_new_po -> new_po
|
||||
# ---------------------------------------------------------------------------
|
||||
def extract_new_po(email_data):
|
||||
"""Extract the new_po contract from the text/plain body.
|
||||
|
||||
TODO(new_po): this is the core extraction work and needs fixtures before it
|
||||
can be trusted. Implement, in order, guarding each with a gate rule below:
|
||||
* labeled single-occurrence fields (PO ID, Status, Order Date, Revision
|
||||
Date, Payment Term, Req #, Submitted By, On Behalf Of, view_order_url,
|
||||
Department/Last Opened/Acknowledged At with literal 'None' -> null).
|
||||
* DUPLICATE labels: 'Supplier', 'Shipping', 'Total' each appear TWICE
|
||||
(summary placeholder + detail block; first 'Shipping' is literally
|
||||
'None'). Anchor supplier.name on the FIRST 'Supplier', the ship_to block
|
||||
on the SECOND 'Shipping', and pick a deterministic 'Total'.
|
||||
* ship_to by SENTINEL anchors ('United States' / 'Location Code:' /
|
||||
'Attn:'), never fixed line offsets (multi-line street unobserved).
|
||||
* line_items: split the metadata line on the bare U+2022 (_BULLET) and
|
||||
assign each segment BY LEADING LABEL word ('Need By '/'Category '/
|
||||
'Account '/'Period '/'Part Number '), never by ordinal position.
|
||||
* amounts via _to_decimal (thousands-separator safe); currency = the exact
|
||||
3-letter token adjacent to the amount.
|
||||
* coupa_category = verbatim 'Category' bullet value.
|
||||
Leave DERIVED_KEYS (site_code, trade, fiscal_year) as None -- filled later by
|
||||
the shared post-stage.
|
||||
"""
|
||||
candidate = _empty_candidate()
|
||||
candidate["email_type"] = "new_po"
|
||||
candidate["po_number"] = _subject_po_id(email_data)
|
||||
# DERIVED_KEYS intentionally left None (shared post-stage fills them).
|
||||
# TODO(new_po): populate the remaining fields per the docstring, then remove
|
||||
# the fail-closed guard in validate().
|
||||
return _normalize(candidate)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Validation gate -- FAIL CLOSED
|
||||
# ---------------------------------------------------------------------------
|
||||
def _structural_keys_ok(candidate):
|
||||
if set(candidate.keys()) != set(CONTRACT_KEYS):
|
||||
return False
|
||||
if set((candidate.get("supplier") or {}).keys()) != set(SUPPLIER_KEYS):
|
||||
return False
|
||||
if set((candidate.get("ship_to") or {}).keys()) != set(SHIP_TO_KEYS):
|
||||
return False
|
||||
items = candidate.get("line_items")
|
||||
if not isinstance(items, list) or not items:
|
||||
return False
|
||||
return all(set((it or {}).keys()) == set(LINE_ITEM_KEYS) for it in items)
|
||||
|
||||
|
||||
def validate(candidate, template_id, email_data):
|
||||
"""Return (True, 'ok') only if provably conformant; else (False, reason).
|
||||
Every rule must hold. See failClosedGateRules in the investigation report."""
|
||||
# (1) known template
|
||||
if template_id not in _TEMPLATE_EMAIL_TYPE:
|
||||
return False, "subject_no_match"
|
||||
|
||||
# (2) exact nested key-set
|
||||
if not _structural_keys_ok(candidate):
|
||||
return False, "key_set_mismatch"
|
||||
|
||||
# (3) derived fields must NOT be populated by the parser
|
||||
for k in DERIVED_KEYS:
|
||||
if candidate.get(k) is not None:
|
||||
return False, "derived_field_set"
|
||||
|
||||
# (4) email_type matches the template's expected type
|
||||
expected_type = _TEMPLATE_EMAIL_TYPE[template_id]
|
||||
et = candidate.get("email_type")
|
||||
if et not in VALID_EMAIL_TYPES:
|
||||
return False, "missing_required_field"
|
||||
if et != expected_type:
|
||||
return False, "email_type_mismatch"
|
||||
|
||||
# (5) po_number: valid shape AND byte-equals the subject id
|
||||
subject_po = _subject_po_id(email_data)
|
||||
po = candidate.get("po_number")
|
||||
if not po or not _PO_ID_RE.match(str(po)):
|
||||
return False, "missing_required_field"
|
||||
if po != subject_po:
|
||||
return False, "po_id_mismatch"
|
||||
|
||||
if template_id == "coupa_cancellation":
|
||||
# Cancellation is complete once po_number is proven; save_cancellation
|
||||
# sets status=Cancelled by po_number and needs nothing else.
|
||||
return True, "ok"
|
||||
|
||||
# ---- coupa_new_po ----
|
||||
# SCAFFOLD GUARD: extract_new_po is not implemented yet, so a new_po parse is
|
||||
# never provable -> fail closed to the LLM with an honest reason. Remove this
|
||||
# line once extract_new_po + the value-level rules below are implemented.
|
||||
return False, "new_po_not_implemented"
|
||||
|
||||
# (6) status must be one of the confirmed-safe strings.
|
||||
status = candidate.get("po_status")
|
||||
if status not in NEW_PO_SAFE_STATUSES:
|
||||
return False, "unrecognized_status"
|
||||
|
||||
# (7) single-line only -- multi-line Lines structure is unobserved.
|
||||
if len(candidate.get("line_items") or []) != 1:
|
||||
return False, "multiline_unsupported"
|
||||
|
||||
# (8) currency must be exactly USD (non-USD path entirely unexercised).
|
||||
if candidate.get("currency") != "USD":
|
||||
return False, "non_usd"
|
||||
|
||||
# TODO(new_po) value-level gate rules -- implement alongside extract_new_po:
|
||||
# * amount fidelity: re-serialize each Decimal to the source grouped format
|
||||
# and require byte-equality with the matched substring; reject if the
|
||||
# token is bordered by a digit/comma (guards '18,624.05' -> '624.05').
|
||||
# Then require sum(line_items[].amount) == total_amount.
|
||||
# * duplicate-label anchoring: reject Shipping=='None'; require ship_to has
|
||||
# a 'Location Code:' line; supplier.name contains 'SEA HAVEN';
|
||||
# ship_to.name != supplier.name.
|
||||
# * bullet metadata: every U+2022 segment carries a recognized leading
|
||||
# label; fail on any unlabeled segment.
|
||||
# * address shape: ship_to.state null-or-valid-USPS; ship_to.zip null-or
|
||||
# ^\d{5}(-\d{4})?$ after pad_zip; tokenize city/state/zip by pattern.
|
||||
# * _UNPARSEABLE discipline: reject any field still == _UNPARSEABLE.
|
||||
# * whitespace hygiene: reject any value still containing \r or \xa0.
|
||||
# Until the above land, new_po is not provable -> fail closed to the LLM.
|
||||
return False, "new_po_not_implemented"
|
||||
|
||||
|
||||
def try_deterministic_parse(email_data):
|
||||
"""Entry point. Returns (parsed|None, parse_method, template_id, reason).
|
||||
|
||||
On a proven-conformant parse returns (dict, 'template', template_id, 'ok').
|
||||
On any miss/invalid/exception returns (None, 'ai_fallback', template_id,
|
||||
reason) -- a failure is NEVER a parsed result."""
|
||||
template_id = "unknown"
|
||||
try:
|
||||
template_id, reason = classify_template(email_data)
|
||||
if template_id == "unknown":
|
||||
return None, "ai_fallback", template_id, reason
|
||||
|
||||
if template_id == "coupa_new_po":
|
||||
candidate = extract_new_po(email_data)
|
||||
else:
|
||||
candidate = extract_cancellation(email_data)
|
||||
|
||||
ok, reason = validate(candidate, template_id, email_data)
|
||||
if not ok:
|
||||
return None, "ai_fallback", template_id, reason
|
||||
return candidate, "template", template_id, "ok"
|
||||
except Exception: # noqa: BLE001 -- fail closed on ANY extractor error
|
||||
return None, "ai_fallback", template_id, "extractor_raised"
|
||||
Loading…
Add table
Reference in a new issue