procurement-ingest/docs/po-template-parser.md

# PO Template-First Parser — Design, Investigation & Progress

> **Status:** in progress · **Branch:** `feat/po-template-parser` (stacked on PR #99) · **Last updated:** 2026-07-16
>
> Living document for making the Coupa **purchase-order** email parser *template-first with LLM fallback*, mirroring the work-order (WO) processor's PR #99. Captures the investigation, the data we gathered, the decisions made, the current scaffold, and everything still to do.

---

## 1. Context & Goal

The PO email processor (`lambdas/po/email_processor/handler.py`) sends **every** inbound Coupa/Amazon purchase-order email to an LLM (Claude Haiku 4.5 on Bedrock) for structured extraction. WO PR #99 established the pattern we want to replicate here:

> Run a **deterministic template parser first**. It returns a result only when the email provably conforms to a known template and passes a **fail-closed validation gate**; otherwise the handler falls back to the Bedrock LLM. This removes the LLM from the hot path for routine traffic while keeping full AI coverage for anything unexpected.

**Goal:** apply the same template-first + fail-closed-fallback approach to the PO processor, without regressing data quality or the downstream `po-ingest-site-extractor` pipeline (which reads `site_code`/`ship_to` off the `purchase-orders` DynamoDB stream).

**Reference implementation:** `lambdas/wo/email_processor/template_parser.py` (PR #99).

### Why this is harder than WO
- The PO contract is ~40 fields with **nested** objects (`supplier{}`, `ship_to{}`, `line_items[]`) vs WO's flat 16.
- Much of the PO extraction prompt is **derived classification** (`trade`, `site_code`, `fiscal_year`) already expressed as deterministic English rules.
- Money fields require `Decimal` (DynamoDB rejects floats).
- `email_type` is **not** in the subject (unlike WO's two subject templates).

---

## 2. Investigation

Two phases: a read-only multi-agent (ultracode) feasibility study over a 120-email sample, then a full-bucket triage over **all 3,448** inbound emails to get real distribution numbers.

### 2.1 Data access
- PO email bucket: `s3://po-ingest-emails-328440206208/inbound/` (AWS account **328440206208**, us-east-1).
- Reached via AWS profile **`amoussa-mgmt`** (SSO). The default CLI creds are the personal account `681986854588`, which is **denied** on the ingest buckets — a recurring gotcha.

### 2.2 Full-bucket triage results (all 3,448 emails, 2026-07-16)

Method: parallel ranged-GET (`bytes=0-12000`) of every object, then local classification (quoted-printable bodies, decodable offline).

| Email kind | Count | % | Notes |
|---|---:|---:|---|
| **new_po** — `***Copy for Reference*** New Purchase Order <PO#> has been issued` | 3,294 | **95.5%** | Single uniform Coupa layout |
| **cancellation** — `<Site> Purchase Order #<PO#> has been cancelled` | 100 | **2.9%** | Distinct subject + minimal body |
| **comment** — `New Comment on Purchase Order for Amazon` | 19 | 0.55% | Type NOT in handler enum (see §7) |
| **non-Coupa senders** (human replies, WO mail, an AWS SES setup notice) | 35 | 1.0% | Rejected at ses_auth before parse |
| **revision** | **0** | 0% | No distinct revision emails exist |

Within **new_po**:
- **Status** values: `Issued - Created` (2,223) and `Issued - Scheduled for email` (1,071) — only these two.
- **Multi-line-item:** 6 / 3,294 (**0.18%**), all ≤3 items.
- **Non-USD:** 0 (after excluding unit-token false positives like `EACH`/`HOUR`).
- So the single-line / USD scope covers **~99.8%** of new_po traffic.

### 2.3 new_po layout (the one template that matters)

`multipart/alternative`; the **text/plain** part is a stable, label-delimited flattening:
- Top: `Amazon Purchase Order #<PO#>`, then `Submitted By`, `On Behalf Of`, `Supplier`, `Total`, `Items`.
- `More Detail` block: `PO ID`, `Department`, `Status`, `Last Opened`, `Order Date`, `Acknowledged At`, `Revision Date`, `Payment Term`, `Req #`, `Shipping`.
- `Supplier` detail block (address).
- `Shipping` detail block (ship-to address, `Location Code:`, `Attn:`).
- `Lines` section: one block per line item; per-line metadata is **U+2022 (`•`) delimited**: `Need By` · `Category` · `Account` · `Period` [· optional `Part Number`].

**Structural hazards found in real data:**
- `Supplier`, `Shipping`, `Total` labels **each appear twice** (summary placeholder + detail block); the *first* `Shipping` value is literally `None`.
- Optional `Part Number` bullet segment appears in a minority of emails, inserted between `Category` and `Account` → shifts any positional splitter.
- Every captured value carries a trailing `\r`; a lone `\xa0` (nbsp) line sits before `Total`.
- Real amounts include thousands separators (`18,624.05`, `32,405.00`).

### 2.4 cancellation layout

Subject: `<SiteName> Purchase Order #<PO#> has been cancelled`. Body is a short dashes-delimited notice — **no `Status` label**, no line items. The only field the handler's `save_cancellation()` needs is `po_number`.

---

## 3. Key Decisions

1. **Build two templates**, not one and not three:
   - `coupa_new_po` (95.5%) — the main workload.
   - `coupa_cancellation` (2.9%) — trivial and low-risk (just `po_number`); we have 100 real examples.
   - Together ≈ **98.4%** of all mail deterministically handled/routed.
2. **No revision template** — 0 distinct revision emails in 3,448. The handler's `revision` type essentially never arrives as its own email.
3. **`email_type` never defaults to new_po.** It is emitted only when the exact new_po subject regex matches **and** `po_status` ∈ the two confirmed-safe strings. Anything else → LLM. (Misrouting a cancellation to new_po would silently defeat the sticky-`Cancelled` guard.)
4. **Fail closed, always.** Any miss / invalid / exception returns `None` and falls back to the LLM. A failure is never a parsed result. (Copies WO's `try/except` posture.)
5. **`Decimal` for all money**, with thousands-separator stripping, to match the LLM path's `parse_float=Decimal`.
6. **Derived fields (`site_code`, `trade`, `fiscal_year`) are NOT computed in the parser.** They are left `None`; a shared post-stage (`enrich_parsed()`, the `pad_zip` precedent) fills them **identically** on both the template and LLM paths, so the gate judges *extraction fidelity only* and both paths write byte-identical shapes downstream. `coupa_category` is a **verbatim** label capture, not a classifier, and IS extracted.
7. **Structural nested contract** with recursive key-set validation (unlike WO's flat tuple).
8. **Two-PR split:** (1) extraction template + gate + handler wiring; (2) derived-classifier factoring, shadow-logged before it becomes authoritative.

---

## 4. Contract

Top-level `CONTRACT_KEYS` (23) plus three nested sub-tuples. Taken from the `EXTRACTION_PROMPT` in `handler.py`.

```
CONTRACT_KEYS = (
  email_type, po_number, po_status, source_system, submitted_by, on_behalf_of,
  order_date, revision_date, last_opened, acknowledged_at, payment_terms,
  requisition_number, department, view_order_url, supplier, site_code, ship_to,
  total_amount, currency, fiscal_year, trade, coupa_category, line_items,
)
SUPPLIER_KEYS  = (name,)
SHIP_TO_KEYS   = (name, address, street, city, state, zip, location_code, attn)
LINE_ITEM_KEYS = (description, amount, currency, need_by, category,
                  account_code, period, quantity, unit, price)
```

| Class | Keys |
|---|---|
| **Verbatim / labeled extract** | po_number, po_status, submitted_by, on_behalf_of, order_date, revision_date, last_opened, acknowledged_at, payment_terms, requisition_number, department, view_order_url, supplier.name, ship_to.location_code, ship_to.attn, coupa_category, all `line_items[*]` |
| **Extract with care** | total_amount, currency, ship_to.name/street/city/state/zip (`Decimal`; sentinel-anchored address) |
| **Constant** | source_system = `"coupa"` |
| **Derived — post-stage, NOT parser, NOT gate-validated** | site_code, trade, fiscal_year |

Handler enrichment metadata (`raw_s3_key`, `processed_at`, `data_source`, `email_subject`, `ship_to_raw`, top-level `state`) is added by `enrich_parsed()` and is **not** part of the parser contract.

A parity test (mirroring WO's `test_contract_keys_match_extraction_prompt`) should assert the flattened contract key set is a subset of the keys quoted in `EXTRACTION_PROMPT`.

---

## 5. Parser Design

File: `lambdas/po/email_processor/template_parser.py` — pure module (no boto3/network), mirroring WO idioms.

- `classify_template(email_data) -> (template_id, reason)` — exact subject regex → `coupa_new_po` | `coupa_cancellation` | `unknown`.
- `extract_new_po` / `extract_cancellation` — build the nested candidate; leave derived fields `None`.
- `_empty_candidate()` / `_normalize()` — build & normalize the **nested** skeleton (`supplier={}`, `ship_to={}`, `line_items=[one item]`).
- `_clean()` — strip trailing `\r` + `\xa0`, map `"None"`/empty → `None`.
- `_to_decimal()` — thousands-separator-safe `Decimal`, `_UNPARSEABLE` sentinel on failure.
- `validate(candidate, template_id, email_data) -> (bool, reason)` — fail closed.
- `try_deterministic_parse(email_data) -> (parsed|None, method, template_id, reason)` — entry point; `try/except` fails closed on any error.

**Handler integration (not yet wired):** after `authenticate_inbound_email()` + `parse_raw_email()`, call `try_deterministic_parse`; if `None`, `extract_with_claude`; then run the shared `enrich_parsed()`+classify stage on `parsed` regardless of path, then the existing `save_cancellation` / `save_revision` / `save_new_po` routing. Emit a `ParseMethod` EMF metric like WO.

---

## 6. Fail-Closed Gate Rules

Reason codes currently emitted by `validate()` / `try_deterministic_parse()`:

`subject_no_match` · `key_set_mismatch` · `derived_field_set` · `email_type_mismatch` · `missing_required_field` · `po_id_mismatch` · `unrecognized_status` · `multiline_unsupported` · `non_usd` · `new_po_not_implemented` · `extractor_raised` · (`ok` / `template` on success).

**Implemented (both templates):**
- Known template only (`subject_no_match` otherwise).
- Exact nested key-set at every level (`key_set_mismatch`).
- Derived fields must be unset by the parser (`derived_field_set`).
- `email_type` in enum and == template's expected type (`email_type_mismatch`).
- `po_number` valid shape `^[A-Z0-9]{1,6}-\d+$` **and** byte-equals the subject id (`po_id_mismatch` / `missing_required_field`).

**Implemented (new_po):** safe-status enum (`unrecognized_status`), single-line only (`multiline_unsupported`), USD only (`non_usd`).

**TODO (new_po value-level — see §8):** amount byte-equality + `sum(lines)==total`; duplicate-label anchoring (reject `Shipping=='None'`, require `Location Code:`, `supplier.name` contains `SEA HAVEN`, `ship_to.name != supplier.name`); bullet-metadata split by leading label; address shape (`state` null-or-USPS, `zip` null-or `^\d{5}(-\d{4})?$`); `_UNPARSEABLE` rejection; residual `\r`/`\xa0` rejection.

**Precondition (handled a layer earlier):** `authenticate_inbound_email()` (SES `dkim=pass` for `amazon.coupahost.com`) must pass before the parser runs — the 35 non-Coupa emails are rejected there.

---

## 7. Scope — In / Out

**IN (template-parsed):**
- Single-line-item, USD, `new_po` with the exact subject and a safe `Status`.
- `cancellation` (subject match → `po_number`).

**OUT (always LLM fallback / rejected):**
- **Comments** (19) — `New Comment on Purchase Order` is an `email_type` the handler enum (`new_po`/`revision`/`cancellation`) does not model. The LLM currently shoehorns these. ⚠️ **Pre-existing data-quality gap, independent of this work** — decide separately whether comments should create/update PO records at all.
- **Revisions** — no template (0 emails).
- **Multi-line-item new_po** (0.18%) — `Lines`-array structure unobserved; `multiline_unsupported`.
- **Non-USD new_po** (0 observed) — path unexercised; `non_usd`.
- **Non-Coupa senders** (1.0%) — rejected at ses_auth.

---

## 8. Progress

- [x] Read-only ultracode feasibility investigation (8 agents, GO-with-conditions).
- [x] Full-bucket triage over all 3,448 emails → distribution + scope numbers (§2.2).
- [x] Confirmed cancellation & comment body layouts against real emails.
- [x] **Scaffold** `template_parser.py` on `feat/po-template-parser`:
  - [x] Nested contract + recursive `_empty_candidate`/`_normalize`.
  - [x] `classify_template` for both templates (real subject regexes).
  - [x] `coupa_cancellation` extract + gate — **implemented & smoke-tested** (parses a real cancellation → `po_number`, `email_type=cancellation`).
  - [x] `coupa_new_po` classify + structural/status/single-line/USD gate.
  - [x] Scaffold guard: `extract_new_po` is a documented TODO → new_po returns `new_po_not_implemented` and falls back to the LLM (safe).
  - [x] `ruff check` + `ruff format --check` pass.
- Smoke test (real corpus): cancellation → `template`; new_po → `ai_fallback/new_po_not_implemented`; comment & non-Coupa → `ai_fallback/subject_no_match`.

> Not yet committed; working-tree only.

---

## 9. TODO / Path Forward

**PR #1 — extraction template + wiring**
- [ ] Implement `extract_new_po`: labeled single-occurrence fields; duplicate-label anchoring (`Supplier`/`Shipping`/`Total`); `ship_to` by sentinel anchors; `line_items` split on `•` by leading label; `Decimal` amounts; `coupa_category` verbatim.
- [ ] Implement the new_po value-level gate rules (§6 TODO) and remove the scaffold guard.
- [ ] Handler integration: `try_deterministic_parse` first, Bedrock fallback on `None`; route through existing `save_*` branches.
- [ ] Emit `ParseMethod`-only EMF set (namespace `Seahaven/PoIngest`, dims `[["ParseMethod"], ["ParseMethod","TemplateId"]]`).
- [ ] Fallback-rate CloudWatch alarm in `cdk/po_stack.py` — **retune thresholds for ~57/day volume** (PO is far lower than WO's ~760/day; the WO 15-min/sample-floor settings won't fit).
- [ ] Offline tests + fixtures: good cancellation + good new_po; one adversarial per gate rule (thousands-sep, dup-label swap, Part-Number bullet shift, multiline, non-USD, bad status); comment + non-Coupa assertions; contract⊆prompt parity test.
- [ ] README + Confluence "AWS Architecture Map" updates.
- [ ] `cross_reviewer` (GPT-4.1) pass — handler contract change (Sea Haven policy).
- [ ] `ruff check` + `ruff format` + `pytest` green in CI.

**PR #2 — derived-classifier factoring (follow-up)**
- [ ] Move `trade`/`site_code`/`fiscal_year` into shared `enrich_parsed()`.
- [ ] Run in **shadow mode**: compute Python values, log Python-vs-LLM disagreement via EMF, keep LLM authoritative during a bake period.
- [ ] Harden `site_code` (7-shape + skip-list, incl. multi-hop ATTN like `CBRE - RME - DLI6`) and `trade` against the full sample with per-shape fixtures.
- [ ] Only after acceptable agreement: make Python authoritative and drop the fields from `EXTRACTION_PROMPT`.

---

## 10. Open Questions

1. **Goal — cost, latency, or determinism?** Determinism (same input → same output) is the strongest justification for the derived-classifier port regardless of per-email cost.
2. **Classifier factoring now or deferred?** Recommendation: extraction template first (PR #1), factoring as shadow-logged PR #2.
3. **Strict single-line/USD-only v1 acceptable?** Triage says it covers ~99.8% of new_po, so yes — but confirm.
4. **Confirm the real cancellation `Status` spelling** — `handler.py` hardcodes `CANCELLED_STATUS='Cancelled'`; the cancellation body doesn't restate it. Verify against a known-cancelled PO in the table.
5. **Comments** — should `New Comment on Purchase Order` emails create/update PO records at all, or be dropped? (Out of scope for the parser, but a real data-quality decision.)
6. **Repo fixtures** — commit a sanitized PO corpus (like WO's checked-in `.eml` fixtures), or keep it out of the repo?

---

## 11. Risk Register (high-severity)

| Risk | Mitigation |
|---|---|
| **Cancellation misrouted to new_po** → sticky-`Cancelled` guard defeated, PO stays active | `email_type` closed-set gate; never default-to-new_po; only emit on exact subject + safe status |
| **Thousands-separator truncation** (`18,624.05`→`624.05`, ~30× too small, passes naive checks) | Amount byte-equality: re-serialize `Decimal` to source grouped format; reject if token bordered by digit/comma; `sum(lines)==total` |
| **Multi-line structure unobserved** | Hard-fail `len(line_items)!=1` → LLM; profile real multi-line before supporting |
| **Derived-field silent divergence** (wrong `trade`/`site_code` still well-formed; `site_code` rules already incomplete) | Factor into shared post-stage; keep OFF the gate; shadow-log before authoritative; per-shape fixtures |
| **Duplicate-label first-match** (`Supplier`/`Shipping`/`Total` twice; first `Shipping`=`None`) | Occurrence/sentinel anchoring; gate rejects `Shipping=='None'`, requires `Location Code:`, `supplier.name` has `SEA HAVEN`, `ship_to.name != supplier.name` |

---

## Appendix — References

- WO reference parser: `lambdas/wo/email_processor/template_parser.py` (PR #99).
- PO handler + `EXTRACTION_PROMPT`: `lambdas/po/email_processor/handler.py`.
- PO stack (alarms, IAM, Bedrock): `cdk/po_stack.py`.
- Bedrock model: inference profile `us.anthropic.claude-haiku-4-5-20251001-v1:0`.
- PO email bucket: `s3://po-ingest-emails-328440206208/inbound/` (account 328440206208, profile `amoussa-mgmt`).