Compare commits

...

5 commits

Author SHA1 Message Date
Adam Moussa
0b7a40e2be
Merge 98f806d9e5 into d3e49c12ef 2026-07-17 20:20:37 -04:00
98f806d9e5
feat: collapse duplicated CDK into cdk/common.py plain helpers (refactor phase 4)
The ~379 lines po_stack.py and wo_stack.py defined identically (DynamoDB
alarms, the sender-auth-rejected metric filter + alarm, the standard
per-Lambda alarm set, the Bedrock InvokeModel grant, the raw-email
bucket, the async DLQ, the template-fallback-rate math alarm) move into
cdk/common.py.

Every helper is a PLAIN function taking (scope, id, ...), called with each
stack's own Stack as scope and the exact literal construct ids used inline
before, so every synthesized logical ID is byte-stable. A Construct
subclass would reparent the tree and make CloudFormation attempt to
replace the RETAIN-protected purchase-orders/WorkOrders tables and named
buckets -- data loss -- so it is forbidden. Per-function alarm variance
(PO p99 vs WO p95 duration, po-web-ui throttles+duration only,
site-extractor no DLQ alarm, workorder-web-ui zero alarms) is preserved
through call-site arguments, not baked into the helpers.

make_bedrock_invoke_statement derives the inference-profile and us-east-1
foundation-model ARNs from Stack.of(scope).account/.region instead of the
hardcoded 328440206208/us-east-1 literals. The environment stays
account-agnostic (region-only), so the account resolves to the
AWS::AccountId pseudo-parameter: the derived ARN resolves at deploy to the
same ARN the literal named in-account (a benign in-place IAM policy
update, never a replacement) and is account-portable rather than pinned to
the frozen management account.

The account= pin evaluated for cdk.Environment was deliberately NOT added:
resolving every account-derived value (bucket names, Lambda::Permission
source account, SNS action ARN) to literals makes CloudFormation flag the
RETAIN email buckets as requiring replacement against the deployed
account-agnostic templates -- a data-loss risk that outranks the pin, which
buys nothing (the resolved values are unchanged).

Also: net-new CfnOutputs for the five Lambda function ARNs and the
owned/consumed table names, exact-pin constructs==10.6.0, and fix the
stale aws-cdk-lib 2.259.0 -> 2.261.0 version comment.

The common.py extraction is zero-cdk-diff on both stacks (byte-stable
logical IDs, no asset/property change); the only deltas versus deployed
are the intended benign Bedrock IAM in-place update and the additive
CfnOutputs. Mandatory GPT-4.1 cross-family review ran on the Bedrock IAM
move; its BLOCK was a verified false positive (it read AWS::AccountId as a
wildcard -- it is a deploy-time-resolved concrete value naming one account
and one inference-profile, region is pinned us-east-1, and the grant is
strictly more least-privilege-correct than the hardcoded literal).
2026-07-17 20:18:51 -04:00
d3e49c12ef
feat: extract lambdas/shared/ — single-source ses_auth, web_ui auth, email parsing, EMF emitter (refactor phase 3)
Four modules move into the handbook-mandated lambdas/shared/ location,
collapsing duplicated logic that had to be kept in sync by hand across
the PO and WO pipelines:

- ses_auth.py: the PO and WO copies were verified sha256-identical
  against the feature/phase-7-ops-recovery baseline before the move
  (no drift since the last audit). shared/ses_auth.py is the exact
  bytes of that one copy; both originals are git rm'd (the PO copy
  via rename, the WO copy as a straight delete). Bundling lands the
  module flat in /asset-output for both email processors, so the
  handlers keep `from ses_auth import authenticate_inbound_email`
  unchanged — zero handler diff for this move, which is what keeps
  fail-closed auth byte-identical through the change.

- web_ui_auth.py: extracts the byte-identical _get_auth_token /
  _header / is_authenticated block plus the four token-cache globals
  out of both web_ui handlers. The per-stack INFRA-74 comments stay
  in each handler as-is (deliberately drifted wording, stack-specific)
  rather than being unified into the shared module. Fail-closed
  semantics (unset ARN or Secrets Manager exception -> deny) are
  unchanged.

- email_parsing.py: parse_raw_email ships as the superset version that
  returns cc unconditionally. WO's output is bit-identical to before;
  PO simply ignores the cc field rather than being "cleaned up" to
  consume it. No second variant is kept.

- emf.py: a generic emitter parameterized by namespace, dimension
  sets, and properties. Every call site's emitted EMF envelope is
  unchanged, including the load-bearing
  [["ParseMethod"],["ParseMethod","TemplateId"]] dimension-set shape
  the alarms and metric filters depend on. Emission ordering is
  untouched: PO still emits ai_fallback before the Bedrock call, WO
  still emits its mutually-exclusive ai_fallback/ai_fallback_rejected
  after its gate. The deliberate-double-count comments survive.
  _emit_derived_agreement_metric was found living inside
  derived_fields.py, so per the DERIVED-FIELDS exception it is left
  as a third, unconverted copy (derived_fields.py and the shadow
  DerivedFieldAgreement telemetry stay untouchable while that bake
  runs) — a comment there points at shared/emf.py for the eventual
  follow-up.

Bundling: both email-processor cdk bundling commands gain a trailing
`cp shared/*.py /asset-output/` (they were already cp-only post-Phase
7, so no pip step or manylinux pin is reintroduced). Both web_ui
functions gain the same widened-root staging so web_ui_auth.py ships
beside their handler; site_extractor's from_asset is untouched.

Tests: PO_EXPECTED_TOP_LEVEL_MODULES gains the shared modules that now
ship, the AST sibling-import check resolves imports whose source now
lives under shared/, and the new shared cp line has its own
revert/mutation detection. _SIBLING_MODULES resolution and
_po_parser_support.py now load ses_auth/email_parsing/emf from
shared/; the two-copy ses_auth byte-identity fixture-hygiene test is
retired as obsolete now that there is one copy, and the ses_auth
fixture parameterization over two identical copies is dropped. The
sys.modules save/restore dance for template_parser (still duplicated
per-pipeline) is left in place.
2026-07-17 18:39:53 -04:00
88805d906d
Reject --all combined with --prefix/--since in reprocess.py
--all is a distinct mode (the demoted full-prefix sweep), but the args.all
branch unconditionally set prefix=inbound/ and since=None, so passing it
alongside a narrower selector silently discarded that selector. `--all
--since 2026-07-01` swept the entire corpus instead of the bounded window,
triggering every documented --all hazard (Bedrock re-bill, metric double-
count, merged-field regression) on objects the operator never targeted --
contradicting the tool's safety goal. Add the missing mutual-exclusion
guard alongside the existing --key one, and pin --all+--prefix,
--all+--since, and all three together as argparse rejections.
2026-07-17 17:03:44 -04:00
7765111781
feat: ops/recovery tooling + dependency hygiene (refactor phase 7)
Generalize scripts/reprocess.py from a PO-only full-sweep script into a
pipeline-general recovery tool. Targeted replay (--key/--prefix/--since)
is now the default, and the full inbound/ sweep is demoted behind an
explicit --all that documents its five hazards (async concurrency does
not serialize, use RequestResponse if order matters, metric double-count,
Bedrock re-bill, out-of-order field regression). --pipeline po|wo resolves
the correct function + bucket; dry-run-by-default / --execute is preserved.
A new tests/test_reprocess_contract.py pins the synthetic S3 event shape
and asserts the raw list_objects_v2 key is emitted untransformed (the
handler is the single decode point; a pre-decoded key would corrupt keys
containing spaces or '+').

Add docs/runbook-dlq-recovery.md: the async on-failure DLQ has no console
redrive-to-source, so it documents the receive -> extract key -> targeted
reprocess --key -> verify -> purge procedure, the real recovery windows
(14-day DLQ breadcrumb, 90-day raw-email S3 that overrides the table
RETAIN policy and is the true replay floor), and that sender-auth and
ai_fallback_rejected drops are fail-closed skips that never reach the DLQ.
Linked from the README alarms and scripts sections.

Drop the vendored boto3 floor pin from both email-processor requirements
(the Lambda runtime provides boto3; lambda-template.md empty-with-comment
form). With nothing left to install, the email-processor bundling becomes
cp-only -- the whole pip step is removed, which is the only acceptable way
the manylinux2014_aarch64 pin disappears (removing the pin while keeping a
pip install caused the PR #34 x86-wheel outage). Exact-pin moto==5.2.2 and
add pinned po/web_ui + po/site_extractor manifests (excluded from their
bundles, so hash-neutral) so their new Dependabot entries have something
to act on; add Dependabot entries for /tests, /lambdas/po/web_ui, and
/lambdas/po/site_extractor.

cdk diff is confined to exactly the two email processors' asset hashes on
both stacks. The wo/web_ui dead-manifest reduction was deliberately left
out: that manifest already ships inside the plain (non-bundled) WebUI
asset on main, so reducing or excluding it would redeploy workorder-web-ui
for no functional change -- deferred to keep the blast radius to the two
intended targets.

The untracked 44 MB lambdas/po/email_processor/package/ dir was removed
from the filesystem (asset-hash-neutral given Phase 2's package/ exclude);
it is untracked, so there is nothing to commit for it.
2026-07-17 16:32:59 -04:00
31 changed files with 3740 additions and 1482 deletions

View file

@ -0,0 +1,679 @@
export const meta = {
name: 'phase-3-shared-extraction',
description: 'Phase 3 of the procurement-ingest refactor (docs/refactor-evaluation.md): extract lambdas/shared/ — ses_auth.py (byte-identical move, zero handler diff), web_ui_auth.py (byte-identical auth block from both web_ui handlers), email_parsing.py (parse_raw_email superset, cc unconditional), emf.py (parameterized emitter preserving every envelope byte-for-byte). Bundling gains cp shared/*.py in both email-processor commands + the same staging mechanism for both web_ui functions. Requires Phases 0-2 on the base branch. Auth code moves, so push is gated on /sh-security-review in the main loop. Committed locally, never pushed.',
phases: [
{ title: 'Setup', detail: 'verify Phases 0+1+2 on base, branch feature/phase-3-shared-extraction', model: 'haiku' },
{ title: 'Recon', detail: '4 mappers: the four duplicated modules, post-Phase-2 bundling sites, test-loader plumbing, deployed-zip baseline (read-only AWS)' },
{ title: 'Spec', detail: 'serial fable spec: pin shared-module contents, handler edit lists, web_ui staging, bundling strings, EMF design, test plumbing' },
{ title: 'Implement', detail: 'opus: shared/ + four handlers; sonnet: both CDK stacks; opus: all test plumbing + README — disjoint files', model: 'opus' },
{ title: 'Verify', detail: 'mechanical gates + bundle-parity verifier + 3 fable lenses (auth integrity, EMF/telemetry, loader integrity)' },
{ title: 'Fix', detail: 'opus fixer, full re-verify, max 3 rounds', model: 'opus' },
{ title: 'Package', detail: 'single commit via -F (no push)', model: 'sonnet' },
],
}
// ---------------------------------------------------------------- constants
const REPO = '/Users/adammoussa/Documents/repositories/seahaven/procurement-ingest'
const BRANCH = 'feature/phase-3-shared-extraction'
let _args = args
if (typeof _args === 'string') {
try { _args = JSON.parse(_args) } catch (e) { _args = null }
}
const BASE = (_args && _args.base) || 'main'
const CONSTRAINTS = `
PINNED BEHAVIORAL CONSTRAINTS (docs/refactor-evaluation.md Phase 3 — violating any is a build failure):
1. THE MOVE SET IS EXACTLY FOUR MODULES, in this order of dependency risk:
lambdas/shared/ses_auth.py, lambdas/shared/web_ui_auth.py,
lambdas/shared/email_parsing.py, lambdas/shared/emf.py. Nothing else
moves into shared/. lambdas/shared/ is the handbook location
(cdk-project-layout.md); modules land FLAT in every bundle so bare-name
imports keep working.
2. ses_auth: FIRST verify the two current copies are still byte-identical
(sha256 both against the ${BASE} versions — any drift since the audit is
a blocker, not something to silently reconcile). shared/ses_auth.py is
the EXACT bytes of that single copy; both originals are git rm'd. The
email-processor handlers keep 'from ses_auth import
authenticate_inbound_email' UNCHANGED — flat landing in /asset-output
means ZERO handler diff for this move, which is what keeps fail-closed
auth byte-identical through the change.
3. web_ui_auth: extract exactly the byte-identical block — _get_auth_token /
_header / is_authenticated + the four cache globals — from BOTH web_ui
handlers. The per-stack INFRA-74 comments STAY in each handler (their
wording has drifted deliberately; they are stack-specific — do NOT unify
or move them into the shared module). Fail-closed semantics (unset ARN,
Secrets Manager exception -> deny) must be unchanged; the token-cache
globals move with the functions that read them.
4. email_parsing.py: parse_raw_email as the SUPERSET version returning cc
unconditionally. WO's output is bit-identical to today; PO simply
ignores cc — do NOT "clean up" PO to consume it, and do NOT preserve two
variants.
5. emf.py: a generic emitter parameterized by namespace / dimension-sets /
properties. Every call site's emitted EMF envelope must be EXACTLY what
it emits today — the dimension-set list
[["ParseMethod"],["ParseMethod","TemplateId"]] is load-bearing for the
alarms and metric filters; making one-sided dimension fixes impossible
is the point of this move. Emission ORDERING is untouchable: PO emits
ai_fallback BEFORE the Bedrock call, WO after its gate with mutually-
exclusive ai_fallback/ai_fallback_rejected — these two deliberate
per-pipeline differences are pinned by tests; converting a call site
must not move it. The deliberate-double-count comments survive.
6. DERIVED-FIELDS EXCEPTION: if recon finds _emit_derived_agreement_metric
lives INSIDE derived_fields.py, do NOT convert it — derived_fields.py
and the shadow DerivedFieldAgreement telemetry are UNTOUCHABLE while the
bake runs (this outranks the emf consolidation). Leave it as a third
copy with a code comment pointing at shared/emf.py and report it in
notes/blockers. Only convert it if it lives outside derived_fields.py.
7. UNTOUCHABLE FILES (git diff ${BASE}...HEAD must be empty for each):
both template_parser.py (990 vs 508 lines, genuinely divergent — stays
per-pipeline), derived_fields.py, both validate_ai_fallback gate
modules, extract_with_claude's Bedrock invocation/prompt logic. Handler
diffs are LIMITED to: deleting moved code, import changes, and
emitter-call swaps per the binding spec. No opportunistic refactors.
8. Bundling (cdk): append 'cp shared/*.py /asset-output/' to BOTH
email-processor bundling commands (Phase 2 made the bundling cwd the
../lambdas asset root, so the path resolves as written).
BASE-AWARENESS — READ THE ACTUAL cdk FILES ON THE BASE, DO NOT ASSUME:
this phase is stacked on the Phase 7 branch, which ALREADY removed the
pip install step from both email-processor commands (they are now
CP-ONLY: no pip line, no manylinux pin, because nothing third-party is
installed). Phase 3 moves only pure first-party modules (no new deps),
so cp-only STAYS cp-only — you simply add another 'cp shared/*.py
/asset-output/' line. DO NOT re-introduce a pip install step and DO NOT
re-add the manylinux pin: there is nothing to pin when nothing installs,
and adding a pip step back would be the regression here. (The PR #34
lesson — never drop the manylinux pin while a pip install runs — still
holds ONLY if recon finds a pip step actually present on the base; on
the cp-only Phase 7 base there is none.) PRESERVE the Phase 2 exclude
lists exactly. Both web_ui functions gain the SAME staging mechanism
(widened-root bundled from_asset) so web_ui_auth.py ships beside their
handler — the spec agent pins the exact form. site_extractor's
from_asset is UNTOUCHED (Phase 6 territory).
9. tests/test_bundle_consistency.py is updated IN THE SAME CHANGE without
losing teeth: the PO_EXPECTED_TOP_LEVEL_MODULES exact-set pin gains the
shared modules that now ship; the AST sibling-import check must resolve
imports whose source file now lives under shared/; the new shared cp
line gets its own revert/mutation detection (a commented-out
'cp shared/*.py' must fail the test).
10. Test plumbing in the same PR: update _SIBLING_MODULES resolution and
_po_parser_support.py (~line 71 — verify the current line) so tests
load ses_auth/email_parsing/emf from shared/; retire or repoint the
fixture-hygiene test that polices the two ses_auth copies for
byte-identity (obsolete once there is one copy — do not leave it
failing); drop the ses_auth fixture params from test_ses_auth
(parameterizing over two identical copies is dead weight — halves the
run). The sys.modules save/restore dance SURVIVES for template_parser
(still a duplicated bare name) — do not delete it. Preserve the
load-bearing moto-before-handler import ordering.
11. STALE-SHADOW HAZARD: after the move, no stale ses_auth.py / .pyc /
__pycache__ copy may remain anywhere it could shadow the shared copy —
in the repo (git rm, don't empty), in staged assets, or on any test
sys.path. Verify explicitly.
12. cdk diff on BOTH stacks: the only resource deltas allowed are Code/
S3Key (asset) changes on the two email processors and the two web_ui
functions (+ CDK metadata). No logical-ID changes, no alarm, IAM,
table, env, runtime, or handler-property deltas of any kind.
13. AWS access is READ-ONLY (get-function, downloading deployed zips via
presigned URLs). NEVER cdk deploy, never invoke, never mutate.
`
const PREAMBLE = `
You are one of several agents building refactor Phase 3 in the git repo at
${REPO} on branch ${BRANCH} (already checked out — do NOT switch branches,
do NOT create branches, do NOT commit, NEVER push, do NOT run cdk deploy or
touch AWS resources beyond read-only calls).
Authoritative spec: docs/refactor-evaluation.md, section "Phase 3".
Work ONLY in the files you are told you own; other agents are concurrently
editing other files in this same working tree.
${CONSTRAINTS}
Your final message is consumed by an orchestrator script, not a human —
return only the structured data requested.
`
// ------------------------------------------------------------------ schemas
const RECON = {
type: 'object',
required: ['summary', 'facts'],
properties: {
summary: { type: 'string' },
facts: { type: 'array', items: { type: 'string' } },
blockers: { type: 'array', items: { type: 'string' } },
},
}
const SPEC = {
type: 'object',
required: ['sharedModules', 'handlerEdits', 'webUiStaging', 'bundlingEdits', 'emfDesign', 'testPlumbing', 'parityRules', 'notes'],
properties: {
sharedModules: { type: 'string', description: 'per shared module: exact provenance (which copy is the source, sha256), full contents decision (verbatim move vs superset vs parameterized), and the public surface each importer uses' },
handlerEdits: { type: 'string', description: 'per handler (po/wo email_processor, po/wo web_ui): the exact deletions, import lines, and emitter-call swaps — file:line, nothing else may change' },
webUiStaging: { type: 'string', description: 'the complete new from_asset blocks for both web_ui functions: asset path, command or cp form, exclude list — exact code' },
bundlingEdits: { type: 'string', description: 'the two email-processor bundling command strings with cp shared/*.py appended, verbatim, plus the expected top-level module set of each resulting bundle' },
emfDesign: { type: 'string', description: 'shared/emf.py signature + per-call-site mapping proving each emitted envelope (namespace, dimension-set list, properties) is byte-equivalent to today; the derived-agreement emitter decision per constraint 6' },
testPlumbing: { type: 'string', description: 'every test/support file change: loader resolution, _SIBLING_MODULES, fixture-hygiene retirement, test_ses_auth de-parameterization, test_bundle_consistency edits with their mutation-detection shapes' },
parityRules: { type: 'string', description: 'how Verify judges staged bundles vs the deployed-zip baseline: expected first-party set per function (= deployed set + the new shared modules), ses_auth byte-identity requirement, expected removals (none beyond moved-file provenance), web_ui bundle expectations' },
notes: { type: 'string' },
},
}
const IMPL = {
type: 'object',
required: ['filesChanged', 'summary', 'checksRun'],
properties: {
filesChanged: { type: 'array', items: { type: 'string' } },
summary: { type: 'string' },
checksRun: { type: 'string' },
blockers: { type: 'array', items: { type: 'string' } },
},
}
const CHECKS = {
type: 'object',
required: ['passed', 'details'],
properties: {
passed: { type: 'boolean' },
details: { type: 'string' },
scopeViolations: { type: 'array', items: { type: 'string' } },
},
}
const PARITY = {
type: 'object',
required: ['passed', 'poVerdict', 'woVerdict', 'webUiVerdict', 'details'],
properties: {
passed: { type: 'boolean' },
poVerdict: { type: 'string', description: 'PO email-processor staged vs deployed: module-set delta exactly as spec, ses_auth bytes identical — full evidence' },
woVerdict: { type: 'string', description: 'same for WO email-processor' },
webUiVerdict: { type: 'string', description: 'both web_ui staged bundles: own *.py + web_ui_auth.py present, nothing stray, hashes deterministic' },
details: { type: 'string' },
},
}
const FINDINGS = {
type: 'object',
required: ['findings'],
properties: {
findings: {
type: 'array',
items: {
type: 'object',
required: ['title', 'severity', 'confirmed', 'evidence', 'fix'],
properties: {
title: { type: 'string' },
severity: { enum: ['critical', 'high', 'medium', 'low'] },
confirmed: { type: 'boolean' },
evidence: { type: 'string' },
fix: { type: 'string' },
},
},
},
},
}
// ------------------------------------------------------------------- setup
phase('Setup')
const setup = await agent(`
In ${REPO}:
1. SEQUENCING GATE — Phases 0, 1 AND 2 must all be on ${BASE} (Phase 3
edits the same stack files as Phase 2 and depends on its widened
../lambdas asset roots to make shared/ reachable). git fetch origin,
then pick the base ref: origin/${BASE} if that remote ref exists,
otherwise the local branch ${BASE} (a stacked local-only base is
expected and fine). Verify on the base ref:
(a) Phase 0: tests/test_bundle_consistency.py exists
(git show <baseref>:tests/test_bundle_consistency.py | head -3);
(b) Phase 1: validate_ai_fallback exists in the PO pipeline
(git grep validate_ai_fallback <baseref> -- lambdas/po);
(c) Phase 2: BOTH cdk/po_stack.py and cdk/wo_stack.py on the base ref
contain Code.from_asset("../lambdas") for the email processors
(git show <baseref>:cdk/po_stack.py | grep -n '\\.\\./lambdas', same
for wo_stack.py).
If any is missing, STOP with a blocker naming the unmet phase and do
nothing else.
2. Verify clean working tree (untracked .coverage / .claude/ / the local
44 MB lambdas/po/email_processor/package/ dir are fine; any OTHER dirt =
blocker, never stash or discard).
3. git checkout ${BASE}; then git pull --ff-only ONLY if the branch has an
upstream (a local-only base skips the pull — not a blocker); then
git checkout -b ${BRANCH}
4. gh pr list --state open --json number,title,headRefName (overlap check).
Return facts: HEAD sha, per-phase gate evidence, open PRs, blockers.
`, { label: 'setup:branch', model: 'haiku', schema: RECON })
if (!setup || (setup.blockers && setup.blockers.length)) {
return { aborted: 'setup blockers', blockers: setup ? setup.blockers : ['setup agent died'], facts: setup ? setup.facts : [] }
}
log(`Branch ${BRANCH} ready off ${BASE}. ${setup.summary}`)
// ------------------------------------------------------------------- recon
phase('Recon')
const recon = await parallel([
() => agent(`${PREAMBLE}
Read-only recon of the FOUR duplicated surfaces being extracted:
1. ses_auth.py both copies — sha256 of each (MUST match; drift = blocker),
line count, the exact import line each email-processor handler uses,
any other importer (git grep 'import ses_auth\\|from ses_auth').
2. web_ui auth block — in BOTH web_ui handlers quote with file:line the
exact boundaries of the byte-identical block (_get_auth_token, _header,
is_authenticated, the four cache globals), diff the two blocks to prove
byte-identity, quote each INFRA-74 comment verbatim (they differ —
that is expected), and note everything else in each handler that CALLS
the block.
3. parse_raw_email both copies — where each lives (own sibling file vs
inside handler.py), the exact cc delta between them, every call site.
4. The THREE EMF emitters — quote each verbatim with file:line (PO
ParseMethod emitter, WO ParseMethod emitter,
_emit_derived_agreement_metric), namespace + dimension-set list +
properties of each, and CRITICALLY: which FILE _emit_derived_agreement_metric
lives in (constraint 6 hinges on whether it is inside derived_fields.py).
Also pin current line numbers of PO's pre-Bedrock ai_fallback emit and
WO's post-gate emit.
25-35 precise facts.`,
{ label: 'recon:duplicated-modules', model: 'sonnet', phase: 'Recon', schema: RECON }),
() => agent(`${PREAMBLE}
Read-only recon of the post-Phase-2 CDK bundling state: for ALL FIVE
Code.from_asset sites in cdk/po_stack.py and cdk/wo_stack.py quote verbatim
with current file:line — asset path, full bundling command (or plain form),
exclude list. For the two web_ui functions additionally record: runtime,
architecture, handler property, memory/timeout, whether any requirements.txt
exists for them, and every function property that must NOT change when
bundling is added. Confirm lambdas/shared/ does not exist yet and list
anything at lambdas/ top level. 15-25 facts.`,
{ label: 'recon:cdk-bundling', model: 'haiku', phase: 'Recon', schema: RECON }),
() => agent(`${PREAMBLE}
Read-only recon of the test plumbing this phase must rewire:
- tests/conftest.py importlib loader: how it resolves module paths, the
sys.modules save/restore, _SIBLING_MODULES (exact current contents).
- _po_parser_support.py: the independent importlib reimplementation, the
load-bearing moto-before-handler import order (~lines 26-33), and what
is at line ~71 (the doc cites it — quote the current code).
- _wo_parser_support.py bare sys.path 'import handler' strategy.
- test_ses_auth.py at root: the fixture parameterization over both copies
(quote it), total line count, the fixture-hygiene test that polices
byte-identity between the two copies (name + assertion).
- test_parse_raw_email.py: how it loads both handlers' parse_raw_email.
- tests/test_bundle_consistency.py: every assertion that will fail red
against the Phase 3 shapes (shared cp line, PO_EXPECTED_TOP_LEVEL_MODULES,
AST sibling-import resolution for moved modules).
20-30 facts.`,
{ label: 'recon:test-plumbing', model: 'sonnet', phase: 'Recon', schema: RECON }),
() => agent(`${PREAMBLE}
Read-only AWS recon (region us-east-1, READ-ONLY): for po-email-processor,
workorder-email-processor, AND both web_ui functions (find their exact
function names via the stacks/aws lambda list-functions) run aws lambda
get-function; for the two email processors download each Code.Location
presigned zip to a scratch dir and produce the COMPLETE file list
(unzip -l) separated into (a) first-party top-level .py, (b) dependency
dirs, (c) anything else; record every function's CodeSha256 and the
sha256 of ses_auth.py inside each deployed zip (the byte-identity baseline
the moved copy is judged against). For the web_ui functions the current
zip file list is the baseline their new bundled asset must cover.
Return the categorized lists as facts.`,
{ label: 'recon:deployed-baseline', model: 'sonnet', phase: 'Recon', schema: RECON }),
])
const reconOk = recon.filter(Boolean)
const pack = reconOk.map(r => `## ${r.summary}\n${r.facts.join('\n')}`).join('\n\n')
const reconBlockers = reconOk.flatMap(r => r.blockers || [])
.filter(b => b && !/^\s*(none|n\/a)\b/i.test(b))
log(`Recon complete: ${reconOk.length}/4 mappers, ${reconBlockers.length} blockers`)
if (reconBlockers.length) {
return { aborted: 'recon blockers (likely copy drift — resolve before extracting)', blockers: reconBlockers, reconPack: pack }
}
// -------------------------------------------------------------------- spec
phase('Spec')
const spec = await agent(`${PREAMBLE}
You are the SPEC agent — the single authority that pins every contested
decision BEFORE parallel implementation (parallel leaves cannot see each
other's choices). Using the recon pack below plus your own reads of the
actual files, produce the binding implementation spec:
- sharedModules: per constraint 1's four modules — provenance, contents
decision, public surface. ses_auth is a verbatim byte-move; web_ui_auth
is the exact block; email_parsing is the WO superset (cc unconditional);
emf is the parameterized emitter.
- handlerEdits: for each of the four handlers, the exact minimal edit list
(constraint 7 — deletions, imports, emitter-call swaps ONLY). State
explicitly that the two email-processor ses_auth import lines are
UNCHANGED.
- webUiStaging: complete replacement from_asset blocks for both web_ui
functions — widened root, cp command staging the function's own *.py
plus web_ui_auth.py (pin whether to cp shared/*.py or just
shared/web_ui_auth.py — pick ONE rule, state why), Phase-2-style
excludes. Mind bundling cwd semantics: the container mounts the asset
root (../lambdas) as the working dir. If a no-Docker staging form is
viable and simpler, you may pin that instead — but ONE mechanism for
both, exact code.
- bundlingEdits: both email-processor command strings verbatim with
'cp shared/*.py /asset-output/' appended, plus the exact expected
top-level module set of each resulting bundle (this feeds
PO_EXPECTED_TOP_LEVEL_MODULES and the parity gate).
- emfDesign: the shared emitter signature and a per-call-site table
proving envelope byte-equivalence; resolve the derived-agreement emitter
per constraint 6 based on where recon found it.
- testPlumbing: every file, every edit — loader resolution for shared/,
_SIBLING_MODULES, _po_parser_support.py, _wo_parser_support.py if it
needs the shared path, fixture-hygiene retirement, test_ses_auth
de-parameterization, test_parse_raw_email (single implementation now —
what does it exercise?), test_bundle_consistency edits including the new
mutation shapes (commented-out shared cp must fail).
- parityRules: exactly how Verify judges staged bundles vs recon's
deployed baseline — per-function expected first-party set (deployed set
minus nothing, plus the shared modules per your cp rule), ses_auth
sha256 must equal the deployed zips' copy, web_ui bundles must cover
their deployed file list plus web_ui_auth.py, nothing else may appear
or vanish.
Recon pack:\n${pack}`,
{ label: 'spec:pin-extraction', phase: 'Spec', schema: SPEC })
if (!spec) return { aborted: 'spec agent died — rerun workflow', reconBlockers }
const specBlock = `BINDING SPEC (from the spec agent — implement EXACTLY this):\n${JSON.stringify(spec, null, 2)}`
log('Spec pinned: shared modules, handler edits, web_ui staging, bundling strings, EMF design, test plumbing')
// --------------------------------------------------------------- implement
phase('Implement')
const impl = await parallel([
() => agent(`${PREAMBLE}
YOU OWN: everything under lambdas/ EXCEPT the tests/ directories (another
agent owns all test and support files). Do not touch cdk/ or README.
Task: execute the four moves per spec.sharedModules + spec.handlerEdits +
spec.emfDesign, in the spec's order:
1. git mv (or create+git rm preserving exact bytes) ses_auth.py ->
lambdas/shared/ses_auth.py; delete both originals; prove sha256
equality in your summary. Handler import lines untouched.
2. Create lambdas/shared/web_ui_auth.py from the byte-identical block;
replace the block in BOTH web_ui handlers with the import; keep each
INFRA-74 comment in place.
3. Create lambdas/shared/email_parsing.py (superset); rewire both
email-processor call sites.
4. Create lambdas/shared/emf.py; swap the emitter call sites per
spec.emfDesign — honoring constraint 6 (derived_fields.py stays
untouched if the third emitter lives there) and constraint 5 (emission
ordering does not move).
Delete every now-empty moved-out file with git rm (constraint 11).
Run before returning: ruff check lambdas && ruff format lambdas --check,
plus an import smoke: python3 -c with sys.path prepended for
lambdas/shared + each function dir, importing every touched module (tests
are NOT yours to run — the plumbing agent lands them in parallel).
${specBlock}`,
{ label: 'impl:shared-lambdas', model: 'opus', phase: 'Implement', schema: IMPL }),
() => agent(`${PREAMBLE}
YOU OWN: cdk/po_stack.py and cdk/wo_stack.py ONLY (and only the from_asset
regions — alarms, IAM, tables, env are all off-limits).
Task: apply spec.bundlingEdits (append the shared cp to both
email-processor commands — the base is the Phase 7 CP-ONLY bundling, so
just add another 'cp shared/*.py /asset-output/' line; DO NOT re-add a pip
install step or a manylinux pin, there is nothing third-party to install
(constraint 8); preserve the Phase 2 excludes verbatim) and
spec.webUiStaging (both web_ui functions). Touch nothing else;
site_extractor's from_asset stays byte-identical.
Run before returning: ruff check cdk && cd cdk &&
npx cdk synth po-ingest -q -o /tmp/phase3-synth &&
npx cdk synth workorder-ingest -q -o /tmp/phase3-synth (artifact-id
selectors, NOT stack_name). Docker bundling runs — confirm each staged
email-processor asset contains the shared modules and each staged web_ui
asset contains web_ui_auth.py; note asset hashes in your summary. If the
lambdas/ moves have not landed yet the synth will fail on missing files —
poll by re-running up to ~10 min before reporting a blocker.
${specBlock}`,
{ label: 'impl:cdk-stacks', model: 'sonnet', phase: 'Implement', schema: IMPL }),
() => agent(`${PREAMBLE}
YOU OWN: all test and support files (tests/ at repo root including
tests/test_bundle_consistency.py and tests/conftest.py,
lambdas/po/email_processor/tests/, lambdas/wo/email_processor/tests/)
and README.md ONLY.
Task A: apply spec.testPlumbing in full — loader/_SIBLING_MODULES
resolution for shared/, _po_parser_support.py edit (preserving the
moto-before-handler ordering comment), fixture-hygiene retirement,
test_ses_auth de-parameterization, test_parse_raw_email update,
test_bundle_consistency updates WITHOUT losing teeth (constraint 9 — the
new shared cp pin must reject a commented-out or narrowed variant; keep
the existing mutation tests green).
Task B: README — document lambdas/shared/ in the repo-layout section
(which modules live there, the flat-landing import rule, the bundling cp
that ships them), update the bundling/deploy-guards paragraphs for the
shared cp + web_ui staging, and note that ses_auth is now single-sourced
(one hardening fix lands once). Match existing README style.
Run before returning: pytest -q --no-cov at repo root (must be green
against the OTHER agents' edits — they land in parallel; poll by
re-running up to ~10 min before reporting a blocker) and ruff check on
every file you touched.
${specBlock}`,
{ label: 'impl:test-plumbing-readme', model: 'opus', phase: 'Implement', schema: IMPL }),
])
const implOk = impl.filter(Boolean)
const implBlockers = implOk.flatMap(r => r.blockers || [])
log(`Implement complete: ${implOk.length}/3 agents, blockers: ${implBlockers.length}`)
// ---------------------------------------------------- verify + fix loop
const EXPECTED_SCOPE = [
'lambdas/shared/',
'lambdas/po/email_processor/',
'lambdas/wo/email_processor/',
'lambdas/po/web_ui/',
'lambdas/wo/web_ui/',
'cdk/po_stack.py',
'cdk/wo_stack.py',
'tests/',
'README.md',
]
const mechanicalPrompt = `${PREAMBLE}
Independent re-verification — trust nothing self-reported. Run ALL gates,
quoting failures verbatim:
1. pytest -q --no-cov (repo root, all three roots green)
2. ruff check . && ruff format --check .
3. cd cdk && npx cdk synth po-ingest -q && npx cdk synth workorder-ingest -q
4. cd cdk && npx cdk diff po-ingest ; npx cdk diff workorder-ingest —
constraint 12: only the four functions' Code/S3Key (+ metadata) deltas.
Any alarm/IAM/env/runtime/handler-prop/logical-ID delta = FAIL. Paste
the diff summaries.
5. UNTOUCHABLES (each must output NOTHING):
git diff ${BASE}...HEAD -- lambdas/po/email_processor/derived_fields.py
git diff ${BASE}...HEAD -- lambdas/po/email_processor/template_parser.py
git diff ${BASE}...HEAD -- lambdas/wo/email_processor/template_parser.py
plus both validate_ai_fallback gate modules (resolve their filenames
first) and lambdas/po/site_extractor/.
6. BYTE-IDENTITY: sha256 of lambdas/shared/ses_auth.py equals sha256 of
git show ${BASE}:lambdas/po/email_processor/ses_auth.py (and the wo
copy). Both originals gone from the tree (git ls-files check) and no
stray ses_auth.py/__pycache__ anywhere under lambdas/ outside shared/
(constraint 11).
7. ORDERING PINS: grep line numbers proving PO's ai_fallback emit still
precedes the Bedrock invoke and WO's emits are untouched relative to
${BASE} (the wo handler diff must contain ONLY the spec's edit classes).
8. INFRA-74: both web_ui handlers still contain their own INFRA-74
comment verbatim per ${BASE}.
9. git status --porcelain scope check: every modified/added/deleted path
under ${EXPECTED_SCOPE.join(', ')} (untracked .coverage/.claude/
package/ tolerated).
passed=true only if all green. YOU MAY NOT edit files.`
const parityPrompt = `${PREAMBLE}
You are the BUNDLE-PARITY verifier — the load-bearing gate of this phase.
Everything is local synth + read-only AWS.
1. cd cdk && npx cdk synth po-ingest -q -o /tmp/phase3-parity &&
npx cdk synth workorder-ingest -q -o /tmp/phase3-parity. Locate all
four staged assets (two email processors, two web_ui).
2. Re-download both email-processor deployed zips fresh (aws lambda
get-function Code.Location, us-east-1) — do not trust a recon cache —
and fetch both web_ui functions' deployed file lists.
3. Judge per spec.parityRules: per email processor, staged first-party
top-level .py set == deployed set PLUS exactly the shared modules the
spec's cp rule ships (list both sets; nothing else may appear or
vanish); sha256 of staged ses_auth.py == sha256 of the ses_auth.py
inside each deployed zip (fail-closed auth byte-identical through the
move); dependency packages compared by name, version drift noted not
failed. Per web_ui function: staged bundle covers the deployed file
list plus web_ui_auth.py, nothing stray (no tests/, no other
pipeline's sources, no .eml, no package/).
4. DETERMINISM: synth po-ingest twice into fresh -o dirs — identical
asset hashes, including the NEW web_ui assets. Then drop a throwaway
__pycache__/junk.pyc under lambdas/shared/ (delete it afterwards),
re-synth, and confirm the excludes keep every hash unchanged.
5. IMPORT-RESOLUTION sanity: inside each staged email-processor asset
dir run python3 -c "import handler" with that dir alone on sys.path
(env-var stubs as needed) — proves the flat landing satisfies every
import including 'from ses_auth import ...' with the moved copy.
passed=true only if every check holds.`
const lenses = [
{ key: 'auth-integrity', prompt: `${PREAMBLE}
ADVERSARIAL REVIEW — auth-integrity lens. Auth code moved; try to prove
the move WEAKENED it. (1) ses_auth: byte-compare the shared copy against
${BASE}'s copies yourself; then attack import resolution — in the BUNDLE
and in TESTS, which ses_auth wins if anything shadows (stale .pyc, a
same-named module on sys.path, the tests loader resolving the old path
silently to a stub)? Could a test now pass against a MOCK of ses_auth
where it previously exercised the real module? (2) web_ui_auth: diff the
extracted block against both originals — any dropped line, changed
global, or reordered check? Is the fail-closed path (unset ARN, Secrets
exception, wrong token -> 401 before any table access) provably
unchanged? Do the four cache globals still behave per-function (module
now shared — could cross-importer state ever leak)? (3) The gate call
sites: could any handler path now reach S3-fetch/Bedrock/save before
authenticate_inbound_email or is_authenticated, where it could not
before? confirmed=true only with a concrete exploit sketch or
file:line proof.` },
{ key: 'telemetry-emf', prompt: `${PREAMBLE}
ADVERSARIAL REVIEW — EMF/telemetry lens. The alarms and metric filters
consume exact EMF shapes; a silent envelope change breaks paging without
failing any test. Read shared/emf.py + every converted call site
(git diff ${BASE}) and the synthesized templates' metric filters/alarms.
Verify per call site: namespace exact, dimension-set list EXACT
([["ParseMethod"],["ParseMethod","TemplateId"]] where applicable, order
included), property keys/types identical, timestamp/CloudWatchMetrics
envelope structure identical — construct a sample emission per call site
and diff it against ${BASE}'s hand-built _aws dict output. Then the
ordering pins: PO ai_fallback still pre-Bedrock, WO still post-gate
mutually-exclusive, the deliberate double-count comment intact. Finally
constraint 6: where does _emit_derived_agreement_metric live and was the
decision honored (derived_fields.py diff empty)? confirmed=true only
with evidence.` },
{ key: 'loader-integrity', prompt: `${PREAMBLE}
ADVERSARIAL REVIEW — test/loader-integrity lens. The three loading idioms
were rewired; attack them. (1) Does test_ses_auth now exercise the REAL
lambdas/shared/ses_auth.py (trace the loader path), and did
de-parameterization silently drop any assertion that only ran under one
param? (2) Fixture-hygiene: the byte-identity police is retired — does
anything still guard against a future stray ses_auth.py copy reappearing
in a pipeline dir (should the bundle-consistency or a hygiene test)? If
nothing does, that is a finding. (3) sys.modules save/restore: prove it
still isolates template_parser between pipelines (run two cross-pipeline
tests back to back); prove moto-before-handler ordering survived in
_po_parser_support.py. (4) test_bundle_consistency: hand-mutate command
strings on scratch copies — a commented-out 'cp shared/*.py', a shared
glob narrowed to one file, and a PO_EXPECTED_TOP_LEVEL_MODULES missing a
shared module must each FAIL. (5) Run pytest twice in one session and in
file-shuffled order (-p no:randomly not installed? then two explicit
orderings) to smoke out import-order coupling introduced by the shared
path. confirmed=true only with file:line or reproduced-failure
evidence.` },
]
let round = 0
let checks = null
let parity = null
let confirmed = []
while (round < 3) {
phase('Verify')
const results = await parallel([
() => agent(mechanicalPrompt, { label: `verify:mechanical-r${round}`, model: 'sonnet', phase: 'Verify', schema: CHECKS }),
() => agent(parityPrompt, { label: `verify:parity-r${round}`, model: 'opus', phase: 'Verify', schema: PARITY }),
...lenses.map(l => () =>
agent(l.prompt, { label: `verify:${l.key}-r${round}`, phase: 'Verify', schema: FINDINGS })),
])
checks = results[0]
parity = results[1]
confirmed = results.slice(2).filter(Boolean)
.flatMap(r => r.findings || [])
.filter(f => f.confirmed && f.severity !== 'low')
const green = checks && checks.passed && parity && parity.passed
log(`Verify round ${round}: mechanical ${checks && checks.passed ? 'GREEN' : 'RED'}, parity ${parity && parity.passed ? 'GREEN' : 'RED'}, confirmed findings: ${confirmed.length}`)
if (green && confirmed.length === 0) break
round += 1
if (round >= 3) break
phase('Fix')
await agent(`${PREAMBLE}
You are the fix agent — you may edit files under: ${EXPECTED_SCOPE.join(', ')}.
Fix EVERY item below minimally; the binding spec and 13 pinned constraints
still hold (a finding that conflicts with a constraint is reported, not
"fixed" — the constraint wins, esp. constraint 6's derived_fields
untouchability and constraint 5's emission ordering). Re-run the specific
failing gate/test per fix.
MECHANICAL:\n${checks ? checks.details : '(agent died — rerun all gates)'}
PARITY:\n${parity ? parity.details : '(agent died — rerun all)'}
CONFIRMED FINDINGS:\n${JSON.stringify(confirmed, null, 2)}
${specBlock}`,
{ label: `fix:round-${round}`, model: 'opus', phase: 'Fix', schema: IMPL })
}
const verifyClean = checks && checks.passed && parity && parity.passed && confirmed.length === 0
if (!verifyClean) {
return {
status: 'NEEDS ATTENTION — verify not clean after 3 rounds; branch left uncommitted',
branch: BRANCH,
mechanical: checks,
parity,
unresolvedFindings: confirmed,
implBlockers,
reconBlockers,
spec,
}
}
// ----------------------------------------------------------------- package
phase('Package')
const commit = await agent(`${PREAMBLE.replace('do NOT commit, ', '')}
YOU are the commit agent:
1. Read ~/Documents/repositories/seahaven/engineering-handbook/commit-messages.md
and follow it exactly.
2. git add only paths under: ${EXPECTED_SCOPE.join(', ')} and
.claude/workflows/phase-3-shared-extraction.js. NOT .coverage, NOT
package/. Verify the staged set with git status — the deletions of the
moved originals MUST be staged too.
3. ONE commit; write the message to /tmp/phase3-commit-msg.txt and use
git commit -F /tmp/phase3-commit-msg.txt (backticks in -m get eaten by
zsh). Suggested subject:
"feat: extract lambdas/shared/ — single-source ses_auth, web_ui auth, email parsing, EMF emitter (refactor phase 3)"
Body: the four moves with the byte-identity evidence one-liner for
ses_auth, the flat-landing import rule, the shared cp bundling change +
web_ui staging, the derived-agreement emitter decision (constraint 6),
and the test-plumbing summary. NO AI attribution / Co-Authored-By
lines.
4. Do NOT push. Return commit sha + shortstat in summary.`,
{ label: 'package:commit', model: 'sonnet', phase: 'Package', schema: IMPL })
return {
status: 'BUILT — committed locally, NOT pushed',
branch: BRANCH,
base: BASE,
commit: commit ? commit.summary : 'commit agent died — commit manually',
spec: { sharedModules: spec.sharedModules, webUiStaging: spec.webUiStaging, emfDesign: spec.emfDesign, derivedAgreementNote: spec.notes },
parityEvidence: parity ? { po: parity.poVerdict, wo: parity.woVerdict, webUi: parity.webUiVerdict } : null,
implementation: implOk.map(r => r.summary),
filesChanged: implOk.flatMap(r => r.filesChanged),
verifyRounds: round + 1,
blockers: implBlockers.concat(reconBlockers),
outstandingGates: [
'/sh-security-review (MANDATORY before push — auth code moved: ses_auth + the web_ui auth gate are exactly the authentication surface; run on the committed diff, and re-run after any post-review fix to this code)',
'cross-family cross_review.py NOT mandatory (no IAM change; handler event/return contracts unchanged — internal module moves only). Opt-in if judgment says so',
'push + PR + gh pr checks green',
'deploy-then-merge: deploy from branch, smoke green, one real PO + WO email each, watch BOTH sender-auth-rejected alarms through live mail (the auth move must not change accept/reject behavior), verify web_ui login still works, THEN merge — and note the post-merge CI redeploy being a no-op (unchanged asset hashes) is itself a verification signal',
],
}

View file

@ -0,0 +1,684 @@
export const meta = {
name: 'phase-4-cdk-common',
description: 'Phase 4 of the procurement-ingest refactor (docs/refactor-evaluation.md): collapse the 379 identical CDK lines into cdk/common.py as PLAIN FUNCTIONS taking (scope, id, ...) — called with the SAME Stack scope and the SAME construct ids the stacks use today, so every logical ID is byte-stable (Construct-subclass wrapping is forbidden: it would reparent the tree and attempt REPLACEMENT of the RETAIN-protected purchase-orders/WorkOrders tables and named buckets = data loss). Extracts add_ddb_alarms, add_sender_auth_rejected_alarm, add_standard_lambda_alarms, make_bedrock_invoke_statement (account/region-derived ARN, not hardcoded 328440206208), make_email_bucket, make_processor_dlq, make_fallback_rate_alarm — preserving every per-function alarm variance byte-for-byte. Same PR: account on both cdk.Environment, constructs== exact pin, stale-comment fix, CfnOutputs for five function ARNs + consumed table names. ZERO cdk diff on both stacks is the acceptance test. The make_bedrock_invoke_statement IAM PolicyStatement move triggers mandatory GPT-4.1 cross-family review even though semantics are identical. Committed locally, never pushed.',
phases: [
{ title: 'Setup', detail: 'verify Phases 0+1+2+3 on base, branch feature/phase-4-cdk-common', model: 'haiku' },
{ title: 'Recon', detail: '4 mappers: the 379 duplicated CDK lines + per-function alarm variance, the two inline Bedrock PolicyStatements, the fallback-rate + rejected alarm math (post-Phase-1), app.py/pins/outputs surface' },
{ title: 'Spec', detail: 'serial fable spec: pin common.py contents + every function signature, per-stack call-site rewrites, alarm-variance table, Bedrock IAM equivalence, fallback-rate call params, app/pins/CfnOutputs, zero-diff judging rules' },
{ title: 'Implement', detail: 'opus: cdk/common.py; opus: both stacks (rewire + CfnOutputs); sonnet: app.py + requirements pin + README — disjoint files', model: 'opus' },
{ title: 'Verify', detail: 'mechanical gates + zero-cdk-diff verifier (the load-bearing gate) + 3 fable lenses (logical-ID safety, alarm variance, IAM equivalence)' },
{ title: 'Fix', detail: 'opus fixer, full re-verify, max 3 rounds', model: 'opus' },
{ title: 'Package', detail: 'single commit via -F (no push); runs cross_review.py inline on the IAM diff', model: 'sonnet' },
],
}
// ---------------------------------------------------------------- constants
const REPO = '/Users/adammoussa/Documents/repositories/seahaven/procurement-ingest'
const BRANCH = 'feature/phase-4-cdk-common'
let _args = args
if (typeof _args === 'string') {
try { _args = JSON.parse(_args) } catch (e) { _args = null }
}
const BASE = (_args && _args.base) || 'main'
const CONSTRAINTS = `
PINNED BEHAVIORAL CONSTRAINTS (docs/refactor-evaluation.md Phase 4 — violating any is a build failure):
1. EXTRACT AS PLAIN FUNCTIONS taking (scope, id, ...), called with the SAME
scope (the Stack instance) and the SAME construct ids the stacks use
today -> 100% logical-ID-safe. NEVER wrap in Construct subclasses: a
subclass inserts a tree node, changes EVERY child logical ID, and would
attempt REPLACEMENT of the RETAIN-protected purchase-orders / WorkOrders
tables and the named buckets = DATA LOSS. This is THE load-bearing rule
of the phase — every other check exists to defend it.
2. New module cdk/common.py. Extract exactly:
- _DDB_ALARM_OPERATIONS + add_ddb_alarms
- add_sender_auth_rejected_alarm
- add_standard_lambda_alarms(scope, id_prefix, fn, name_prefix, topic, *,
duration_statistic, errors=True, dlq=None, descriptions=...)
- make_bedrock_invoke_statement (DERIVE the inference-profile ARN + the
per-region foundation-model ARNs from Stack.of(scope).account /
Stack.of(scope).region — NOT hardcoded 328440206208)
- make_email_bucket
- make_processor_dlq
- make_fallback_rate_alarm(namespace, rejected_included, period, threshold,
floor, evaluation_periods, datapoints_to_alarm) reproducing the
expression strings / FILL / labels BYTE-FOR-BYTE.
3. PER-FUNCTION ALARM VARIANCE — preserve EXACTLY, do NOT homogenize:
PO email-processor duration p99 vs wo-email-processor p95; po-web-ui
throttles+duration only; site_extractor no-DLQ; wo web_ui has ZERO alarms
(do NOT let the shared helper silently add any); every bespoke alarm
DESCRIPTION string is passed through verbatim.
4. FALLBACK-RATE RECONCILIATION (post-Phase-1): PO's template-fallback-rate
EXCLUDES the rejected series (byte-identical to today, a pre-call
double-count would result otherwise) -> call make_fallback_rate_alarm with
rejected_included=False; WO's INCLUDES rejected -> rejected_included=True.
The two REJECTED alarms are a DIFFERENT shape and are NOT
make_fallback_rate_alarm: PO's ai-fallback-rejected is a 6h count-floor
IF(FILL(rej,0)>=1,...) alarm (added in Phase 1); WO's is the 5-min /
2-of-6 sparse idiom. Keep those as DISTINCT call sites (or a separate
dedicated helper) — do NOT force them through make_fallback_rate_alarm.
NO element-wise MAX anywhere (post-#102 rule).
5. Do NOT import stack-specific services (kms / ssm / event_sources) into
common.py — only the constructs the shared helpers actually need.
6. SAME PR, net-new & logical-ID-safe additions:
- add account='328440206208' to BOTH cdk.Environment calls
- pin constructs== to the exact installed version (not a floor >=)
- fix the stale "2.259.0" version comments
- add CfnOutputs for the FIVE function ARNs + the consumed table names.
These are additive; CfnOutputs and account are ID-safe. Verify none of
them perturbs an existing logical ID.
7. ZERO lambdas/ diff: git diff ${BASE}...HEAD -- lambdas/ must be EMPTY.
This phase is CDK-ONLY. tests/ may gain a cdk-diff / synth-only test for
the acceptance gate, but NO other tests/ change and NO lambdas/ change.
8. ZERO cdk diff on BOTH stacks is the acceptance test: npx cdk diff
po-ingest and npx cdk diff workorder-ingest must show ZERO resource
changes (no logical-ID, alarm, IAM, table, bucket, env, or metadata
delta beyond CDK-tooling noise). The CfnOutputs are the ONLY net-new
resources allowed to appear, and only as additions.
9. The wo artifact id is 'workorder-ingest' (the construct id / 2nd
positional arg), NOT 'WorkorderIngestStack' (that is stack_name). ALWAYS
drive synth/diff by the artifact id: npx cdk synth workorder-ingest,
npx cdk diff workorder-ingest. Using the stack_name selector fails.
10. NO cdk deploy, NO invoke, NO AWS mutation. Read-only AWS only if needed
(e.g. confirming deployed alarm names) — the diff gate is a pure local
synth-vs-synth comparison.
`
const PREAMBLE = `
You are one of several agents building refactor Phase 4 in the git repo at
${REPO} on branch ${BRANCH} (already checked out — do NOT switch branches,
do NOT create branches, do NOT commit, NEVER push, do NOT run cdk deploy or
touch AWS resources beyond read-only calls).
Authoritative spec: docs/refactor-evaluation.md, section "Phase 4".
Work ONLY in the files you are told you own; other agents are concurrently
editing other files in this same working tree.
${CONSTRAINTS}
Your final message is consumed by an orchestrator script, not a human —
return only the structured data requested.
`
// ------------------------------------------------------------------ schemas
const RECON = {
type: 'object',
required: ['summary', 'facts'],
properties: {
summary: { type: 'string' },
facts: { type: 'array', items: { type: 'string' } },
blockers: { type: 'array', items: { type: 'string' } },
},
}
const SPEC = {
type: 'object',
required: ['commonModule', 'poStackEdits', 'woStackEdits', 'alarmVariance', 'bedrockStatement', 'fallbackRateCalls', 'appAndPins', 'diffRules', 'notes'],
properties: {
commonModule: { type: 'string', description: 'the full cdk/common.py: every function (add_ddb_alarms + _DDB_ALARM_OPERATIONS, add_sender_auth_rejected_alarm, add_standard_lambda_alarms, make_bedrock_invoke_statement, make_email_bucket, make_processor_dlq, make_fallback_rate_alarm) with its EXACT signature, and the exact imports it needs (no stack-specific kms/ssm/event_sources per constraint 5)' },
poStackEdits: { type: 'string', description: 'cdk/po_stack.py: every inline block replaced by a common.* call, file:line, with the exact scope + construct-id + kwargs each call passes so the emitted resource is byte-identical; plus the PO CfnOutput additions (function ARNs + consumed table names)' },
woStackEdits: { type: 'string', description: 'cdk/wo_stack.py: same — call-site rewrites file:line preserving construct ids, plus WO CfnOutput additions; explicitly note wo web_ui gets NO alarms (constraint 3)' },
alarmVariance: { type: 'string', description: 'the per-function alarm-variance table proving each helper call reproduces exactly what the inline code emits today: PO p99 vs wo-email-processor p95, po-web-ui throttles+duration only, site_extractor no-DLQ, wo web_ui ZERO alarms, every bespoke description string mapped verbatim' },
bedrockStatement: { type: 'string', description: 'make_bedrock_invoke_statement: the exact actions + resources, showing how the inference-profile ARN and per-region FM ARNs are derived from Stack.of(scope).account/.region, and PROVING the derived strings resolve in-account to the SAME ARNs the two inline PolicyStatements hardcode today' },
fallbackRateCalls: { type: 'string', description: 'the make_fallback_rate_alarm call params for PO (rejected_included=False) and WO (rejected_included=True) reproducing the expression/FILL/label strings byte-for-byte; PLUS how the two DISTINCT rejected alarms stay distinct call sites (PO 6h count-floor IF(FILL(rej,0)>=1,...); WO 5-min/2-of-6 sparse) — NOT folded into make_fallback_rate_alarm, NO element-wise MAX' },
appAndPins: { type: 'string', description: 'app.py account= additions to both cdk.Environment calls; the exact constructs== pin (installed version); the stale "2.259.0" comment fix locations + new text; confirmation none perturbs a logical ID' },
diffRules: { type: 'string', description: 'exactly how Verify proves zero cdk diff: synth BASE and HEAD into separate temp dirs, diff the two stacks templates, ANY resource/logical-ID/property delta = FAIL, the ONLY allowed additions are the net-new CfnOutputs' },
notes: { type: 'string' },
},
}
const IMPL = {
type: 'object',
required: ['filesChanged', 'summary', 'checksRun'],
properties: {
filesChanged: { type: 'array', items: { type: 'string' } },
summary: { type: 'string' },
checksRun: { type: 'string' },
blockers: { type: 'array', items: { type: 'string' } },
},
}
const CHECKS = {
type: 'object',
required: ['passed', 'details'],
properties: {
passed: { type: 'boolean' },
details: { type: 'string' },
scopeViolations: { type: 'array', items: { type: 'string' } },
},
}
const DIFF = {
type: 'object',
required: ['passed', 'poDiffVerdict', 'woDiffVerdict', 'details'],
properties: {
passed: { type: 'boolean' },
poDiffVerdict: { type: 'string', description: 'po-ingest BASE-synth vs HEAD-synth: ZERO resource/logical-ID/property changes (CfnOutputs the only allowed net-new additions) — full template-diff evidence' },
woDiffVerdict: { type: 'string', description: 'workorder-ingest (artifact id, NOT WorkorderIngestStack): same zero-change evidence' },
details: { type: 'string' },
},
}
const FINDINGS = {
type: 'object',
required: ['findings'],
properties: {
findings: {
type: 'array',
items: {
type: 'object',
required: ['title', 'severity', 'confirmed', 'evidence', 'fix'],
properties: {
title: { type: 'string' },
severity: { enum: ['critical', 'high', 'medium', 'low'] },
confirmed: { type: 'boolean' },
evidence: { type: 'string' },
fix: { type: 'string' },
},
},
},
},
}
// ------------------------------------------------------------------- setup
phase('Setup')
const setup = await agent(`
In ${REPO}:
1. SEQUENCING GATE — Phases 0, 1, 2 AND 3 must all be on ${BASE} (Phase 4
dedups BOTH cdk stacks, which Phases 1 (po_stack alarms), 2 (bundling
roots) and 3 (shared cp) all edited — building against a pre-Phase-3
stack file guarantees a conflict and a wrong diff baseline). git fetch
origin, then pick the base ref: origin/${BASE} if that remote ref
exists, otherwise the local branch ${BASE} (a stacked local-only base is
expected and fine). Verify on the base ref:
(a) Phase 3: lambdas/shared/ exists
(git ls-tree <baseref> -- lambdas/shared | head);
(b) Phase 2: BOTH cdk/po_stack.py and cdk/wo_stack.py contain
Code.from_asset("../lambdas") for the email processors
(git show <baseref>:cdk/po_stack.py | grep -n '\\.\\./lambdas', same
for wo_stack.py);
(c) Phase 1: the po-email-processor-ai-fallback-rejected alarm /
EmailProcessorAiFallbackRejectedAlarm construct exists in po_stack.py
(git show <baseref>:cdk/po_stack.py | grep -n 'ai-fallback-rejected\\|AiFallbackRejected').
If Phase 3 is not on ${BASE}, STOP with a blocker (Phase 4 needs the
post-Phase-3 stack files as its zero-diff baseline). If any other phase
is missing, STOP with a blocker naming the unmet phase and do nothing
else.
2. Verify clean working tree (untracked .coverage / .claude/ / the local
lambdas/po/email_processor/package/ dir are fine; any OTHER dirt =
blocker, never stash or discard).
3. git checkout ${BASE}; then git pull --ff-only ONLY if the branch has an
upstream (a local-only base skips the pull — not a blocker); then
git checkout -b ${BRANCH}
4. gh pr list --state open --json number,title,headRefName (overlap check).
Return facts: HEAD sha, per-phase gate evidence, open PRs, blockers.
`, { label: 'setup:branch', model: 'haiku', schema: RECON })
if (!setup || (setup.blockers && setup.blockers.length)) {
return { aborted: 'setup blockers', blockers: setup ? setup.blockers : ['setup agent died'], facts: setup ? setup.facts : [] }
}
log(`Branch ${BRANCH} ready off ${BASE}. ${setup.summary}`)
// ------------------------------------------------------------------- recon
phase('Recon')
const recon = await parallel([
() => agent(`${PREAMBLE}
Read-only recon of the ~379 duplicated CDK lines and the PER-FUNCTION ALARM
VARIANCE that MUST survive the dedup (constraint 3). In cdk/po_stack.py and
cdk/wo_stack.py, quote verbatim with file:line:
1. _DDB_ALARM_OPERATIONS + add_ddb_alarms (both copies — are they
byte-identical? diff them).
2. add_sender_auth_rejected_alarm (both copies).
3. Every add_standard_lambda_alarms-shaped block: for EACH function
(po email-processor, wo email-processor, po web_ui, wo web_ui,
po site_extractor) list the exact alarm set, the duration statistic
(prove PO email p99 vs wo email p95), whether throttles/errors/dlq
alarms are present, and QUOTE every bespoke alarm description string.
CRITICALLY: confirm wo web_ui has ZERO alarms today.
4. make_email_bucket / make_processor_dlq shaped blocks (both copies), and
which functions get a DLQ (site_extractor has none).
5. The exact construct ids (2nd positional arg to every alarm / bucket /
dlq / statement construct) — these are the logical-ID roots that must be
passed UNCHANGED into the shared helpers.
30-40 precise facts. Any block that is NOT actually identical between
stacks (genuine drift) is a blocker to report, not to silently reconcile.`,
{ label: 'recon:duplicated-cdk', model: 'sonnet', phase: 'Recon', schema: RECON }),
() => agent(`${PREAMBLE}
Read-only recon of the two inline Bedrock IAM PolicyStatements (the
make_bedrock_invoke_statement source — constraint 2, the cross-family-review
surface). In both stacks quote verbatim with file:line: the full
iam.PolicyStatement (effect, actions, resources, conditions). For EACH
resource ARN record whether the account (328440206208) and region are
hardcoded or referenced, and enumerate every inference-profile ARN and every
per-region foundation-model ARN. Determine the EXACT list of regions / model
ids baked into the resources so the derived form (Stack.of(scope).account /
.region) can be proven to resolve to the identical strings in-account. Note
which principal/role each statement is attached to and how (add_to_role_policy
vs inline policy). 15-25 facts.`,
{ label: 'recon:bedrock-iam', model: 'sonnet', phase: 'Recon', schema: RECON }),
() => agent(`${PREAMBLE}
Read-only recon of the fallback-rate + rejected alarm math AS IT STANDS
POST-PHASE-1 (constraint 4). In both stacks quote verbatim with file:line:
1. PO's template-fallback-rate alarm: the full metric-math expression
string(s), every FILL(), every label, the period/threshold/
evaluation_periods/datapoints_to_alarm — and CONFIRM it EXCLUDES the
rejected series today (rejected_included=False).
2. WO's template-fallback-rate alarm: same, and CONFIRM it INCLUDES the
rejected series (rejected_included=True).
3. PO's ai-fallback-rejected alarm (added in Phase 1): confirm it is the
6h count-floor IF(FILL(rej,0)>=1,...) shape — quote the expression.
4. WO's rejected alarm: confirm it is the 5-min / 2-of-6 sparse idiom —
quote it. These two are DIFFERENT shapes and must stay distinct call
sites, NOT folded into make_fallback_rate_alarm.
5. Confirm NO element-wise MAX exists anywhere in either expression
(post-#102 rule).
Return the exact parameter values each make_fallback_rate_alarm call must
carry so Verify can prove byte-identity. 15-25 facts.`,
{ label: 'recon:fallback-alarms', model: 'sonnet', phase: 'Recon', schema: RECON }),
() => agent(`${PREAMBLE}
Read-only recon of the app.py / pins / outputs surface (constraint 6):
1. cdk/app.py: quote both cdk.Environment(...) calls verbatim with line
numbers (region-only today; account must be added).
2. The constructs dependency pin: quote cdk/requirements.txt line(s) and
find the stale "2.259.0" version comment(s) wherever they live (app.py,
stacks, requirements — grep the whole cdk/ tree) with file:line and the
actual installed constructs / aws-cdk-lib versions.
3. The five Lambda function construct variables in the stacks whose ARNs
need CfnOutputs (po email-processor, po web_ui, po site_extractor,
wo email-processor, wo web_ui) and the table constructs whose names are
consumed (purchase-orders, WorkOrders, and any comments table) — quote
the construct handles + ids so the CfnOutputs reference them correctly.
4. Any existing CfnOutput in either stack (WO reportedly has zero).
5. Confirm the wo artifact id is 'workorder-ingest' (the 2nd positional
arg to the Stack constructor in app.py), distinct from
stack_name='WorkorderIngestStack' (constraint 9).
15-25 facts.`,
{ label: 'recon:app-pins-outputs', model: 'haiku', phase: 'Recon', schema: RECON }),
])
const reconOk = recon.filter(Boolean)
const pack = reconOk.map(r => `## ${r.summary}\n${r.facts.join('\n')}`).join('\n\n')
const reconBlockers = reconOk.flatMap(r => r.blockers || [])
.filter(b => b && !/^\s*(none|n\/a)\b/i.test(b))
log(`Recon complete: ${reconOk.length}/4 mappers, ${reconBlockers.length} advisory notes`)
// Recon "blockers" for this phase are advisory design-notes / intended
// constraint-2 & 6 work items (derive the Bedrock ARN from Stack.of(scope),
// add account= to the environments, exact-pin constructs, fix the stale
// aws-cdk-lib version comment, decide the one common sender-auth docstring),
// NOT stop conditions — main-loop verified. The real gate is the ZERO cdk diff
// acceptance test in Verify. Fold the notes into the spec context instead of
// aborting so the spec agent must address each.
const reconAdvisories = reconBlockers.length
? `\n\nRECON ADVISORIES (recon flagged these; they are the intended constraint-2/6 work + a docstring choice, NOT drift that blocks dedup — resolve each per the constraints; the zero-cdk-diff test is the real acceptance gate):\n- ${reconBlockers.join('\n- ')}`
: ''
// -------------------------------------------------------------------- spec
phase('Spec')
const spec = await agent(`${PREAMBLE}
You are the SPEC agent — the single authority that pins every contested
decision BEFORE parallel implementation (parallel leaves cannot see each
other's choices). Using the recon pack below plus your own reads of the
actual files, produce the binding implementation spec:
- commonModule: the full cdk/common.py — every function per constraint 2
with its EXACT signature (add_standard_lambda_alarms keyword-only params
exactly as the doc pins them), and ONLY the imports the helpers need
(constraint 5 — no kms/ssm/event_sources). Every function is a PLAIN
function taking (scope, id, ...) — NO Construct subclass anywhere
(constraint 1).
- poStackEdits / woStackEdits: for each stack, the exact call-site rewrites
(file:line) mapping each inline block to a common.* call, passing the
SAME scope (the Stack) and the SAME construct id it uses today so every
logical ID is byte-stable; plus the CfnOutput additions (five function
ARNs split across the two stacks + consumed table names). State
explicitly that wo web_ui receives NO alarm call (constraint 3).
- alarmVariance: the per-function variance table (constraint 3) proving each
helper call reproduces today's emitted alarms EXACTLY — PO p99 vs wo-email
p95, po-web-ui throttles+duration only, site_extractor no-DLQ, wo web_ui
zero alarms, every bespoke description string mapped verbatim.
- bedrockStatement: make_bedrock_invoke_statement's actions + resources and
the derivation of the inference-profile / per-region FM ARNs from
Stack.of(scope).account/.region, with a proof table showing each derived
string equals the inline hardcoded ARN in-account (this is the
cross-family-review surface — be exhaustive).
- fallbackRateCalls: the make_fallback_rate_alarm call params for PO
(rejected_included=False) and WO (rejected_included=True) reproducing the
expression/FILL/label strings byte-for-byte, PLUS the plan for keeping the
two DISTINCT rejected alarms as separate call sites (PO 6h count-floor,
WO 5-min/2-of-6) — NOT folded, NO element-wise MAX (constraint 4).
- appAndPins: app.py account= on both Environment calls; the exact
constructs== pin; the stale "2.259.0" comment fix (file:line + new text);
confirmation each is logical-ID-neutral.
- diffRules: exactly how Verify proves ZERO cdk diff — synth ${BASE} and
HEAD each into a separate temp dir, diff both stacks' templates, ANY
resource/logical-ID/property delta = FAIL, the ONLY allowed net-new is
the CfnOutputs; drive synth/diff by artifact id (po-ingest /
workorder-ingest, constraint 9).
Recon pack:\n${pack}${reconAdvisories}`,
{ label: 'spec:pin-common', phase: 'Spec', schema: SPEC })
if (!spec) return { aborted: 'spec agent died — rerun workflow', reconBlockers }
const specBlock = `BINDING SPEC (from the spec agent — implement EXACTLY this):\n${JSON.stringify(spec, null, 2)}`
log('Spec pinned: common.py contents, per-stack rewrites, alarm variance, Bedrock IAM, fallback-rate calls, app/pins/outputs')
// --------------------------------------------------------------- implement
phase('Implement')
const impl = await parallel([
() => agent(`${PREAMBLE}
YOU OWN: cdk/common.py ONLY (create it). Do not touch po_stack.py,
wo_stack.py, app.py, requirements.txt, README, tests, or anything under
lambdas/.
Task: author cdk/common.py per spec.commonModule EXACTLY — every function
(add_ddb_alarms + _DDB_ALARM_OPERATIONS, add_sender_auth_rejected_alarm,
add_standard_lambda_alarms with the pinned keyword-only signature,
make_bedrock_invoke_statement deriving ARNs from Stack.of(scope).account/
.region, make_email_bucket, make_processor_dlq, make_fallback_rate_alarm)
as a PLAIN function taking (scope, id, ...) — NEVER a Construct subclass
(constraint 1). Import ONLY what the helpers need — no kms/ssm/
event_sources (constraint 5). Match the fallback-rate expression/FILL/label
strings byte-for-byte (constraint 4). Do NOT put the two rejected alarms in
make_fallback_rate_alarm.
Run before returning: ruff check cdk/common.py && ruff format cdk/common.py
--check, plus a py_compile import smoke (python3 -c "import common" from
cdk/ with the CDK venv). Full synth is the stacks agent's job — but if you
can import common cleanly, report it.
${specBlock}`,
{ label: 'impl:common-module', model: 'opus', phase: 'Implement', schema: IMPL }),
() => agent(`${PREAMBLE}
YOU OWN: cdk/po_stack.py and cdk/wo_stack.py ONLY. Do not touch
cdk/common.py (another agent authors it), app.py, requirements.txt, README,
or lambdas/.
Task: apply spec.poStackEdits + spec.woStackEdits — replace each inline
block with the matching common.* call, passing the SAME scope (the Stack
instance, NOT a new Construct) and the SAME construct id used today so every
logical ID is byte-stable (constraint 1). Preserve every per-function alarm
variance (constraint 3): PO email p99 / wo email p95, po-web-ui throttles+
duration only, site_extractor no-DLQ, wo web_ui gets NO alarm call, bespoke
descriptions passed verbatim. Wire the fallback-rate calls per
spec.fallbackRateCalls (PO rejected_included=False, WO True) and keep the
two rejected alarms as distinct call sites (constraint 4). Add the CfnOutputs
per spec (function ARNs + consumed table names) — additive, ID-safe.
Import from common (bare 'import common' / 'from common import ...' — cdk/
is on sys.path via app.py's imports). Touch nothing else (tables, KMS, SSM,
event sources, the RETAIN policies stay byte-identical).
Run before returning: ruff check cdk && cd cdk &&
npx cdk synth po-ingest -q -o /tmp/phase4-synth &&
npx cdk synth workorder-ingest -q -o /tmp/phase4-synth (artifact-id
selectors, NOT stack_name — constraint 9). If cdk/common.py has not landed
yet the synth fails on the missing import — poll by re-running up to ~10 min
before reporting a blocker. Confirm both stacks synth and note that the
ONLY template delta vs ${BASE} is the net-new CfnOutputs (spot-check a
couple of alarm logical IDs are unchanged).
${specBlock}`,
{ label: 'impl:cdk-stacks', model: 'opus', phase: 'Implement', schema: IMPL }),
() => agent(`${PREAMBLE}
YOU OWN: cdk/app.py, cdk/requirements.txt and README.md ONLY. Do not touch
common.py, the stacks, tests, or lambdas/.
Task A: apply spec.appAndPins — add account='328440206208' to BOTH
cdk.Environment calls in app.py, pin constructs== to the exact installed
version in cdk/requirements.txt, and fix the stale "2.259.0" comment(s) at
the file:line spec.appAndPins gives (only those in files you own — if a
stale comment lives in a stack file, note it for the stacks agent, do NOT
edit their file). Every edit here must be logical-ID-neutral (account on
Environment does not change resource logical IDs; verify in your summary).
Task B: README — document cdk/common.py in the CDK/architecture section
(the shared plain-function helpers, the logical-ID-safety rule, the
account/region-derived Bedrock ARN, the new CfnOutputs), and note the
account is now explicit on both stacks. Match existing README style. If the
README documents the stacks' resource inventory, keep it accurate.
Run before returning: ruff check cdk (app.py) and a py_compile of app.py;
you cannot run the full synth without the stacks agent's edits — if you want
to smoke-test, poll cd cdk && npx cdk synth po-ingest -q up to ~10 min, but
a clean app.py parse is sufficient for your scope.
${specBlock}`,
{ label: 'impl:app-pins-readme', model: 'sonnet', phase: 'Implement', schema: IMPL }),
])
const implOk = impl.filter(Boolean)
const implBlockers = implOk.flatMap(r => r.blockers || [])
log(`Implement complete: ${implOk.length}/3 agents, blockers: ${implBlockers.length}`)
// ---------------------------------------------------- verify + fix loop
const EXPECTED_SCOPE = [
'cdk/common.py',
'cdk/po_stack.py',
'cdk/wo_stack.py',
'cdk/app.py',
'cdk/requirements.txt',
'README.md',
'tests/',
]
const mechanicalPrompt = `${PREAMBLE}
Independent re-verification — trust nothing self-reported. Run ALL gates,
quoting failures verbatim:
1. pytest -q --no-cov (repo root — all suites green; a synth-only cdk-diff
test may now exist under tests/)
2. ruff check . && ruff format --check .
3. cd cdk && npx cdk synth po-ingest -q && npx cdk synth workorder-ingest -q
(artifact-id selectors, NOT stack_name — constraint 9)
4. ZERO-CODE invariant (constraint 7): git diff ${BASE}...HEAD -- lambdas/
must output NOTHING.
5. NO CONSTRUCT SUBCLASS (constraint 1): grep cdk/common.py for
'class .*Construct' / 'class .*(Construct)' — there must be NONE; every
extracted symbol is a plain 'def'. Report any subclass as a hard FAIL.
6. cdk/common.py imports NO kms/ssm/event_sources (constraint 5) — grep and
confirm.
7. Both cdk.Environment calls in app.py carry account='328440206208';
constructs== is an exact pin (not >=); no "2.259.0" stale comment
remains (grep the cdk/ tree).
8. CfnOutputs: five function ARNs + the consumed table names are present
across the two stacks (grep CfnOutput).
9. git status --porcelain scope check: every modified/added path under
${EXPECTED_SCOPE.join(', ')} (untracked .coverage/.claude/package/
tolerated). No lambdas/ or non-cdk-diff tests/ change.
passed=true only if all green. YOU MAY NOT edit files.`
const diffPrompt = `${PREAMBLE}
You are the ZERO-CDK-DIFF verifier — the load-bearing gate of this phase
(the doc names it the acceptance test). Everything is local synth-vs-synth.
1. From a clean worktree state, synth the BASE templates: check out (via
git worktree add or git stash-free 'git show'-based synth — prefer
'git worktree add /tmp/phase4-base ${BASE}' so HEAD is untouched), then
in that BASE tree cd cdk && npx cdk synth po-ingest -q -o
/tmp/phase4-diff-base && npx cdk synth workorder-ingest -q -o
/tmp/phase4-diff-base.
2. Synth the HEAD templates: in the working tree cd cdk && npx cdk synth
po-ingest -q -o /tmp/phase4-diff-head && npx cdk synth workorder-ingest
-q -o /tmp/phase4-diff-head. (Both by ARTIFACT ID, constraint 9.)
3. Diff the two CloudFormation templates per stack
(/tmp/phase4-diff-base/<stack>.template.json vs
/tmp/phase4-diff-head/<stack>.template.json). Normalize only CDK-tooling
noise (the CDKMetadata Analytics string, asset-hash-derived S3Key values
that were ALSO equal on BASE). Then judge:
- ZERO logical-ID changes (no renamed/removed/added Resources except the
net-new CfnOutputs).
- ZERO alarm property deltas (thresholds, statistics p99/p95, expression
strings, FILL, labels, evaluation_periods, datapoints_to_alarm).
- ZERO IAM deltas: the Bedrock PolicyStatement actions/resources must be
byte-identical after the account/region derivation resolves in-account.
- ZERO DynamoDB table / bucket / DLQ / env / runtime deltas.
- The ONLY allowed net-new is the Outputs block (the five function ARNs
+ consumed table names).
ANY delta outside that allowance = FAIL. Paste the actual per-stack
template diff (or 'identical' with the normalized-noise list).
4. Cross-check with the live tool: cd cdk && npx cdk diff po-ingest ;
npx cdk diff workorder-ingest against the deployed state must also show
only the CfnOutput additions (network permitting; if AWS creds are
read-only-absent, the BASE-vs-HEAD template diff in steps 1-3 is
authoritative).
5. Clean up any /tmp worktrees you created (git worktree remove).
passed=true only if BOTH stacks show zero resource change beyond the
CfnOutput additions.`
const lenses = [
{ key: 'logical-id-safety', prompt: `${PREAMBLE}
ADVERSARIAL REVIEW — logical-ID-safety lens (the load-bearing rule,
constraint 1). Try to prove a construct-id or tree-shape change slipped in.
(1) Read cdk/common.py: is EVERY extracted symbol a plain 'def' taking
(scope, id, ...)? Any 'class X(Construct)' / Construct subclass / nested
Construct is an automatic CRITICAL — a subclass reparents the tree and
would attempt REPLACEMENT of the RETAIN-protected purchase-orders /
WorkOrders tables and the named buckets. (2) For every rewritten call site
in both stacks, prove the scope passed is the Stack instance (self), NOT a
new intermediate construct, and the construct id string is IDENTICAL to the
BASE inline id (git diff ${BASE} the id strings). (3) Synth BASE and HEAD
and diff the full Resources logical-ID SET — it must be identical (only
Outputs added). Any renamed/removed logical ID = confirmed CRITICAL.
(4) Confirm the RETAIN removal policies on the tables and the bucket
names/policies are byte-identical post-refactor. confirmed=true only with a
concrete logical-ID delta or a subclass-smell file:line.` },
{ key: 'alarm-variance', prompt: `${PREAMBLE}
ADVERSARIAL REVIEW — alarm-variance lens (constraint 3). The shared helpers
must NOT homogenize the deliberate per-function differences. Read
cdk/common.py + every helper call site + the synthesized templates' alarms.
Prove each of these survived EXACTLY: (1) PO email-processor duration alarm
uses p99, wo-email-processor uses p95 — quote both from the synthesized
templates. (2) po-web-ui has ONLY throttles+duration alarms (no errors/dlq
beyond what it had). (3) site_extractor has NO DLQ alarm. (4) wo web_ui has
ZERO alarms — prove the shared helper did NOT silently add any (search the
wo template for any alarm whose dimensions point at the wo web_ui
function). (5) Every bespoke alarm description string is byte-identical to
BASE (diff the AlarmDescription fields). (6) Fallback-rate: PO excludes the
rejected series (rejected_included=False), WO includes it; the two rejected
alarms kept their distinct shapes (PO 6h count-floor, WO 5-min/2-of-6); NO
element-wise MAX anywhere. confirmed=true only with a template-level diff
showing a homogenized or dropped alarm.` },
{ key: 'iam-equivalence', prompt: `${PREAMBLE}
ADVERSARIAL REVIEW — IAM-equivalence lens (the cross-family-review surface).
make_bedrock_invoke_statement moved the PolicyStatement construction and
swapped hardcoded 328440206208 for Stack.of(scope).account/.region. Prove
the produced IAM is IDENTICAL. (1) Synth both stacks and extract the Bedrock
PolicyStatement from each template; diff actions and resources against
${BASE}'s synthesized statements — the resolved ARN strings (account +
region substituted) must be byte-identical in-account. (2) Confirm the
resources still enumerate the SAME inference-profile ARN and the SAME
per-region foundation-model ARNs (no region dropped/added, no wildcard
broadening). (3) Confirm effect/conditions/principal attachment unchanged
and the statement is attached to the SAME role. (4) Flag any broadening
(e.g. a Ref/Sub that resolves to a wildcard, or Stack.region producing a
different region than the hardcoded one) as confirmed HIGH. confirmed=true
only with the two synthesized statements diffed.` },
]
let round = 0
let checks = null
let diff = null
let confirmed = []
while (round < 3) {
phase('Verify')
const results = await parallel([
() => agent(mechanicalPrompt, { label: `verify:mechanical-r${round}`, model: 'sonnet', phase: 'Verify', schema: CHECKS }),
() => agent(diffPrompt, { label: `verify:cdk-diff-r${round}`, model: 'opus', phase: 'Verify', schema: DIFF }),
...lenses.map(l => () =>
agent(l.prompt, { label: `verify:${l.key}-r${round}`, phase: 'Verify', schema: FINDINGS })),
])
checks = results[0]
diff = results[1]
confirmed = results.slice(2).filter(Boolean)
.flatMap(r => r.findings || [])
.filter(f => f.confirmed && f.severity !== 'low')
const green = checks && checks.passed && diff && diff.passed
log(`Verify round ${round}: mechanical ${checks && checks.passed ? 'GREEN' : 'RED'}, cdk-diff ${diff && diff.passed ? 'GREEN' : 'RED'}, confirmed findings: ${confirmed.length}`)
if (green && confirmed.length === 0) break
round += 1
if (round >= 3) break
phase('Fix')
await agent(`${PREAMBLE}
You are the fix agent — you may edit files under: ${EXPECTED_SCOPE.join(', ')}.
Fix EVERY item below minimally; the binding spec and 10 pinned constraints
still hold (a finding that conflicts with a constraint is reported, not
"fixed" — the constraint wins, esp. constraint 1's plain-function /
no-Construct-subclass rule, constraint 3's alarm variance, and constraint 4's
distinct rejected alarms / no element-wise MAX). Re-run the specific failing
gate per fix (the zero-cdk-diff check is authoritative — a fix that
introduces ANY logical-ID or property delta is worse than the finding).
MECHANICAL:\n${checks ? checks.details : '(agent died — rerun all gates)'}
CDK-DIFF:\n${diff ? diff.details : '(agent died — rerun all)'}
CONFIRMED FINDINGS:\n${JSON.stringify(confirmed, null, 2)}
${specBlock}`,
{ label: `fix:round-${round}`, model: 'opus', phase: 'Fix', schema: IMPL })
}
const verifyClean = checks && checks.passed && diff && diff.passed && confirmed.length === 0
if (!verifyClean) {
return {
status: 'NEEDS ATTENTION — verify not clean after 3 rounds; branch left uncommitted',
branch: BRANCH,
mechanical: checks,
cdkDiff: diff,
unresolvedFindings: confirmed,
implBlockers,
reconBlockers,
spec,
}
}
// ----------------------------------------------------------------- package
phase('Package')
const commit = await agent(`${PREAMBLE.replace('do NOT commit, ', '')}
YOU are the commit agent:
1. MANDATORY GPT-4.1 CROSS-FAMILY REVIEW (the doc mandates it for the
make_bedrock_invoke_statement IAM PolicyStatement move, even though
semantics are identical). Capture the Bedrock-statement diff first:
git diff ${BASE}...HEAD -- cdk/common.py cdk/po_stack.py cdk/wo_stack.py
(isolate the make_bedrock_invoke_statement + its two call sites), then
RUN inline:
python3 ~/Documents/repositories/seahaven/security-review/cross_review.py
"Review this CDK IAM PolicyStatement move for breaking changes: the two
inline Bedrock invoke PolicyStatements (hardcoded account 328440206208)
were consolidated into make_bedrock_invoke_statement in cdk/common.py,
deriving the inference-profile + per-region foundation-model ARNs from
Stack.of(scope).account/.region. Confirm the produced actions/resources
are byte-identical in-account and no privilege broadening. <paste diff>"
Record the verdict verbatim in your summary. If cross_review.py is
unavailable, DO NOT block the local commit but flag the review as
OUTSTANDING in your summary (it must be run before merge).
2. Read ~/Documents/repositories/seahaven/engineering-handbook/commit-messages.md
and follow it exactly.
3. git add only paths under: ${EXPECTED_SCOPE.join(', ')} and
.claude/workflows/phase-4-cdk-common.js. NOT .coverage, NOT package/,
NOT cdk.out. Verify the staged set with git status.
4. ONE commit; write the message to /tmp/phase4-commit-msg.txt and use
git commit -F /tmp/phase4-commit-msg.txt (backticks in -m get eaten by
zsh). Suggested subject:
"feat: extract cdk/common.py — dedup 379 CDK lines as logical-ID-safe plain helpers (refactor phase 4)"
Body: the plain-function (scope, id, ...) approach and WHY no Construct
subclass (RETAIN-table replacement), the account/region-derived Bedrock
ARN, the zero-cdk-diff acceptance evidence for both stacks, the
account=/constructs pin/stale-comment/CfnOutput net-new additions, and
the cross_review.py verdict one-liner. NO AI attribution / Co-Authored-By
lines.
5. Do NOT push. Return commit sha + shortstat + the cross_review.py verdict
in summary.`,
{ label: 'package:commit', model: 'sonnet', phase: 'Package', schema: IMPL })
return {
status: 'BUILT — committed locally, NOT pushed',
branch: BRANCH,
base: BASE,
commit: commit ? commit.summary : 'commit agent died — commit manually',
spec: { commonModule: spec.commonModule, bedrockStatement: spec.bedrockStatement, fallbackRateCalls: spec.fallbackRateCalls, appAndPins: spec.appAndPins, notes: spec.notes },
diffEvidence: diff ? { po: diff.poDiffVerdict, wo: diff.woDiffVerdict } : null,
implementation: implOk.map(r => r.summary),
filesChanged: implOk.flatMap(r => r.filesChanged),
verifyRounds: round + 1,
blockers: implBlockers.concat(reconBlockers),
outstandingGates: [
'MANDATORY cross-family GPT-4.1 review (cross_review.py) on the make_bedrock_invoke_statement IAM PolicyStatement move — the Package agent runs it inline and records the verdict; confirm that verdict is clean (or re-run) before merge. If cross_review.py was unavailable at commit time, this review is OUTSTANDING — do not merge without it.',
'/sh-security-review NOT required (pure CDK refactor — no untrusted-input/auth-logic change; the pre-push scanners still run as the unattended backstop)',
'push + PR + gh pr checks green',
'deploy-then-merge with cdk diff zero-change on BOTH stacks as the live acceptance signal: deploy from branch, confirm cdk diff po-ingest / cdk diff workorder-ingest show only the CfnOutput additions, both stacks reach UPDATE_COMPLETE, all alarms still OK, THEN merge (a no-op resource redeploy is itself the verification signal). Drive synth/diff by artifact id workorder-ingest, NOT stack_name WorkorderIngestStack (constraint 9).',
'update the Confluence "AWS Architecture Map" if the new CfnOutputs / account-explicit envs change the documented resource inventory',
],
}

View file

@ -0,0 +1,661 @@
export const meta = {
name: 'phase-7-ops-recovery',
description: 'Phase 7 of the procurement-ingest refactor (docs/refactor-evaluation.md): ops/recovery + dependency hygiene. Generalize scripts/reprocess.py (--pipeline po|wo, --key/--prefix/--since; TARGETED replay primary, full-prefix DEMOTED behind --all with documented caveats) + a synthetic-event-shape contract test pinning the raw-key/no-URL-decode S3 event. New docs/runbook-dlq-recovery.md for the no-redrive async-destination DLQ (receive->key->re-invoke->verify->purge; 14d DLQ / 90d S3 windows; sender-auth + ai_fallback_rejected drops never reach the DLQ), linked from README alarms. Dependency hygiene: drop vendored boto3 from both email-processor requirements (handbook empty-with-comment form; runtime copy suffices per lambda-template.md), simplify bundling to cp-only where the vendored dep is gone, exact-pin moto==, pin-then-add dependabot entries for /tests + po/web_ui + po/site_extractor, reduce wo/web_ui dead manifest to empty-with-comment. Optional local-only rm of the untracked 44 MB package/ dir. Gates on Phase 2 merged (../lambdas asset root). Ops tooling + dep hygiene, no auth/untrusted-input/IAM change. Committed locally, never pushed.',
phases: [
{ title: 'Setup', detail: 'verify Phase 2 on base (Code.from_asset("../lambdas")), branch feature/phase-7-ops-recovery', model: 'haiku' },
{ title: 'Recon', detail: '4 mappers: reprocess.py + event shape, dependency/manifest/dependabot state, post-Phase-2 bundling + package/ dir, DLQ/alarm/S3-lifecycle names (read-only AWS)' },
{ title: 'Spec', detail: 'serial fable spec: reprocess CLI + caveats, runbook contents, requirements/moto/dependabot edits, cp-only bundling form, package/ cleanup, contract test, verify rules' },
{ title: 'Implement', detail: 'opus: scripts/ + reprocess contract test; sonnet: requirements + dependabot + cdk bundling; opus: runbook + README — disjoint files', model: 'opus' },
{ title: 'Verify', detail: 'mechanical gates + 3 fable lenses (bundling-integrity, reprocess-safety, dependency)', model: 'sonnet' },
{ title: 'Fix', detail: 'opus fixer, full re-verify, max 3 rounds', model: 'opus' },
{ title: 'Package', detail: 'single commit via -F (no push); optional local rm of package/ reported, not committed', model: 'sonnet' },
],
}
// ---------------------------------------------------------------- constants
const REPO = '/Users/adammoussa/Documents/repositories/seahaven/procurement-ingest'
const BRANCH = 'feature/phase-7-ops-recovery'
let _args = args
if (typeof _args === 'string') {
try { _args = JSON.parse(_args) } catch (e) { _args = null }
}
const BASE = (_args && _args.base) || 'main'
const CONSTRAINTS = `
PINNED BEHAVIORAL CONSTRAINTS (docs/refactor-evaluation.md Phase 7 — violating any is a build failure):
1. GENERALIZE scripts/reprocess.py: add --pipeline po|wo (selects the
function name + bucket per pipeline — po-email-processor /
po-ingest-emails-<acct>, workorder-email-processor / its bucket), plus
--key (single object), --prefix, and --since (time filter on
LastModified). TARGETED replay (--key / --prefix / --since) is the
PRIMARY, default mode. FULL-PREFIX replay is DEMOTED behind an explicit
--all flag. The --all caveats MUST be documented BOTH in --help text and
in code comments: Event-type (async) invocation is CONCURRENT so sorting
does NOT serialize; switch to RequestResponse if order matters; metrics
get double-counted on replay; Bedrock is re-billed; out-of-order replay
can REGRESS already-merged fields. Keep the existing dry-run-by-default /
--execute safety (targeted replay stays dry-run unless --execute).
2. SYNTHETIC-EVENT-SHAPE CONTRACT TEST (§4.7): pin the exact S3 event shape
reprocess emits — Records[0].s3.bucket.name + Records[0].s3.object.key —
and assert the key is the RAW object key with NO URL-decoding (a real S3
notification URL-encodes the key; reprocess builds from the raw
list_objects_v2 key, and the handler is what decodes — replaying a
pre-decoded key would double-decode). Test lives under scripts' tests or
tests/ and runs in the standard pytest collection.
3. NEW docs/runbook-dlq-recovery.md: the async on-failure destination DLQ
has NO console redrive-to-source. Documented procedure: receive-message
-> extract the S3 key from the event body -> targeted re-invoke (via the
generalized reprocess.py --key) -> verify the write -> purge the message.
State the recovery WINDOWS explicitly: 14-day DLQ breadcrumb retention,
90-day raw-email S3 retention (the inbound/ lifecycle rule OVERRIDES the
table RETAIN policy — S3 is the real replay floor). Document that
sender-auth rejections AND ai_fallback_rejected drops INTENTIONALLY never
reach the DLQ (they are fail-closed SKIPS, not errors — no retry, no DLQ
message). Link the runbook from the README alarms section.
4. DEPENDENCY HYGIENE — DROP vendored boto3 from BOTH email-processor
requirements.txt: replace 'boto3>=1.43.47' with the handbook
empty-with-comment form (a comment explaining the Lambda runtime provides
boto3 per lambda-template.md — only-boto3 functions correctly use the
runtime copy; no pinned third-party dep remains). Same empty-with-comment
reduction for wo/web_ui/requirements.txt (a dead manifest today).
5. SIMPLIFY BUNDLING TO CP-ONLY where the vendored dep is gone: once the
email-processor requirements are empty-with-comment, 'pip install -r ... '
installs nothing, so the pip step is removable and the command becomes
cp-only. This is ONLY safe BECAUSE there are no third-party binary deps
left. Do NOT regress: the Phase 2 exclude lists
(['**/__pycache__/**','**/tests/**','**/package/**']) STAY; the widened
'../lambdas' asset root STAYS; if a Phase 3 'cp shared/*.py' line is
present on the base it STAYS. The '--platform manylinux2014_aarch64
--only-binary=:all:' pin is only meaningful while a pip install runs — its
accidental removal caused the PR #34 outage, so removing the WHOLE pip
step (deliberately, because nothing is installed) is the only acceptable
way it disappears; you may NOT keep a pip install while dropping the pin.
If recon finds ANY third-party dep still required by an email processor,
cp-only is a blocker — keep the pinned pip step.
6. EXACT-PIN moto: change tests/requirements.txt 'moto>=5.0.0' to
'moto==<resolved-version>' (resolve the actually-installed/current 5.x
version; floor pins make the dependabot entry a no-op).
7. DEPENDABOT: add pip entries for '/tests', '/lambdas/po/web_ui', and
'/lambdas/po/site_extractor' — but PIN THOSE MANIFESTS FIRST. po/web_ui
and po/site_extractor have NO requirements.txt today, so a dependabot
entry pointed at them is a no-op until a PINNED manifest exists; create
each manifest (exact-pinned boto3, or empty-with-comment ONLY if the
dependabot entry would then be pointless — a dependabot entry needs at
least one pinned dep to act on). Match the existing dependabot.yml entry
shape (weekly, minor-and-patch group). tests/ gets its entry once moto is
exact-pinned (constraint 6).
8. DELETE the untracked 44 MB lambdas/po/email_processor/package/ dir as an
OPTIONAL local-cleanup step: it is UNTRACKED / local-only, so this is a
filesystem 'rm -rf', NOT a 'git rm' (there is nothing tracked to commit).
With Phase 2's '**/package/**' exclude already deployed, the deletion is
asset-hash-NEUTRAL. The Package agent PERFORMS and REPORTS it; it is NOT
part of the commit.
9. UNTOUCHABLE: no handler / lambda runtime code change under lambdas/*/
email_processor/*.py (this is dep + ops + bundling only). No alarm, IAM,
table, env, runtime, memory, timeout, or logical-ID delta on either stack
— the ONLY cdk delta permitted is the bundling command string on the two
email processors (+ any resulting asset-hash / CDK metadata change). The
dropped-vendored-boto3 functions MUST still 'import boto3' at runtime from
the Lambda runtime copy.
10. AWS access is READ-ONLY (get-function, get-function-event-invoke-config,
SQS get-queue-attributes, S3 get-bucket-lifecycle-configuration,
describe-alarms). NEVER cdk deploy, never invoke, never mutate, never
purge a real queue.
`
const PREAMBLE = `
You are one of several agents building refactor Phase 7 in the git repo at
${REPO} on branch ${BRANCH} (already checked out — do NOT switch branches,
do NOT create branches, do NOT commit, NEVER push, do NOT run cdk deploy or
touch AWS resources beyond read-only calls).
Authoritative spec: docs/refactor-evaluation.md, section "Phase 7 — Ops/recovery + dependency hygiene".
Work ONLY in the files you are told you own; other agents are concurrently
editing other files in this same working tree.
${CONSTRAINTS}
Your final message is consumed by an orchestrator script, not a human —
return only the structured data requested.
`
// ------------------------------------------------------------------ schemas
const RECON = {
type: 'object',
required: ['summary', 'facts'],
properties: {
summary: { type: 'string' },
facts: { type: 'array', items: { type: 'string' } },
blockers: { type: 'array', items: { type: 'string' } },
},
}
const SPEC = {
type: 'object',
required: ['reprocessCli', 'contractTest', 'runbook', 'dependencyEdits', 'bundlingEdits', 'packageCleanup', 'verifyRules', 'notes'],
properties: {
reprocessCli: { type: 'string', description: 'the complete generalized scripts/reprocess.py design: argparse surface (--pipeline po|wo, --key/--prefix/--since, --all, --execute), per-pipeline function-name/bucket resolution, which modes are dry-run-by-default, and the EXACT --help + code-comment caveat text for --all (concurrency/no-serialize, RequestResponse-if-order-matters, double-count, Bedrock re-bill, out-of-order regression)' },
contractTest: { type: 'string', description: 'the synthetic-event-shape contract test: file path, what it asserts about Records[0].s3.bucket.name / object.key, and the exact raw-key / no-URL-decoding assertion (encoded vs raw key example)' },
runbook: { type: 'string', description: 'the docs/runbook-dlq-recovery.md outline: the no-redrive procedure steps (receive->key->targeted re-invoke via reprocess --key->verify->purge), the 14d-DLQ / 90d-S3 windows with the lifecycle-overrides-RETAIN note, the intentional sender-auth + ai_fallback_rejected non-DLQ statement, and the exact README alarms-section link edit' },
dependencyEdits: { type: 'string', description: 'per requirements.txt: exact new contents (both email-processor empty-with-comment, wo/web_ui empty-with-comment, po/web_ui + po/site_extractor new pinned manifests, tests/ moto==<version>); the resolved moto version; the exact dependabot.yml additions matching the existing entry shape' },
bundlingEdits: { type: 'string', description: 'both email-processor bundling command strings BEFORE and AFTER, verbatim: the cp-only result, proof that the pip step is safely removable (no third-party dep remains), and confirmation the exclude lists + widened root + any Phase-3 shared cp line are preserved' },
packageCleanup: { type: 'string', description: 'the package/ dir cleanup: the exact rm command, why it is a filesystem rm not git rm (untracked), and why it is asset-hash-neutral given the Phase 2 exclude — this is a REPORTED local step, not a commit' },
verifyRules: { type: 'string', description: 'how Verify judges success: which pytest tests must pass (incl. the new contract test), ruff scope now including scripts/, both cdk synth expectations, the dependabot yaml-parse check, requirements resolve/import check, and the runtime import-boto3 proof for the cp-only assets' },
notes: { type: 'string' },
},
}
const IMPL = {
type: 'object',
required: ['filesChanged', 'summary', 'checksRun'],
properties: {
filesChanged: { type: 'array', items: { type: 'string' } },
summary: { type: 'string' },
checksRun: { type: 'string' },
blockers: { type: 'array', items: { type: 'string' } },
},
}
const CHECKS = {
type: 'object',
required: ['passed', 'details'],
properties: {
passed: { type: 'boolean' },
details: { type: 'string' },
scopeViolations: { type: 'array', items: { type: 'string' } },
},
}
const FINDINGS = {
type: 'object',
required: ['findings'],
properties: {
findings: {
type: 'array',
items: {
type: 'object',
required: ['title', 'severity', 'confirmed', 'evidence', 'fix'],
properties: {
title: { type: 'string' },
severity: { enum: ['critical', 'high', 'medium', 'low'] },
confirmed: { type: 'boolean' },
evidence: { type: 'string' },
fix: { type: 'string' },
},
},
},
},
}
// ------------------------------------------------------------------- setup
phase('Setup')
const setup = await agent(`
In ${REPO}:
1. SEQUENCING GATE — Phase 2 must be on ${BASE}. Phase 7 "can run parallel
to 4-6" (largely independent), BUT the cp-only bundling simplification and
the package/ cleanup touch the same cdk bundling that Phases 2/3
established, so this phase must NOT race Phase 2. git fetch origin, then
pick the base ref: origin/${BASE} if that remote ref exists, otherwise the
local branch ${BASE} (a stacked local-only base is expected and fine).
Verify on the base ref that BOTH cdk/po_stack.py and cdk/wo_stack.py
contain Code.from_asset("../lambdas") for the email processors
(git show <baseref>:cdk/po_stack.py | grep -n '\\.\\./lambdas', same for
wo_stack.py). If either is missing, STOP with a blocker naming Phase 2 as
unmet and do nothing else.
NOTE (report, do not block): Phase 7 is otherwise independent of Phases
4-6. If Phase 4 (cdk/common.py dedup) lands on ${BASE} FIRST, the
stack-touching bundling edits here must be rebased onto the moved bundling
code — flag that in your facts so the implement agents know whether the
bundling lives inline or in a common helper.
2. Verify clean working tree (untracked .coverage / .claude/ / the local
44 MB lambdas/po/email_processor/package/ dir are fine; any OTHER dirt =
blocker, never stash or discard).
3. git checkout ${BASE}; then git pull --ff-only ONLY if the branch has an
upstream (a local-only base skips the pull — not a blocker); then
git checkout -b ${BRANCH}
4. gh pr list --state open --json number,title,headRefName (overlap check —
especially any in-flight Phase 2/3/4 stack PR).
Return facts: HEAD sha, Phase 2 gate evidence, whether Phase 4 has landed
(bundling inline vs helper), open PRs, blockers.
`, { label: 'setup:branch', model: 'haiku', schema: RECON })
if (!setup || (setup.blockers && setup.blockers.length)) {
return { aborted: 'setup blockers', blockers: setup ? setup.blockers : ['setup agent died'], facts: setup ? setup.facts : [] }
}
log(`Branch ${BRANCH} ready off ${BASE}. ${setup.summary}`)
// ------------------------------------------------------------------- recon
phase('Recon')
const recon = await parallel([
() => agent(`${PREAMBLE}
Read-only recon of scripts/reprocess.py and the event shape it must pin:
1. Quote scripts/reprocess.py verbatim with file:line — the argparse surface
today (--execute only), FUNCTION_NAME, get_bucket_name (po-only,
po-ingest-emails-<acct>), list_inbound_keys (paginator, Prefix='inbound/'),
build_s3_event (the exact Records/s3/bucket.name/object.key shape), the
Event (async) InvocationType.
2. Find the WO analogues so --pipeline wo can resolve: the workorder
email-processor function name and its email bucket naming (grep the cdk
stacks / function_name / bucket_name). State them exactly.
3. Determine how the email-processor handler reads the key from an S3 event:
does it URL-decode Records[].s3.object.key (urllib.parse.unquote_plus)?
Quote the handler line. This decides the contract test's no-double-decode
assertion (reprocess must emit the RAW key so the handler decode path is
exercised exactly once — a pre-decoded key double-decodes '+'/'%20').
4. Is there ANY existing test that touches scripts/reprocess.py? (grep tests/
and scripts/). Where should the new synthetic-event contract test live so
it is collected by the standard pytest run (repo-root pytest collects
tests/ and both email_processor tests/ roots)? Note whether scripts/ is on
sys.path for import.
15-25 precise facts.`,
{ label: 'recon:reprocess', model: 'sonnet', phase: 'Recon', schema: RECON }),
() => agent(`${PREAMBLE}
Read-only recon of the dependency / manifest / dependabot state:
1. Quote every requirements.txt under lambdas/ and tests/ verbatim with path:
lambdas/po/email_processor (boto3>=1.43.47), lambdas/wo/email_processor
(boto3>=1.43.47), lambdas/wo/web_ui (boto3>=1.43.47 — dead manifest),
tests/ (moto>=5.0.0), cdk/. Confirm po/web_ui and po/site_extractor have
NO requirements.txt (they must be CREATED + pinned before a dependabot
entry is not a no-op).
2. Resolve the moto version to exact-pin: what 5.x is actually installed /
currently resolves (pip show moto or the lockfile / venv)? State the exact
version string to pin.
3. Quote .github/dependabot.yml verbatim: the existing pip entries (/cdk,
/lambdas/po/email_processor, /lambdas/wo/email_processor, /lambdas/wo/web_ui)
and the github-actions entry — the exact shape (weekly interval,
minor-and-patch group) the three NEW entries must match.
4. Confirm via lambda-template.md (handbook) the exact empty-with-comment
form for only-boto3 functions (the Lambda runtime provides boto3) — quote
the canonical comment wording so the edits match the handbook.
5. Does anything under lambdas/*/email_processor actually import a THIRD-PARTY
package other than boto3 (grep imports; anthropic/requests/etc.)? If yes,
cp-only is unsafe — that is a blocker for constraint 5.
15-25 facts.`,
{ label: 'recon:dependencies', model: 'sonnet', phase: 'Recon', schema: RECON }),
() => agent(`${PREAMBLE}
Read-only recon of the post-Phase-2 CDK bundling + the package/ dir:
1. Quote BOTH email-processor Code.from_asset blocks verbatim with current
file:line (cdk/po_stack.py ~241, cdk/wo_stack.py ~240): asset root, the
FULL exclude list, the entire bundling command list — the pip install
'--platform manylinux2014_aarch64 --only-binary=:all: -r
<pipeline>/email_processor/requirements.txt -t /asset-output' step, the
'&&', the 'cp <pipeline>/email_processor/*.py /asset-output/' step, and
whether a Phase-3 'cp shared/*.py /asset-output/' line is already present
on this branch's base. Note the explanatory NOTE comment in po_stack.
2. State the exact cp-only rewrite for each (drop the pip step entirely,
keep every cp line + the widened root + the exclude list). Confirm the
pip step is the ONLY thing removed.
3. The three plain from_asset calls (po/web_ui, po/site_extractor,
wo/web_ui) — quote them; they are NOT bundling-simplified here (no pip
step), just confirm they are untouched by this phase.
4. lambdas/po/email_processor/package/ — du -sh it, confirm it is UNTRACKED
(git status / git ls-files must not list it), and confirm the Phase 2
'**/package/**' exclude already keeps it out of the asset hash (so its
removal is asset-hash-neutral).
5. If Phase 4 has landed (bundling moved into a cdk/common helper) note the
new location.
12-20 facts.`,
{ label: 'recon:cdk-bundling', model: 'haiku', phase: 'Recon', schema: RECON }),
() => agent(`${PREAMBLE}
Read-only AWS + docs recon for the runbook (region us-east-1, READ-ONLY):
1. Both email-processor DLQs: exact SQS queue names/ARNs and the
MessageRetentionPeriod (confirm the ~14-day breadcrumb window). Find them
via the stacks (dead_letter_queue / make_processor_dlq) and
aws sqs get-queue-attributes. Also the async event-invoke config /
on-failure destination if present (get-function-event-invoke-config) —
confirm it is an async on-failure destination DLQ with NO console
redrive-to-source.
2. Both raw-email S3 buckets: exact names and the inbound/ prefix lifecycle
rule (get-bucket-lifecycle-configuration) — confirm the ~90-day expiry
that overrides the table RETAIN policy (S3 is the replay floor).
3. The relevant CloudWatch alarm names the runbook references: the
sender-auth-rejected alarm(s) and the ai_fallback_rejected alarm(s) per
pipeline (describe-alarms / grep the stacks) — so the runbook states that
those two drop classes are fail-closed SKIPS that never reach the DLQ.
4. Quote the README alarms section (its heading + surrounding lines) so the
Implement agent can add the runbook link in the right place, matching
style.
Return names/ARNs/windows as facts; flag any name you could not resolve so
the runbook can be validated against real infra later.`,
{ label: 'recon:dlq-alarms', model: 'sonnet', phase: 'Recon', schema: RECON }),
])
const reconOk = recon.filter(Boolean)
const pack = reconOk.map(r => `## ${r.summary}\n${r.facts.join('\n')}`).join('\n\n')
const reconBlockers = reconOk.flatMap(r => r.blockers || [])
.filter(b => b && !/^\s*(none|n\/a)\b/i.test(b))
log(`Recon complete: ${reconOk.length}/4 mappers, ${reconBlockers.length} blockers`)
if (reconBlockers.length) {
return { aborted: 'recon blockers (likely a surviving third-party dep blocking cp-only, or an unresolvable pipeline name)', blockers: reconBlockers, reconPack: pack }
}
// -------------------------------------------------------------------- spec
phase('Spec')
const spec = await agent(`${PREAMBLE}
You are the SPEC agent — the single authority that pins every contested
decision BEFORE parallel implementation (parallel leaves cannot see each
other's choices). Using the recon pack below plus your own reads of the
actual files, produce the binding implementation spec:
- reprocessCli: the complete generalized scripts/reprocess.py — argparse
surface (--pipeline po|wo required; --key / --prefix / --since targeted
modes; --all for full-prefix; --execute preserving dry-run-by-default),
the per-pipeline function-name + bucket resolution table, and the EXACT
--help + in-code caveat text for --all (all five caveats from constraint 1
verbatim). TARGETED is the default path; --all is the ONLY way to sweep the
whole prefix.
- contractTest: the synthetic-event-shape test — file path (collected by the
standard pytest run), the assertions on Records[0].s3.bucket.name and
object.key, and the raw-key / no-URL-decoding assertion with a concrete
encoded-vs-raw example (e.g. a key with a space or '+').
- runbook: the full docs/runbook-dlq-recovery.md outline with the real
DLQ/queue/alarm/bucket names from recon, the no-redrive procedure steps
(receive -> extract S3 key from event body -> targeted re-invoke via
reprocess.py --key -> verify -> purge), the 14d-DLQ / 90d-S3 windows +
lifecycle-overrides-RETAIN note, the intentional sender-auth +
ai_fallback_rejected non-DLQ statement, and the exact README alarms-section
link edit.
- dependencyEdits: exact new contents of every manifest — both
email-processor requirements.txt (handbook empty-with-comment), wo/web_ui
(empty-with-comment), po/web_ui + po/site_extractor (NEW; decide pinned dep
vs whether the dependabot entry is worthwhile — a dependabot entry needs at
least one pinned dep to act on, so pin boto3 exactly if you want the entry
live), tests/ (moto==<resolved version> from recon) — plus the exact
dependabot.yml additions (three entries) matching the existing shape.
- bundlingEdits: both email-processor bundling command strings BEFORE and
AFTER, verbatim — the cp-only result, an explicit proof line that no
third-party dep remains (so removing the whole pip step is safe, NOT a
regression of the manylinux pin), and confirmation the exclude lists +
widened '../lambdas' root + any Phase-3 'cp shared/*.py' line survive
untouched. If Phase 4 moved bundling into a helper, target that location.
- packageCleanup: the exact rm -rf command for
lambdas/po/email_processor/package/, the note that it is a filesystem rm
(UNTRACKED — never git rm) performed and REPORTED by the Package agent, not
committed, and why it is asset-hash-neutral (Phase 2 exclude).
- verifyRules: exactly how Verify judges success — the pytest set incl. the
new contract test, ruff now INCLUDING scripts/, both cdk synth,
dependabot.yml yaml-parse, requirements resolve/import, and the runtime
'import boto3' proof for a cp-only staged asset.
Recon pack:\n${pack}`,
{ label: 'spec:pin-ops-recovery', phase: 'Spec', schema: SPEC })
if (!spec) return { aborted: 'spec agent died — rerun workflow', reconBlockers }
const specBlock = `BINDING SPEC (from the spec agent — implement EXACTLY this):\n${JSON.stringify(spec, null, 2)}`
log('Spec pinned: reprocess CLI + caveats, contract test, runbook, dependency edits, cp-only bundling, package cleanup')
// --------------------------------------------------------------- implement
phase('Implement')
const impl = await parallel([
() => agent(`${PREAMBLE}
YOU OWN: scripts/reprocess.py and the new synthetic-event contract test file
ONLY (per spec.contractTest's path — do not touch any other test). Do not
touch cdk/, requirements, dependabot, docs, or README.
Task: rewrite scripts/reprocess.py per spec.reprocessCli — add --pipeline
po|wo, --key/--prefix/--since (TARGETED, the default), --all (DEMOTED
full-prefix, with all five caveats in BOTH --help and code comments),
preserving dry-run-by-default / --execute. Keep build_s3_event emitting the
RAW key (no URL-decoding). Then add the contract test per spec.contractTest
pinning the event shape + the raw-key / no-double-decode assertion.
Run before returning: ruff check scripts && ruff format scripts --check,
python3 scripts/reprocess.py --help (must render the caveats), and
pytest -q --no-cov -k <the contract test> (tests land in parallel — if
collection of other roots is red from another agent's in-flight edit, run
your test file directly; poll up to ~10 min before reporting a blocker).
${specBlock}`,
{ label: 'impl:reprocess', model: 'opus', phase: 'Implement', schema: IMPL }),
() => agent(`${PREAMBLE}
YOU OWN: every requirements.txt (lambdas/po/email_processor,
lambdas/wo/email_processor, lambdas/wo/web_ui, tests/, and the two NEW
manifests lambdas/po/web_ui/requirements.txt +
lambdas/po/site_extractor/requirements.txt), .github/dependabot.yml, and the
bundling regions of cdk/po_stack.py + cdk/wo_stack.py ONLY (only the
email-processor from_asset bundling command — alarms, IAM, tables, env,
memory, timeout, the three plain from_asset calls are all off-limits).
Task: apply spec.dependencyEdits (drop vendored boto3 -> empty-with-comment
in both email-processor + wo/web_ui; create the two pinned po manifests;
exact-pin moto==) + the three dependabot entries + spec.bundlingEdits
(cp-only: remove ONLY the pip step from both email-processor commands,
preserving the exclude lists, widened root, and any Phase-3 shared cp line).
Do NOT delete package/ (that is the Package agent's reported step).
Run before returning: ruff check cdk, python3 -c 'import yaml,pathlib;
yaml.safe_load(pathlib.Path(".github/dependabot.yml").read_text())' (parses),
cd cdk && npx cdk synth po-ingest -q -o /tmp/phase7-synth &&
npx cdk synth workorder-ingest -q -o /tmp/phase7-synth (artifact-id
selectors). Docker bundling runs — confirm each staged email-processor asset
still contains handler.py + its siblings (and shared/*.py if Phase 3 landed)
and that 'import boto3' resolves at runtime from the Lambda runtime copy (the
staged asset need not vendor it). Note asset hashes. If the other agents'
edits have not landed the synth is unaffected (bundling is yours) — but poll
~10 min if Docker is slow before a blocker.
${specBlock}`,
{ label: 'impl:deps-bundling', model: 'sonnet', phase: 'Implement', schema: IMPL }),
() => agent(`${PREAMBLE}
YOU OWN: docs/runbook-dlq-recovery.md (NEW) and README.md ONLY.
Task A: write docs/runbook-dlq-recovery.md per spec.runbook — the no-redrive
async-destination DLQ procedure (receive-message -> extract the S3 key from
the event body -> targeted re-invoke via 'python scripts/reprocess.py
--pipeline <po|wo> --key <key> --execute' -> verify the write -> purge the
message), the real queue/bucket/alarm names from the spec, the recovery
windows (14-day DLQ breadcrumb, 90-day raw-email S3 with the
lifecycle-overrides-RETAIN note), and the explicit statement that sender-auth
rejections AND ai_fallback_rejected drops are fail-closed SKIPS that never
reach the DLQ (no retry, no DLQ message).
Task B: README — link the new runbook from the alarms section per
spec.runbook's exact edit, and note reprocess.py is now pipeline-general
(--pipeline / --key / --prefix / --since targeted; --all demoted). Match
existing README style.
Run before returning: confirm the runbook links resolve and the README edit
is in the alarms section. (No pytest owned by you; if you want a doc-lint,
ruff does not lint .md.)
${specBlock}`,
{ label: 'impl:runbook-readme', model: 'opus', phase: 'Implement', schema: IMPL }),
])
const implOk = impl.filter(Boolean)
const implBlockers = implOk.flatMap(r => r.blockers || [])
log(`Implement complete: ${implOk.length}/3 agents, blockers: ${implBlockers.length}`)
// ---------------------------------------------------- verify + fix loop
const EXPECTED_SCOPE = [
'scripts/reprocess.py',
'tests/',
'docs/runbook-dlq-recovery.md',
'lambdas/po/email_processor/requirements.txt',
'lambdas/wo/email_processor/requirements.txt',
'lambdas/wo/web_ui/requirements.txt',
'lambdas/po/web_ui/requirements.txt',
'lambdas/po/site_extractor/requirements.txt',
'.github/dependabot.yml',
'cdk/po_stack.py',
'cdk/wo_stack.py',
'README.md',
]
const mechanicalPrompt = `${PREAMBLE}
Independent re-verification — trust nothing self-reported. Run ALL gates,
quoting failures verbatim:
1. pytest -q --no-cov (repo root, all roots green) — INCLUDING the new
synthetic-event contract test (name it and quote its pass line).
2. ruff check . && ruff format --check . — scripts/ is now IN scope (it was
previously omitted); confirm scripts/reprocess.py lints and formats clean.
3. cd cdk && npx cdk synth po-ingest -q && npx cdk synth workorder-ingest -q
(Docker bundling runs; both must succeed).
4. cd cdk && npx cdk diff po-ingest ; npx cdk diff workorder-ingest —
constraint 9: the ONLY resource delta allowed is the two email
processors' Code/S3Key (asset) + CDK metadata. Any alarm/IAM/env/runtime/
memory/timeout/handler-prop/logical-ID delta = FAIL. Paste the diff
summaries.
5. DEPENDABOT: python3 -c 'import yaml,pathlib; d=yaml.safe_load(pathlib.Path(".github/dependabot.yml").read_text()); print([u["directory"] for u in d["updates"]])'
— the three new dirs (/tests, /lambdas/po/web_ui, /lambdas/po/site_extractor)
present, shape matches existing entries.
6. MANIFESTS: every email-processor + wo/web_ui requirements.txt is the
empty-with-comment form (no pinned third-party dep); tests/requirements.txt
is moto==<exact>; po/web_ui + po/site_extractor manifests EXIST and are
PINNED (a dependabot entry pointed at a floor/absent manifest is a no-op —
FAIL if unpinned). pip install --dry-run (or pip-compile check) resolves
each without error.
7. RUNTIME IMPORT: no lambda handler .py under lambdas/*/email_processor
changed (git diff ${BASE}...HEAD -- must be empty for those); 'import boto3'
still resolves — prove the Lambda runtime copy suffices (the cp-only staged
asset need not vendor boto3).
8. RUNBOOK: docs/runbook-dlq-recovery.md exists, states the 14d/90d windows,
the no-redrive procedure, and the sender-auth + ai_fallback_rejected
non-DLQ note; README alarms section links it.
9. git status --porcelain scope check: every modified/added path under
${EXPECTED_SCOPE.join(', ')} (untracked .coverage/.claude/package/
tolerated; package/ must NOT be staged).
passed=true only if all green. YOU MAY NOT edit files.`
const lenses = [
{ key: 'bundling-integrity', prompt: `${PREAMBLE}
ADVERSARIAL REVIEW — bundling-integrity lens. The bundling was simplified to
cp-only and vendored boto3 was dropped; try to prove a runtime ImportError
was introduced (the PR #34 / PR #105 failure class). (1) cd cdk && synth both
stacks; inside each staged email-processor asset dir list the files and run
python3 -c "import handler" with that dir alone on sys.path (env-var stubs as
needed) — does every runtime-imported module still ship (handler siblings +
shared/*.py if Phase 3 landed)? (2) Does 'import boto3' succeed against the
Lambda runtime copy — i.e. is boto3 genuinely NOT needed in the zip, or does
some code path need a newer boto3 than the runtime ships? (3) Are the Phase
2/3 excludes (['**/__pycache__/**','**/tests/**','**/package/**']) STILL
present and is the widened '../lambdas' root intact? (4) Was the manylinux
pin removed ONLY as part of removing the whole pip step (because nothing is
installed), NOT while keeping a pip install? A cp-only command that still
tries to pip-install without the pin, or a surviving third-party dep, is a
critical finding. (5) DETERMINISM: synth po-ingest twice into fresh -o dirs —
identical asset hashes. confirmed=true only with a concrete ImportError
sketch or file:line proof.` },
{ key: 'reprocess-safety', prompt: `${PREAMBLE}
ADVERSARIAL REVIEW — reprocess-safety lens. Attack the generalized
reprocess.py. (1) Is the DEFAULT behavior TARGETED (—key/—prefix/—since), and
is it IMPOSSIBLE to sweep the whole inbound/ prefix without the explicit
--all flag? Trace the arg parsing — can a missing/empty --prefix silently
fall through to a full-prefix replay? That would be a critical finding
(accidental mass re-invoke = mass Bedrock re-bill + merged-field regression).
(2) Are ALL FIVE --all caveats present in BOTH --help and code comments
(concurrency/no-serialize, RequestResponse-if-order-matters, double-count,
Bedrock re-bill, out-of-order regression)? (3) Does build_s3_event emit the
RAW key with NO URL-decoding, and does the contract test actually FAIL if
someone adds an unquote_plus (mutate it on a scratch copy and confirm red)?
A key with a space/'+' must round-trip raw so the handler decodes exactly
once. (4) --pipeline wo resolves the correct function name + bucket (not the
PO defaults)? confirmed=true only with file:line or reproduced-failure
evidence.` },
{ key: 'dependency', prompt: `${PREAMBLE}
ADVERSARIAL REVIEW — dependency lens. (1) moto: is tests/requirements.txt
EXACT-pinned (moto==<version>), not a floor (>=)? A floor makes the tests/
dependabot entry a no-op. (2) The three NEW dependabot dirs: do
/lambdas/po/web_ui and /lambdas/po/site_extractor now have PINNED manifests
that actually exist? A dependabot entry pointed at an absent or floor-pinned
manifest does nothing — that is a finding. Cross-check dependabot.yml
directory strings against the real file paths (a typo'd directory silently
no-ops). (3) The email-processor + wo/web_ui requirements are the handbook
empty-with-comment form — confirm no pinned third-party dep is left that
would then require the pip step back (which would contradict the cp-only
bundling). (4) Does the empty-with-comment wording match lambda-template.md
(runtime provides boto3)? confirmed=true only with file:line evidence.` },
]
let round = 0
let checks = null
let confirmed = []
while (round < 3) {
phase('Verify')
const results = await parallel([
() => agent(mechanicalPrompt, { label: `verify:mechanical-r${round}`, model: 'sonnet', phase: 'Verify', schema: CHECKS }),
...lenses.map(l => () =>
agent(l.prompt, { label: `verify:${l.key}-r${round}`, phase: 'Verify', schema: FINDINGS })),
])
checks = results[0]
confirmed = results.slice(1).filter(Boolean)
.flatMap(r => r.findings || [])
.filter(f => f.confirmed && f.severity !== 'low')
const green = checks && checks.passed
log(`Verify round ${round}: mechanical ${checks && checks.passed ? 'GREEN' : 'RED'}, confirmed findings: ${confirmed.length}`)
if (green && confirmed.length === 0) break
round += 1
if (round >= 3) break
phase('Fix')
await agent(`${PREAMBLE}
You are the fix agent — you may edit files under: ${EXPECTED_SCOPE.join(', ')}.
Fix EVERY item below minimally; the binding spec and 10 pinned constraints
still hold (a finding that conflicts with a constraint is reported, not
"fixed" — the constraint wins, esp. constraint 5's cp-only-only-if-no-dep
rule, constraint 8's do-NOT-git-rm-package, and constraint 9's no-handler /
no-alarm change). Re-run the specific failing gate/test per fix.
MECHANICAL:\n${checks ? checks.details : '(agent died — rerun all gates)'}
CONFIRMED FINDINGS:\n${JSON.stringify(confirmed, null, 2)}
${specBlock}`,
{ label: `fix:round-${round}`, model: 'opus', phase: 'Fix', schema: IMPL })
}
const verifyClean = checks && checks.passed && confirmed.length === 0
if (!verifyClean) {
return {
status: 'NEEDS ATTENTION — verify not clean after 3 rounds; branch left uncommitted',
branch: BRANCH,
mechanical: checks,
unresolvedFindings: confirmed,
implBlockers,
reconBlockers,
spec,
}
}
// ----------------------------------------------------------------- package
phase('Package')
const commit = await agent(`${PREAMBLE.replace('do NOT commit, ', '')}
YOU are the commit agent:
1. Read ~/Documents/repositories/seahaven/engineering-handbook/commit-messages.md
and follow it exactly.
2. OPTIONAL LOCAL CLEANUP (constraint 8): rm -rf
lambdas/po/email_processor/package/ — it is UNTRACKED, so this is a
filesystem rm, NOT git rm; there is NOTHING to stage from it and it is
asset-hash-neutral (Phase 2 exclude). PERFORM it and REPORT the reclaimed
~44 MB in your summary. It is NOT part of the commit.
3. git add only paths under: ${EXPECTED_SCOPE.join(', ')} and
.claude/workflows/phase-7-ops-recovery.js. NOT .coverage, NOT package/
(it is gone / untracked either way). Verify the staged set with
git status — the two NEW manifests and the new runbook MUST be staged.
4. ONE commit; write the message to /tmp/phase7-commit-msg.txt and use
git commit -F /tmp/phase7-commit-msg.txt (backticks in -m get eaten by
zsh). Suggested subject:
"feat: ops/recovery tooling + dependency hygiene — generalized reprocess, DLQ runbook, cp-only bundling (refactor phase 7)"
Body: the reprocess generalization (targeted-primary, --all demoted with
caveats) + contract test, the DLQ runbook + windows, the dropped vendored
boto3 / cp-only bundling, the moto exact-pin + three dependabot entries
with pinned manifests, and the reported (not committed) package/ cleanup.
NO AI attribution / Co-Authored-By lines.
5. Do NOT push. Return commit sha + shortstat + the package/ cleanup result
in summary.`,
{ label: 'package:commit', model: 'sonnet', phase: 'Package', schema: IMPL })
return {
status: 'BUILT — committed locally, NOT pushed',
branch: BRANCH,
base: BASE,
commit: commit ? commit.summary : 'commit agent died — commit manually',
spec: { reprocessCli: spec.reprocessCli, runbook: spec.runbook, dependencyEdits: spec.dependencyEdits, bundlingEdits: spec.bundlingEdits, packageCleanup: spec.packageCleanup },
implementation: implOk.map(r => r.summary),
filesChanged: implOk.flatMap(r => r.filesChanged),
verifyRounds: round + 1,
blockers: implBlockers.concat(reconBlockers),
outstandingGates: [
'/sh-security-review NOT required (ops tooling + dependency hygiene only — no untrusted-input parsing, no auth code, no IAM/policy change). The pre-push deterministic scanners still run as the unattended backstop.',
'cross-family cross_review.py NOT required (no IAM/policy change; no Lambda handler signature / event-shape / return-contract change — reprocess.py emits the SAME S3 event shape it always did, pinned by the new contract test).',
'push + PR + gh pr checks green',
'deploy-then-merge for the bundling/dep changes: deploy from branch, smoke green, one real email per pipeline (the dropped-vendored-boto3 email processors MUST still import boto3 at runtime from the Lambda runtime copy), expect ONE benign asset-hash redeploy from the package/ cleanup + cp-only change, THEN merge.',
'reprocess.py + docs/runbook-dlq-recovery.md are ops/docs (no deploy risk), but the runbook should be VALIDATED against the real DLQ / alarm / bucket names before it is relied on in an incident.',
'Confluence "AWS Architecture Map" — Phase 7 adds the DLQ-recovery runbook to the inventory; note the new runbook when the resource inventory is next refreshed.',
],
}

View file

@ -36,6 +36,33 @@ updates:
update-types:
- "minor"
- "patch"
- package-ecosystem: "pip"
directory: "/tests"
schedule:
interval: "weekly"
groups:
minor-and-patch:
update-types:
- "minor"
- "patch"
- package-ecosystem: "pip"
directory: "/lambdas/po/web_ui"
schedule:
interval: "weekly"
groups:
minor-and-patch:
update-types:
- "minor"
- "patch"
- package-ecosystem: "pip"
directory: "/lambdas/po/site_extractor"
schedule:
interval: "weekly"
groups:
minor-and-patch:
update-types:
- "minor"
- "patch"
- package-ecosystem: "github-actions"
directory: "/"
schedule:

View file

@ -88,7 +88,7 @@ Amazon APM work order emails (from Hexagon EAM / HxGN SmartCloud) are received a
## Architecture
**IaC:** AWS CDK (Python), two stacks in one app, region `us-east-1`.
**IaC:** AWS CDK (Python), two stacks in one app, region `us-east-1`. The `cdk.Environment` is deliberately **account-agnostic** (region-only, no `account=`): the stacks deploy to whichever account the deploy credentials target (`328440206208` today), and every account-derived template value — bucket names, `Lambda::Permission` source account, the site-alerts SNS action ARN, the Bedrock ARN below — renders as the CloudFormation `AWS::AccountId` pseudo-parameter rather than a literal. Pinning `account=` was evaluated in Phase 4 and rejected: it would resolve those tokens to literals, and against the deployed (account-agnostic) templates CloudFormation flags the `RemovalPolicy.RETAIN` email buckets as requiring replacement — a data-loss risk — for no functional gain.
All Lambdas: Python 3.12, ARM64, 60-day log retention.
@ -98,9 +98,21 @@ All Lambdas: Python 3.12, ARM64, 60-day log retention.
**SES:** Both stacks add rules to the shared `INBOUND_MAIL` receipt rule set on `int.seahaven.com`.
### Shared CDK helpers (`cdk/common.py`, Phase 4)
The ~379 lines that `po_stack.py` and `wo_stack.py` both defined identically (DynamoDB alarms, the sender-auth-rejected metric filter + alarm, the standard per-Lambda alarm set, the Bedrock `InvokeModel` grant, the raw-email bucket, the async-invoke DLQ, and the template-fallback-rate math alarm) are collapsed into `cdk/common.py`.
**Logical-ID-safety rule (load-bearing).** Every helper is a **plain function** taking `(scope, id, ...)` — never a `Construct` subclass. Each stack calls a helper with its **own Stack instance as `scope`** and the **exact same literal construct id** it used inline before the extraction, so every synthesized logical ID is byte-stable. A `Construct` subclass would insert an extra tree node, reparent every child's logical ID, and CloudFormation would attempt to **replace** the `RemovalPolicy.RETAIN`-protected `purchase-orders` / `WorkOrders` / `WorkOrderComments` tables and the named S3 buckets — a data-loss event. Nothing in `common.py` subclasses `Construct`, and it imports only the CDK constructs its helpers touch (`dynamodb`, `cloudwatch`/`cw_actions`, `iam`, `logs`, `s3`, `sqs` — no `kms`, `ssm`, or Lambda event-source imports).
Extracted helpers: `add_ddb_alarms`, `add_sender_auth_rejected_alarm`, `add_standard_lambda_alarms` (bespoke `descriptions=` dict passed through verbatim per call site — no alarm text is generated or homogenized), `make_bedrock_invoke_statement`, `make_email_bucket`, `make_processor_dlq`, `make_fallback_rate_alarm`. Per-function alarm variance is preserved exactly through call-site arguments, not baked into the helpers: PO's email-processor duration alarm uses `p99`, WO's uses `p95`; `po-web-ui` gets throttles + duration only (no errors, no DLQ); `po-ingest-site-extractor` has no DLQ alarm (it's a DynamoDB-stream consumer, not async-invoked); `workorder-web-ui` gets **zero** alarms — the helper is simply never called for it, so it cannot silently add any. The two "AI-fallback-rejected" alarms (PO's 6-hour count-floor `IF(FILL(rej,0)>=1,...)`, WO's 5-minute/2-of-6 sparse idiom) are a different shape from `make_fallback_rate_alarm` and stay as distinct inline call sites in each stack rather than being forced through the shared helper.
**Bedrock ARN, now account/region-derived.** `make_bedrock_invoke_statement(scope)` builds the inference-profile ARN and the `us-east-1` foundation-model ARN from `Stack.of(scope).account` / `.region` instead of the previous hardcoded `328440206208`/`us-east-1` literals (the `us-east-2`/`us-west-2` foundation-model ARNs stay literal — they're fixed cross-region reach targets, not the stack's own region). Because the environment is account-agnostic (above), `stack.account` is the `AWS::AccountId` pseudo-parameter, so the derived ARN synthesizes as an `Fn::Sub`/`Ref` that **resolves at deploy time to the same ARN** the hardcoded literal named in-account. Versus the deployed stack this is a benign **in-place** IAM policy update (IAM policies never require replacement) — and it makes the grant account-portable instead of pinned to the management account. This IAM PolicyStatement change is the surface the mandatory GPT-4.1 cross-family review covers.
**New `CfnOutput`s.** Each stack now exports its Lambda function ARNs and the DynamoDB table names it owns/consumes, all net-new/additive (no existing output changes): `po-ingest` — `EmailProcessorFunctionArn`, `WebUiFunctionArn`, `SiteExtractorFunctionArn`, `PurchaseOrdersTableName`, `PendingSiteReviewTableName` (the existing `VerifiedSitesTableName` output is unchanged); `workorder-ingest` — `EmailProcessorFunctionArn`, `WebUiFunctionArn`, `WorkOrdersTableName`, `WorkOrderCommentsTableName`.
### Sender authentication (INFRA-107)
The `From` header and any `Authentication-Results` header inside the raw MIME are attacker-forgeable, so neither is trusted. Instead, both email processors (`lambdas/*/email_processor/ses_auth.py`) authenticate the sender against the verdicts SES itself stamps at delivery time, failing closed:
The `From` header and any `Authentication-Results` header inside the raw MIME are attacker-forgeable, so neither is trusted. Instead, both email processors authenticate the sender against the verdicts SES itself stamps at delivery time, failing closed. The authenticator is **single-sourced** at `lambdas/shared/ses_auth.py` (Phase 3) — previously duplicated byte-for-byte in each pipeline's `email_processor/` dir and kept in sync by a fixture-hygiene test; now one copy, so a future hardening fix to the fail-closed logic lands **once** instead of needing two identical edits. The CDK bundling `cp`s it flat beside each handler so the handlers' unchanged `from ses_auth import authenticate_inbound_email` still resolves at runtime (see [`lambdas/shared/`](#deploy-pipeline-guards-phase-0)). The gate:
1. Take **only the topmost** `Authentication-Results` header (SES prepends its trace headers; any lower copies arrived inside the message and are ignored).
2. Require its authserv-id to be `amazonses.com`.
@ -138,7 +150,7 @@ The `<fn>-sender-auth-rejected` alarm closes the silent-drop gap in INFRA-107: a
The `<fn>-duration` and `<fn>-throttles` alarms for `po-email-processor` and `workorder-email-processor` supersede the orphaned, CLI-created `Lambda-Duration-*` / `Lambda-Throttles-*` alarms (deleted post-deploy).
**DLQ alarms** (`AWS/SQS`): `po-email-processor-dlq-messages` and `workorder-email-processor-dlq-messages` fire when any message is visible on an email-processor DLQ (`ApproximateNumberOfMessagesVisible` Maximum, 5 min, `> 0`, eval 1) — a message there means an email was dropped after Lambda exhausted its async retries.
**DLQ alarms** (`AWS/SQS`): `po-email-processor-dlq-messages` and `workorder-email-processor-dlq-messages` fire when any message is visible on an email-processor DLQ (`ApproximateNumberOfMessagesVisible` Maximum, 5 min, `> 0`, eval 1) — a message there means an email was dropped after Lambda exhausted its async retries. Recovery from a DLQ message (no console redrive) is documented in the [DLQ recovery runbook](docs/runbook-dlq-recovery.md).
**Parse-outcome metric + fallback-rate alarm (workorder-ingest):** the WO processor writes one CloudWatch **EMF** line per email to namespace `Seahaven/WorkorderIngest`, metric `ParseOutcome` (Unit Count, value 1), dimensioned by `ParseMethod` (`template` | `ai_fallback` | `ai_fallback_rejected`) and `TemplateId` (`update_plaintext` | `assign_html` | `unknown`). `ai_fallback_rejected` counts AI-fallback output that failed the fail-closed `validate_ai_fallback()` gate (schema/enum/date contract on raw Bedrock output — prompt-injection defence) and was dropped without a DynamoDB write. Non-dimension EMF properties `ReasonCode` and `work_order_id` are queryable in Logs Insights but not promoted to metrics (kept low-cardinality). EMF is used instead of `PutMetricData` so there is no extra sync call / latency / IAM grant on the async hot path (the role already has `logs:PutLogEvents`). The alarm `workorder-email-processor-template-fallback-rate` fires when the AI-fallback share of parses — rejected fallback parses included, so a drift outage whose AI output also fails the gate cannot lower the observed rate while dropping mail — exceeds **15%** sustained (a `MathExpression` with `FILL(...,0)` and a ≥10-sample volume floor over 15-minute periods, eval 3 / datapoints 2) — catching Hexagon template-drift coverage collapse while the volume floor + `FILL` prevent low-volume false pages / `INSUFFICIENT_DATA`. ALARM-only `SnsAction` to `site-alerts`, no OK action, `NOT_BREACHING`. The 15-minute period is a deliberate deviation from the 5-minute house style to accumulate a stable denominator at the low ~760/day volume. A second alarm, `workorder-email-processor-ai-fallback-rejected`, pages on the rejected series itself (≥1 rejection per 5-min period, 2 of the last 6 periods — the sender-auth-rejected sparse-arrival idiom) because a gate rejection drops mail without error/retry/DLQ and would otherwise be silent.
@ -189,6 +201,25 @@ This is cleanup, not a regression: none of those file classes are imported at ru
`tests/test_bundle_consistency.py` guards all of the above with a pure-AST check (no synth, no boto3, no handler import): it parses each handler.py's top-level first-party sibling imports, extracts the bundling `command=[...]` string from the corresponding CDK stack file, and asserts every required sibling module is guaranteed to ship. It recognizes both the scoped glob (`cp po/email_processor/*.py` / `cp wo/email_processor/*.py`, with or without a path prefix) and a whole-dir recursive copy (`cp -r . /asset-output/`) as unconditionally-safe shapes, and falls back to literal filename matching for any other (allowlist-style) shape. It pins each stack's command to the scoped-glob form specifically — a future revert to a narrowed single-file copy, a commented-out glob, or a filename allowlist missing a sibling all fail CI loudly instead of silently shipping a broken bundle. Runs in the existing pytest step, before synth.
### Phase 3: shared module extraction (`lambdas/shared/`)
Four first-party modules that were previously duplicated per pipeline (or inlined in each handler) are now **single-sourced** under `lambdas/shared/`, following the handbook's `lambdas/shared/` convention:
| Module | What it is | Imported by |
|---|---|---|
| `ses_auth.py` | fail-closed SES sender-authentication (INFRA-107) | both email processors |
| `web_ui_auth.py` | fail-closed `X-Auth-Token` gate + token cache (INFRA-74) | both web_ui handlers |
| `email_parsing.py` | `parse_raw_email` (the WO superset that returns `cc` unconditionally; PO simply ignores `cc`) | both email processors |
| `emf.py` | generic CloudWatch EMF emitter (`emit_metric`, `emit_parse_outcome`) parameterized by namespace / dimension-sets / properties | both email processors (PO also uses `emit_metric` for `DerivedFieldAgreement`) |
**Flat-landing import rule.** The shared dir has **no `__init__.py`** — the modules are consumed by bare name (`from ses_auth import ...`, `from emf import emit_parse_outcome`), exactly as when they were siblings. This works because the bundling `cp` lands them **flat in `/asset-output/`** beside `handler.py`, so at runtime each shared module sits on the function's own `sys.path` under its bare name — the handler import lines are unchanged, which is what keeps the byte-identical fail-closed `ses_auth` behavior through the move. The load-bearing `emf` dimension-set list `[["ParseMethod"], ["ParseMethod", "TemplateId"]]` is now pinned **once** in `emf.py` (one-sided dimension drift between the two pipelines becomes structurally impossible), while the deliberate per-pipeline **emission-ordering** differences stay in the handlers (PO emits `ai_fallback` *before* the Bedrock call with an intentional double-count; WO emits mutually-exclusive `ai_fallback`/`ai_fallback_rejected` after its gate).
**Bundling — email processors.** Both email-processor commands append a second glob, `cp shared/*.py /asset-output/`, after their own `cp <pipeline>/email_processor/*.py`. This ships all four shared modules flat into each email-processor zip. `web_ui_auth.py` therefore rides along into both email-processor bundles even though the email handlers never import it — a harmless, deliberate consequence of the all-of-`shared/` glob (`PO_EXPECTED_TOP_LEVEL_MODULES` and the bundle-parity expectations account for it). The base is cp-only (Phase 7 removed the pip install / `manylinux` pin from both email-processor commands, and this phase moves only pure first-party modules with no new dependencies, so it **stays** cp-only — no pip step is reintroduced).
**Bundling — web UIs.** Both `po-web-ui` and `workorder-web-ui` gain the same widened-root Docker bundling mechanism: their `Code.from_asset` root widens to `../lambdas` and their command copies the function's own dir contents plus **only** `shared/web_ui_auth.py` (`cp shared/web_ui_auth.py`, *not* `cp shared/*.py`) — the web UIs need only the auth module, and shipping the email-processor-only modules would break the "deployed set + `web_ui_auth`, nothing else" parity. The per-stack INFRA-74 comments stay in each handler (their wording is deliberately pipeline-specific and is not unified). `site_extractor`'s asset is untouched.
`tests/test_bundle_consistency.py` is updated in lockstep without losing teeth: `_first_party_sibling_imports` resolves shared-sourced imports under `lambdas/shared/`; the command extractor selects the email-processor command now that each stack has two bundled functions; `_bundling_ships_all` accumulates shipped module stems across **both** globs; `PO_EXPECTED_TOP_LEVEL_MODULES` gains the four shared modules; and a new pin + mutation test require the `cp shared/*.py` line to be actually executed (a commented-out or removed shared `cp` fails CI).
## Security
**Web UI auth (defense-in-depth).** The `po-web-ui` / `workorder-web-ui` handlers refuse unauthenticated requests even though their public Function URLs were removed (INFRA-74). Each requires a shared secret in the `X-Auth-Token` header (or `Authorization: Bearer <token>`), compared in constant time against the configured token. The handler **fails closed** if the token is unset or unreadable (denies all). The token lives in the Secrets Manager secret `procurement-ingest/web-ui-auth-token`; only its ARN is passed to the Lambda (`WEB_UI_AUTH_TOKEN_SECRET_ARN`), and the value is fetched at runtime — never embedded in the CloudFormation template or Lambda env vars. The fetched value is cached in the warm container with a short TTL (5 min) so a rotated secret propagates without waiting for the execution environment to recycle. This is a defense-in-depth floor for a detached URL, not primary auth.
@ -305,10 +336,11 @@ All three roots are discovered by `pytest.ini` (`testpaths`).
## Scripts
**Reprocess PO emails** (re-run parser against all emails still in S3):
**Reprocess emails** (re-invoke an email-processor with a synthetic S3 event). `reprocess.py` is now **pipeline-general**: `--pipeline po|wo` selects the function + raw-email bucket. **Targeted replay** (`--key` one object, `--prefix`, or `--since` a `LastModified` timestamp) is the default, preferred mode; the full-prefix sweep is demoted behind an explicit `--all`. Every mode is dry-run unless `--execute`. See the [DLQ recovery runbook](docs/runbook-dlq-recovery.md) for the targeted single-key re-invoke flow.
```bash
python scripts/reprocess.py # dry-run
python scripts/reprocess.py --execute # invoke po-email-processor for each
python scripts/reprocess.py --pipeline po --key inbound/2026/msg.eml # targeted, dry-run
python scripts/reprocess.py --pipeline po --key inbound/2026/msg.eml --execute # targeted, re-invoke one
python scripts/reprocess.py --pipeline wo --all --execute # demoted full-prefix sweep (see --all caveats)
```
**Backfill verified sites** (one-time scan of historical POs):
@ -320,27 +352,39 @@ python scripts/backfill_sites.py
```
cdk/
app.py # Two stacks: po-ingest + WorkorderIngestStack
app.py # Two stacks: po-ingest + WorkorderIngestStack (region-only env)
common.py # Phase 4: shared plain-function CDK helpers (alarms, Bedrock grant,
# email bucket, processor DLQ, fallback-rate alarm) -- called with
# each stack's own scope + literal construct ids, logical-ID-safe
po_stack.py # Purchase order pipeline resources
wo_stack.py # Work order pipeline resources
lambdas/ # Phase 2: shared Code.from_asset("../lambdas") bundling root for
# BOTH po-email-processor and workorder-email-processor -- each
# bundling command `cp`s only its own po/email_processor/*.py or
# wo/email_processor/*.py subset out; site_extractor and both
# web_ui assets keep their own narrower, non-bundled asset root
# BOTH po-email-processor and workorder-email-processor (and, since
# Phase 3, both web_ui functions) -- each email-processor command
# `cp`s its own po/email_processor/*.py (or wo/...) subset PLUS
# `cp shared/*.py`; each web_ui command `cp`s its own dir PLUS only
# `shared/web_ui_auth.py`; site_extractor keeps its narrower, non-
# bundled asset root
shared/ # Phase 3: single-sourced first-party modules, landed FLAT (no
# __init__.py -- bare-name imports) into each bundle by the cp above
ses_auth.py # fail-closed SES sender-auth (INFRA-107) -- one copy, one fix
web_ui_auth.py # fail-closed X-Auth-Token gate + token cache (INFRA-74)
email_parsing.py # parse_raw_email (WO superset; returns cc unconditionally)
emf.py # generic CloudWatch EMF emitter (dimension-sets pinned once)
po/ # PO pipeline Lambdas
email_processor/
handler.py # template-first + Bedrock fallback, EMF metric, merge writes, {"healthcheck": true} early-return
template_parser.py # pure deterministic Coupa parser + fail-closed validation gate
derived_fields.py # deterministic site_code/trade/fiscal_year classifier (untouched by Phase 3)
tests/ # golden-file + validation-gate + fallback-dispatch + healthcheck tests + fixtures
site_extractor/
web_ui/
web_ui/ # handler.py imports `from web_ui_auth import is_authenticated` (shared)
wo/ # WO pipeline Lambdas
email_processor/
handler.py # template-first + Bedrock fallback, EMF metric, #23 comment_id, {"healthcheck": true} early-return
template_parser.py # pure deterministic parser + fail-closed validation gate
tests/ # golden-file + validation-gate + comment_id + fallback + healthcheck tests
web_ui/
web_ui/ # handler.py imports `from web_ui_auth import is_authenticated` (shared)
scripts/
reprocess.py
backfill_sites.py
@ -348,7 +392,7 @@ scripts/
test_local.py # Parse sample emails through Bedrock locally (no AWS mutation)
tests/
requirements.txt # Test-only deps (moto)
conftest.py # AWS env stubs + per-pipeline module loader
conftest.py # AWS env stubs + module loader (per-pipeline siblings + shared/ fallback)
test_pad_zip.py # PO zip-code padding tests
test_parse_raw_email.py # MIME parsing tests (PO + WO handlers)
test_po_merge.py # PO merge-write semantics tests (#97)

331
cdk/common.py Normal file
View file

@ -0,0 +1,331 @@
"""Shared CDK helpers for the procurement-ingest stacks (Phase 4).
Plain free functions extracted from po_stack.py / wo_stack.py. Each takes the
same Stack `scope` and the same literal construct `id` the stacks passed inline,
so every synthesized logical ID is byte-stable. NOTHING here is a Construct
subclass: a subclass would insert a tree node and reparent/replace the
RETAIN-protected tables and named buckets.
"""
from aws_cdk import (
Duration,
RemovalPolicy,
Stack,
aws_cloudwatch as cloudwatch,
aws_cloudwatch_actions as cw_actions,
aws_dynamodb as dynamodb,
aws_iam as iam,
aws_logs as logs,
aws_s3 as s3,
aws_sqs as sqs,
)
# --- DynamoDB alarm operations (moved verbatim from both stacks) ---
_DDB_ALARM_OPERATIONS = [
dynamodb.Operation.GET_ITEM,
dynamodb.Operation.BATCH_GET_ITEM,
dynamodb.Operation.QUERY,
dynamodb.Operation.SCAN,
dynamodb.Operation.PUT_ITEM,
dynamodb.Operation.UPDATE_ITEM,
dynamodb.Operation.DELETE_ITEM,
dynamodb.Operation.BATCH_WRITE_ITEM,
]
# CloudWatch namespace for the log-derived sender-authentication metrics.
_SENDER_AUTH_METRIC_NAMESPACE = "Seahaven/ProcurementIngest"
def add_ddb_alarms(scope, id_prefix, table, alarm_name_prefix, alarm_topic):
"""Add throttle + system-error alarms for a DynamoDB table.
Both fire on any non-zero datapoint in a 5-min window. ALARM-only SnsAction
to site-alerts (no OK action); TreatMissingData NOT_BREACHING.
"""
table.metric_throttled_requests_for_operations(
operations=_DDB_ALARM_OPERATIONS,
period=Duration.minutes(5),
statistic="Sum",
).create_alarm(
scope,
f"{id_prefix}ThrottlesAlarm",
alarm_name=f"{alarm_name_prefix}-throttles",
alarm_description=f"{alarm_name_prefix} DynamoDB throttled requests",
threshold=0,
evaluation_periods=1,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
table.metric_system_errors_for_operations(
operations=_DDB_ALARM_OPERATIONS,
period=Duration.minutes(5),
statistic="Sum",
).create_alarm(
scope,
f"{id_prefix}SystemErrorsAlarm",
alarm_name=f"{alarm_name_prefix}-system-errors",
alarm_description=f"{alarm_name_prefix} DynamoDB server-side (5xx) errors",
threshold=0,
evaluation_periods=1,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
def add_sender_auth_rejected_alarm(scope, id_prefix, function_name, alarm_topic):
"""Metric-filter + alarm on ``sender_auth_rejected`` warnings (INFRA-107).
A rejected inbound email is skipped without erroring the invocation, so it
is invisible to the Errors/Throttles/DLQ alarms. This turns the structured
warning log into a CloudWatch metric and pages when rejections spike --
catching a silent false-reject storm (allowlist wrong, signing-domain
drift, SES header-format change) that would otherwise discard legitimate
mail while the pipeline reports healthy.
ALARM-only SnsAction to site-alerts; no OK action. The metric filter reads
the function's own log group (imported by the deterministic
``/aws/lambda/<fn>`` name, created by the function's log_retention). A plain
substring pattern is used because Lambda prefixes each line with its own
level/timestamp/request-id, so the JSON payload is not a standalone JSON
log event a `{$.event=...}` pattern could match.
"""
metric_name = f"{function_name}-sender-auth-rejected"
logs.MetricFilter(
scope,
f"{id_prefix}SenderAuthRejectedFilter",
log_group=logs.LogGroup.from_log_group_name(
scope,
f"{id_prefix}LogGroup",
f"/aws/lambda/{function_name}",
),
filter_pattern=logs.FilterPattern.literal('"sender_auth_rejected"'),
metric_namespace=_SENDER_AUTH_METRIC_NAMESPACE,
metric_name=metric_name,
metric_value="1",
default_value=0,
)
cloudwatch.Metric(
namespace=_SENDER_AUTH_METRIC_NAMESPACE,
metric_name=metric_name,
period=Duration.minutes(5),
statistic="Sum",
).create_alarm(
scope,
f"{id_prefix}SenderAuthRejectedAlarm",
alarm_name=f"{function_name}-sender-auth-rejected",
alarm_description=(
f"{function_name} rejected inbound mail on sender authentication "
"(possible allowlist/DKIM-domain drift silently dropping real mail)"
),
threshold=1,
evaluation_periods=6,
datapoints_to_alarm=2,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_OR_EQUAL_TO_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
def add_standard_lambda_alarms(
scope,
id_prefix,
fn,
name_prefix,
topic,
*,
duration_statistic,
errors=True,
dlq=None,
descriptions,
):
"""Standard per-Lambda alarm set: errors (optional), throttles, dlq
(optional), duration. Every alarm bespoke-described via `descriptions`
(keys: errors/throttles/dlq/duration passed through VERBATIM). Construct
ids are f"{id_prefix}<Kind>Alarm", alarm names f"{name_prefix}-<kind>",
exactly the inline literals. Order of creation is irrelevant to logical IDs
(ids are explicit) so the fixed errors->throttles->dlq->duration order here
reproduces both PO (dlq before duration) and WO (duration before dlq)
templates identically.
"""
if errors:
fn.metric_errors(period=Duration.minutes(5), statistic="Sum").create_alarm(
scope,
f"{id_prefix}ErrorsAlarm",
alarm_name=f"{name_prefix}-errors",
alarm_description=descriptions["errors"],
threshold=0,
evaluation_periods=1,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(topic))
fn.metric_throttles(period=Duration.minutes(5), statistic="Sum").create_alarm(
scope,
f"{id_prefix}ThrottlesAlarm",
alarm_name=f"{name_prefix}-throttles",
alarm_description=descriptions["throttles"],
threshold=0,
evaluation_periods=1,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(topic))
if dlq is not None:
dlq.metric_approximate_number_of_messages_visible(
period=Duration.minutes(5),
statistic="Maximum",
).create_alarm(
scope,
f"{id_prefix}DlqMessagesAlarm",
alarm_name=f"{name_prefix}-dlq-messages",
alarm_description=descriptions["dlq"],
threshold=0,
evaluation_periods=1,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(topic))
fn.metric_duration(
period=Duration.minutes(5),
statistic=duration_statistic,
).create_alarm(
scope,
f"{id_prefix}DurationAlarm",
alarm_name=f"{name_prefix}-duration",
alarm_description=descriptions["duration"],
threshold=45000,
evaluation_periods=3,
datapoints_to_alarm=2,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_OR_EQUAL_TO_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(topic))
def make_bedrock_invoke_statement(scope):
"""Return the Bedrock InvokeModel PolicyStatement for the email processor.
The us.* inference profile can route cross-region, so the grant covers the
inference-profile ARN plus the per-region foundation-model ARNs
(us-east-1/us-east-2/us-west-2). ARN #1 account and ARN #1/#2 region are
DERIVED from Stack.of(scope).account/.region (not hardcoded 328440206208);
ARN #3/#4 regions (us-east-2/us-west-2) are cross-region reach targets, NOT
the stack's own region, so they stay literal. Caller attaches via
fn.add_to_role_policy(...) on the SAME function -> logical-ID-safe.
"""
stack = Stack.of(scope)
account = stack.account
region = stack.region
return iam.PolicyStatement(
actions=[
"bedrock:InvokeModel",
"bedrock:InvokeModelWithResponseStream",
],
resources=[
f"arn:aws:bedrock:{region}:{account}:inference-profile/us.anthropic.claude-haiku-4-5-20251001-v1:0",
f"arn:aws:bedrock:{region}::foundation-model/anthropic.claude-haiku-4-5-20251001-v1:0",
"arn:aws:bedrock:us-east-2::foundation-model/anthropic.claude-haiku-4-5-20251001-v1:0",
"arn:aws:bedrock:us-west-2::foundation-model/anthropic.claude-haiku-4-5-20251001-v1:0",
],
)
def make_email_bucket(scope, id, name_prefix):
"""Raw-email S3 bucket: BLOCK_ALL public, RETAIN, 90-day expiration.
bucket_name = f"{name_prefix}-{account}" (account derived from the stack),
reproducing f"po-ingest-emails-{self.account}" /
f"workorder-ingest-emails-{self.account}" exactly.
"""
return s3.Bucket(
scope,
id,
bucket_name=f"{name_prefix}-{Stack.of(scope).account}",
block_public_access=s3.BlockPublicAccess.BLOCK_ALL,
removal_policy=RemovalPolicy.RETAIN,
lifecycle_rules=[
s3.LifecycleRule(expiration=Duration.days(90)),
],
)
def make_processor_dlq(scope, id):
"""Async-invoke DLQ: 14-day retention, enforce_ssl. CDK-generated name."""
return sqs.Queue(
scope,
id,
retention_period=Duration.days(14),
enforce_ssl=True,
)
def make_fallback_rate_alarm(
scope,
id,
*,
namespace,
alarm_topic,
alarm_name,
alarm_description,
rejected_included,
period,
threshold,
floor,
evaluation_periods,
datapoints_to_alarm,
):
"""Template-fallback-rate MathExpression alarm.
rejected_included=False (PO): numerator FILL(fb,0), denom fb+tmpl.
rejected_included=True (WO): numerator (fb+rej), denom fb+rej+tmpl.
Expression string, FILL, label reproduced BYTE-FOR-BYTE. GREATER_THAN,
NOT_BREACHING. NO element-wise MAX (post-#102). The two DISTINCT rejected
alarms are NOT built here.
"""
fb_metric = cloudwatch.Metric(
namespace=namespace,
metric_name="ParseOutcome",
dimensions_map={"ParseMethod": "ai_fallback"},
statistic="Sum",
period=period,
)
tmpl_metric = cloudwatch.Metric(
namespace=namespace,
metric_name="ParseOutcome",
dimensions_map={"ParseMethod": "template"},
statistic="Sum",
period=period,
)
if rejected_included:
fb_rej_metric = cloudwatch.Metric(
namespace=namespace,
metric_name="ParseOutcome",
dimensions_map={"ParseMethod": "ai_fallback_rejected"},
statistic="Sum",
period=period,
)
using_metrics = {"fb": fb_metric, "rej": fb_rej_metric, "tmpl": tmpl_metric}
sum_terms = "FILL(fb,0)+FILL(rej,0)+FILL(tmpl,0)"
numerator = "(FILL(fb,0)+FILL(rej,0))"
else:
using_metrics = {"fb": fb_metric, "tmpl": tmpl_metric}
sum_terms = "FILL(fb,0)+FILL(tmpl,0)"
numerator = "FILL(fb,0)"
expression = f"IF(({sum_terms})>={floor}, 100*{numerator}/({sum_terms}), 0)"
fallback_rate = cloudwatch.MathExpression(
expression=expression,
using_metrics=using_metrics,
period=period,
label="TemplateFallbackRatePct",
)
fallback_rate.create_alarm(
scope,
id,
alarm_name=alarm_name,
alarm_description=alarm_description,
threshold=threshold,
evaluation_periods=evaluation_periods,
datapoints_to_alarm=datapoints_to_alarm,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))

View file

@ -8,7 +8,6 @@ from aws_cdk import (
aws_cloudwatch as cloudwatch,
aws_cloudwatch_actions as cw_actions,
aws_dynamodb as dynamodb,
aws_iam as iam,
aws_kms as kms,
aws_lambda as lambda_,
aws_lambda_event_sources as lambda_event_sources,
@ -19,136 +18,11 @@ from aws_cdk import (
aws_ses_actions as ses_actions,
aws_secretsmanager as secretsmanager,
aws_sns as sns,
aws_sqs as sqs,
aws_ssm as ssm,
)
from constructs import Construct
# Operations these tables actually issue (PutItem/UpdateItem/DeleteItem writes,
# GetItem/Query/BatchGetItem reads). DynamoDB emits ThrottledRequests/SystemErrors
# keyed by TableName + Operation only, so the CDK *_for_operations helpers (which
# render a SUM MathExpression across these per-operation metrics) are the correct,
# non-deprecated way to roll a table up to a single alarmable series.
_DDB_ALARM_OPERATIONS = [
dynamodb.Operation.GET_ITEM,
dynamodb.Operation.BATCH_GET_ITEM,
dynamodb.Operation.QUERY,
dynamodb.Operation.SCAN,
dynamodb.Operation.PUT_ITEM,
dynamodb.Operation.UPDATE_ITEM,
dynamodb.Operation.DELETE_ITEM,
dynamodb.Operation.BATCH_WRITE_ITEM,
]
def _add_ddb_alarms(scope, id_prefix, table, alarm_name_prefix, alarm_topic):
"""Add throttle + system-error alarms for a DynamoDB table.
Both fire on any non-zero datapoint in a 5-min window. ALARM-only SnsAction
to site-alerts (no OK action); TreatMissingData NOT_BREACHING.
"""
table.metric_throttled_requests_for_operations(
operations=_DDB_ALARM_OPERATIONS,
period=Duration.minutes(5),
statistic="Sum",
).create_alarm(
scope,
f"{id_prefix}ThrottlesAlarm",
alarm_name=f"{alarm_name_prefix}-throttles",
alarm_description=f"{alarm_name_prefix} DynamoDB throttled requests",
threshold=0,
evaluation_periods=1,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
table.metric_system_errors_for_operations(
operations=_DDB_ALARM_OPERATIONS,
period=Duration.minutes(5),
statistic="Sum",
).create_alarm(
scope,
f"{id_prefix}SystemErrorsAlarm",
alarm_name=f"{alarm_name_prefix}-system-errors",
alarm_description=f"{alarm_name_prefix} DynamoDB server-side (5xx) errors",
threshold=0,
evaluation_periods=1,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
# CloudWatch namespace for the log-derived sender-authentication metrics.
_SENDER_AUTH_METRIC_NAMESPACE = "Seahaven/ProcurementIngest"
def _add_sender_auth_rejected_alarm(scope, id_prefix, function_name, alarm_topic):
"""Metric-filter + alarm on ``sender_auth_rejected`` warnings (INFRA-107).
A rejected inbound email is skipped without erroring the invocation, so it
is invisible to the Errors/Throttles/DLQ alarms. This turns the structured
warning log into a CloudWatch metric and pages when rejections spike --
catching a silent false-reject storm (allowlist wrong, signing-domain
drift, SES header-format change) that would otherwise discard legitimate
mail while the pipeline reports healthy.
ALARM-only SnsAction to site-alerts; no OK action. The metric filter reads
the function's own log group (imported by the deterministic
``/aws/lambda/<fn>`` name, created by the function's log_retention). A plain
substring pattern is used because Lambda prefixes each line with its own
level/timestamp/request-id, so the JSON payload is not a standalone JSON
log event a `{$.event=...}` pattern could match.
"""
metric_name = f"{function_name}-sender-auth-rejected"
logs.MetricFilter(
scope,
f"{id_prefix}SenderAuthRejectedFilter",
log_group=logs.LogGroup.from_log_group_name(
scope,
f"{id_prefix}LogGroup",
f"/aws/lambda/{function_name}",
),
filter_pattern=logs.FilterPattern.literal('"sender_auth_rejected"'),
metric_namespace=_SENDER_AUTH_METRIC_NAMESPACE,
metric_name=metric_name,
metric_value="1",
default_value=0,
)
# Fire on a *sustained* reject condition rather than a volume spike. The
# earlier Sum>=3-over-15-min threshold had a blind spot that is exactly the
# failure this alarm exists to catch: a low-traffic pipeline in total
# drift outage (allowlist wrong / signing-domain changed) may only produce
# a trickle of rejects -- one every few minutes -- that never sums to 3 in
# any window, so the outage never pages. Instead: >=1 reject per 5-min
# period, alarming when 2 of the last 6 periods breach (evaluation_periods=6
# / datapoints_to_alarm=2). Six periods (30 min) with only 2 required
# datapoints closes the sparse-outage residual: even rejections >10-15 min
# apart can still place two breaching datapoints in a single 30-min
# evaluation window. A single stray spoof probe (one lone period) is
# tolerated and self-clears, but a sustained reject condition trips even
# at very low arrival rates. default_value=0 on the metric filter keeps the
# series continuous so NOT_BREACHING only applies before the first datapoint
# ever arrives.
cloudwatch.Metric(
namespace=_SENDER_AUTH_METRIC_NAMESPACE,
metric_name=metric_name,
period=Duration.minutes(5),
statistic="Sum",
).create_alarm(
scope,
f"{id_prefix}SenderAuthRejectedAlarm",
alarm_name=f"{function_name}-sender-auth-rejected",
alarm_description=(
f"{function_name} rejected inbound mail on sender authentication "
"(possible allowlist/DKIM-domain drift silently dropping real mail)"
),
threshold=1,
evaluation_periods=6,
datapoints_to_alarm=2,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_OR_EQUAL_TO_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
import common
class PoIngestStack(Stack):
@ -167,16 +41,7 @@ class PoIngestStack(Stack):
)
# --- S3 bucket for raw emails ---
email_bucket = s3.Bucket(
self,
"EmailBucket",
bucket_name=f"po-ingest-emails-{self.account}",
block_public_access=s3.BlockPublicAccess.BLOCK_ALL,
removal_policy=RemovalPolicy.RETAIN,
lifecycle_rules=[
s3.LifecycleRule(expiration=Duration.days(90)),
],
)
email_bucket = common.make_email_bucket(self, "EmailBucket", "po-ingest-emails")
# --- Shared customer-managed CMK for sensitive DynamoDB tables ---
# Owned by the account-baseline app (alias/seahaven-dynamodb, INFRA-95 /
@ -223,12 +88,7 @@ class PoIngestStack(Stack):
# parse (bad email, transient error) is silently dropped after Lambda's
# retries. CDK generates the queue name to avoid colliding with the
# interim CLI-created po-email-processor-dlq (removed post-deploy).
email_processor_dlq = sqs.Queue(
self,
"EmailProcessorDlq",
retention_period=Duration.days(14),
enforce_ssl=True,
)
email_processor_dlq = common.make_processor_dlq(self, "EmailProcessorDlq")
# --- Lambda function ---
email_processor = lambda_.Function(
@ -246,8 +106,6 @@ class PoIngestStack(Stack):
command=[
"bash",
"-c",
"pip install --platform manylinux2014_aarch64 --only-binary=:all: "
"-r po/email_processor/requirements.txt -t /asset-output && "
# NOTE: non-recursive glob (not `cp -r`) so tests/ and
# the stale package/ dir are never shipped -- only
# top-level .py siblings of handler.py. This replaces
@ -261,7 +119,16 @@ class PoIngestStack(Stack):
# handler.py's first-party imports and asserts this
# command ships all of them, so a future revert back
# to an allowlist that omits a sibling fails CI.
"cp po/email_processor/*.py /asset-output/",
# pip step removed in Phase 7: requirements.txt is now
# empty (boto3 comes from the Lambda runtime), so nothing
# is installed and the manylinux pin has nothing to pin.
# shared/*.py ships the four modules extracted to
# lambdas/shared/ (Phase 3): ses_auth, web_ui_auth,
# email_parsing, emf. Flat cp keeps the bare-name
# imports (e.g. `from ses_auth import ...`) resolving
# unchanged in /asset-output.
"cp po/email_processor/*.py /asset-output/ && "
"cp shared/*.py /asset-output/",
],
),
),
@ -294,37 +161,29 @@ class PoIngestStack(Stack):
# (us-east-1/us-east-2/us-west-2). A profile-only grant AccessDenies at
# runtime whenever the profile routes to a region whose foundation-model
# ARN is not allowed.
email_processor.add_to_role_policy(
iam.PolicyStatement(
actions=[
"bedrock:InvokeModel",
"bedrock:InvokeModelWithResponseStream",
],
resources=[
"arn:aws:bedrock:us-east-1:328440206208:inference-profile/us.anthropic.claude-haiku-4-5-20251001-v1:0",
"arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-haiku-4-5-20251001-v1:0",
"arn:aws:bedrock:us-east-2::foundation-model/anthropic.claude-haiku-4-5-20251001-v1:0",
"arn:aws:bedrock:us-west-2::foundation-model/anthropic.claude-haiku-4-5-20251001-v1:0",
],
)
)
email_processor.add_to_role_policy(common.make_bedrock_invoke_statement(self))
# --- Errors alarm (INFRA-41 / audit H-8) ---
# ALARM-only (no OK action, per the CloudWatch-alarm preference) to the
# shared site-alerts topic. Any errored invocation in a 5-min window pages.
email_processor.metric_errors(
period=Duration.minutes(5),
statistic="Sum",
).create_alarm(
# --- Standard per-Lambda alarms: po-email-processor ---
# errors (INFRA-41 / audit H-8), throttles, DLQ-messages (dropped PO
# emails), and a p99 duration alarm (orphan adoption of the CLI
# Lambda-Duration-po-email-processor under <fn>-duration naming, 45000 ms
# = 75% of the 60s timeout, eval 3 / dp 2). All ALARM-only to site-alerts.
common.add_standard_lambda_alarms(
self,
"EmailProcessorErrorsAlarm",
alarm_name="po-email-processor-errors",
alarm_description="po-email-processor async invocation errors",
threshold=0,
evaluation_periods=1,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
"EmailProcessor",
email_processor,
"po-email-processor",
alarm_topic,
duration_statistic="p99",
errors=True,
dlq=email_processor_dlq,
descriptions={
"errors": "po-email-processor async invocation errors",
"throttles": "po-email-processor invocation throttles",
"dlq": "po-email-processor DLQ has messages (dropped PO emails)",
"duration": "po-email-processor p99 duration approaching the 60s timeout",
},
)
# --- Sender-auth rejection alarm (INFRA-107) ---
# A rejected email (bad/unaligned DKIM verdict) returns normally, so it
@ -336,70 +195,10 @@ class PoIngestStack(Stack):
# A CloudWatch Logs metric filter turns those warnings into a metric so a
# false-reject storm pages instead of vanishing. default_value=0 keeps the
# series populated (alarm stays OK, never INSUFFICIENT_DATA) between events.
_add_sender_auth_rejected_alarm(
common.add_sender_auth_rejected_alarm(
self, "EmailProcessor", "po-email-processor", alarm_topic
)
# --- Throttles alarm: po-email-processor ---
# Any throttled invocation (concurrency cap hit) in a 5-min window pages.
# ALARM-only to site-alerts; no OK action; NOT_BREACHING when no data.
email_processor.metric_throttles(
period=Duration.minutes(5),
statistic="Sum",
).create_alarm(
self,
"EmailProcessorThrottlesAlarm",
alarm_name="po-email-processor-throttles",
alarm_description="po-email-processor invocation throttles",
threshold=0,
evaluation_periods=1,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
# --- DLQ messages-present alarm ---
# Pages when any message lands in the EmailProcessorDlq: a message here
# means a PO email was permanently dropped after Lambda exhausted its
# async retries. Maximum over a single 5-min window > 0 fires; missing
# data (no messages metric emitted) is not breaching. Reuses the shared
# site-alerts topic, ALARM-only, like the errors alarm above. The metric
# helper derives the QueueName dimension from the queue construct, so the
# alarm tracks the CDK-generated queue name without hardcoding it.
email_processor_dlq.metric_approximate_number_of_messages_visible(
period=Duration.minutes(5),
statistic="Maximum",
).create_alarm(
self,
"EmailProcessorDlqMessagesAlarm",
alarm_name="po-email-processor-dlq-messages",
alarm_description="po-email-processor DLQ has messages (dropped PO emails)",
threshold=0,
evaluation_periods=1,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
# --- Duration alarm: po-email-processor (orphan adoption) ---
# Adopts the orphaned CLI alarm Lambda-Duration-po-email-processor under
# the repo's <fn>-duration naming (NEW logical name → no deploy collision;
# delete the orphan post-deploy). p99 / 45000 ms
# (75% of the 60s timeout) / eval 3 of 3 — tighter than the orphan's
# Maximum>=48000 / 1-of-1.
email_processor.metric_duration(
period=Duration.minutes(5),
statistic="p99",
).create_alarm(
self,
"EmailProcessorDurationAlarm",
alarm_name="po-email-processor-duration",
alarm_description="po-email-processor p99 duration approaching the 60s timeout",
threshold=45000,
evaluation_periods=3,
datapoints_to_alarm=2,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_OR_EQUAL_TO_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
# --- Template fallback-rate alarm: po-email-processor ---
# The processor tries a deterministic template parse first and only calls
# the Bedrock AI extractor on a miss/invalid. A sustained rise in the
@ -445,43 +244,23 @@ class PoIngestStack(Stack):
# falsely page this template-drift alarm on top of the dedicated
# rejected alarm below. The rejected series gets its own alarm
# instead (EmailProcessorAiFallbackRejectedAlarm, below).
fb_metric = cloudwatch.Metric(
namespace="Seahaven/PoIngest",
metric_name="ParseOutcome",
dimensions_map={"ParseMethod": "ai_fallback"},
statistic="Sum",
period=Duration.hours(6),
)
tmpl_metric = cloudwatch.Metric(
namespace="Seahaven/PoIngest",
metric_name="ParseOutcome",
dimensions_map={"ParseMethod": "template"},
statistic="Sum",
period=Duration.hours(6),
)
fallback_rate = cloudwatch.MathExpression(
expression=(
"IF((FILL(fb,0)+FILL(tmpl,0))>=8, "
"100*FILL(fb,0)/(FILL(fb,0)+FILL(tmpl,0)), 0)"
),
using_metrics={"fb": fb_metric, "tmpl": tmpl_metric},
period=Duration.hours(6),
label="TemplateFallbackRatePct",
)
fallback_rate.create_alarm(
common.make_fallback_rate_alarm(
self,
"EmailProcessorTemplateFallbackRateAlarm",
namespace="Seahaven/PoIngest",
alarm_topic=alarm_topic,
alarm_name="po-email-processor-template-fallback-rate",
alarm_description=(
"po-email-processor deterministic-template coverage collapse: "
">20% of parses fell back to the Bedrock AI extractor"
),
rejected_included=False,
period=Duration.hours(6),
threshold=20,
floor=8,
evaluation_periods=4,
datapoints_to_alarm=2,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
)
# --- AI-fallback rejected alarm: po-email-processor (Phase 1) ---
# The validate_ai_fallback gate (template_parser.py) fail-closes Bedrock
@ -601,7 +380,32 @@ class PoIngestStack(Stack):
architecture=lambda_.Architecture.ARM_64,
handler="handler.handler",
code=lambda_.Code.from_asset(
"../lambdas/po/web_ui", exclude=["**/__pycache__/**"]
"../lambdas",
exclude=["**/__pycache__/**", "requirements.txt"],
bundling=cdk.BundlingOptions(
image=lambda_.Runtime.PYTHON_3_12.bundling_image,
command=[
"bash",
"-c",
# web_ui_auth.py is shared (lambdas/shared) and must land
# FLAT beside handler.py so the bare
# `from web_ui_auth import is_authenticated` resolves at
# runtime. Only web_ui_auth is copied from shared/ -- the
# other shared modules (ses_auth/email_parsing/emf) are
# email-processor-only and must not bloat the web UI zip.
# requirements.txt stays excluded (Dependabot anchor only,
# never runtime): keeps the deployed file list = {handler,
# web_ui_auth}. NOTE: the top-level `exclude=` on
# from_asset only filters the asset-hash fingerprint, NOT
# the directory Docker bundling actually mounts, so a
# local __pycache__/requirements.txt on disk at synth
# time WOULD otherwise leak into the bundled zip -- strip
# them explicitly post-cp instead of relying on exclude.
"cp -r po/web_ui/. /asset-output/ && "
"cp shared/web_ui_auth.py /asset-output/ && "
"rm -rf /asset-output/__pycache__ /asset-output/requirements.txt",
],
),
),
timeout=Duration.seconds(60),
memory_size=256,
@ -620,38 +424,24 @@ class PoIngestStack(Stack):
po_table.grant_read_data(web_ui)
web_ui_auth_secret.grant_read(web_ui)
# --- Throttles alarm: po-web-ui ---
web_ui.metric_throttles(
period=Duration.minutes(5),
statistic="Sum",
).create_alarm(
# --- Standard per-Lambda alarms: po-web-ui ---
# Throttles + p99 duration only (no errors alarm, no DLQ -- web_ui is a
# synchronous read path with no async DLQ). p99 / 45000 ms (75% of the
# 60s timeout) / eval 3, datapoints 2.
common.add_standard_lambda_alarms(
self,
"WebUiThrottlesAlarm",
alarm_name="po-web-ui-throttles",
alarm_description="po-web-ui invocation throttles",
threshold=0,
evaluation_periods=1,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
# --- Duration alarm: po-web-ui ---
# Net-new (no orphan exists for this function).
# p99 / 45000 ms (75% of the 60s timeout) / eval 3, datapoints 2.
web_ui.metric_duration(
period=Duration.minutes(5),
statistic="p99",
).create_alarm(
self,
"WebUiDurationAlarm",
alarm_name="po-web-ui-duration",
alarm_description="po-web-ui p99 duration approaching the 60s timeout",
threshold=45000,
evaluation_periods=3,
datapoints_to_alarm=2,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_OR_EQUAL_TO_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
"WebUi",
web_ui,
"po-web-ui",
alarm_topic,
duration_statistic="p99",
errors=False,
dlq=None,
descriptions={
"throttles": "po-web-ui invocation throttles",
"duration": "po-web-ui p99 duration approaching the 60s timeout",
},
)
# Public Function URL removed 2026-06-08 (INFRA-74 / audit C-5): the
# unauthenticated FunctionUrlAuthType.NONE URL was deleted out-of-band
@ -682,7 +472,14 @@ class PoIngestStack(Stack):
architecture=lambda_.Architecture.ARM_64,
handler="handler.handler",
code=lambda_.Code.from_asset(
"../lambdas/po/site_extractor", exclude=["**/__pycache__/**"]
# requirements.txt is excluded from the bundle: it exists only
# as a Dependabot anchor (git-based scan sees it), never pip-
# installed (this is a plain non-bundled asset) and never needed
# at runtime (boto3 comes from the Lambda runtime). Excluding it
# keeps the deployed asset hash neutral vs base while the manifest
# still lands in git for Dependabot.
"../lambdas/po/site_extractor",
exclude=["**/__pycache__/**", "requirements.txt"],
),
timeout=Duration.seconds(60),
memory_size=256,
@ -706,56 +503,27 @@ class PoIngestStack(Stack):
)
)
# --- Errors alarm: po-ingest-site-extractor ---
# --- Standard per-Lambda alarms: po-ingest-site-extractor ---
# errors + throttles + p99 duration. NO DLQ alarm: site_extractor is a
# DynamoEventSource stream consumer with no async DLQ attached (dlq=None).
# Stream-consumer errors retry per the event-source config, but a
# persistent failure stalls the verified-sites pipeline. ALARM-only to
# site-alerts; no OK action; NOT_BREACHING when no data.
site_extractor.metric_errors(
period=Duration.minutes(5),
statistic="Sum",
).create_alarm(
# persistent failure stalls the verified-sites pipeline. p99 / 45000 ms
# (75% of the 60s timeout) / eval 3, datapoints 2.
common.add_standard_lambda_alarms(
self,
"SiteExtractorErrorsAlarm",
alarm_name="po-ingest-site-extractor-errors",
alarm_description="po-ingest-site-extractor invocation errors",
threshold=0,
evaluation_periods=1,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
# --- Throttles alarm: po-ingest-site-extractor ---
site_extractor.metric_throttles(
period=Duration.minutes(5),
statistic="Sum",
).create_alarm(
self,
"SiteExtractorThrottlesAlarm",
alarm_name="po-ingest-site-extractor-throttles",
alarm_description="po-ingest-site-extractor invocation throttles",
threshold=0,
evaluation_periods=1,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
# --- Duration alarm: po-ingest-site-extractor ---
# Net-new (no orphan exists for this function).
# p99 / 45000 ms (75% of the 60s timeout) / eval 3, datapoints 2.
site_extractor.metric_duration(
period=Duration.minutes(5),
statistic="p99",
).create_alarm(
self,
"SiteExtractorDurationAlarm",
alarm_name="po-ingest-site-extractor-duration",
alarm_description="po-ingest-site-extractor p99 duration approaching the 60s timeout",
threshold=45000,
evaluation_periods=3,
datapoints_to_alarm=2,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_OR_EQUAL_TO_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
"SiteExtractor",
site_extractor,
"po-ingest-site-extractor",
alarm_topic,
duration_statistic="p99",
errors=True,
dlq=None,
descriptions={
"errors": "po-ingest-site-extractor invocation errors",
"throttles": "po-ingest-site-extractor invocation throttles",
"duration": "po-ingest-site-extractor p99 duration approaching the 60s timeout",
},
)
cdk.CfnOutput(
self,
@ -783,24 +551,56 @@ class PoIngestStack(Stack):
# --- DynamoDB throttle + system-error alarms ---
# ThrottledRequests / SystemErrors emit at TableName + Operation only
# (verified against live CloudWatch: no TableName-only rollup exists, and
# metric_throttled_requests is deprecated/invalid in aws-cdk-lib 2.259.0).
# metric_throttled_requests is deprecated/invalid in aws-cdk-lib 2.261.0).
# Each table currently has zero throttle/error datapoints, so the series
# only materialise on first occurrence — NOT_BREACHING keeps them OK until
# then.
_add_ddb_alarms(
common.add_ddb_alarms(
self, "PurchaseOrdersTable", po_table, "purchase-orders", alarm_topic
)
_add_ddb_alarms(
common.add_ddb_alarms(
self,
"VerifiedSitesTable",
verified_sites_table,
"verified-sites",
alarm_topic,
)
_add_ddb_alarms(
common.add_ddb_alarms(
self,
"PendingSiteReviewTable",
pending_review_table,
"pending-site-review",
alarm_topic,
)
# --- Function ARN + consumed-table-name outputs (Phase 4, additive) ---
cdk.CfnOutput(
self,
"EmailProcessorFunctionArn",
value=email_processor.function_arn,
description="ARN of the po-email-processor Lambda",
)
cdk.CfnOutput(
self,
"WebUiFunctionArn",
value=web_ui.function_arn,
description="ARN of the po-web-ui Lambda",
)
cdk.CfnOutput(
self,
"SiteExtractorFunctionArn",
value=site_extractor.function_arn,
description="ARN of the po-ingest-site-extractor Lambda",
)
cdk.CfnOutput(
self,
"PurchaseOrdersTableName",
value=po_table.table_name,
description="purchase-orders DynamoDB table consumed by this stack",
)
cdk.CfnOutput(
self,
"PendingSiteReviewTableName",
value=pending_review_table.table_name,
description="pending-site-review DynamoDB table",
)

View file

@ -1,2 +1,2 @@
aws-cdk-lib==2.261.0
constructs>=10.6.0
constructs==10.6.0

View file

@ -8,7 +8,6 @@ from aws_cdk import (
aws_cloudwatch as cloudwatch,
aws_cloudwatch_actions as cw_actions,
aws_dynamodb as dynamodb,
aws_iam as iam,
aws_lambda as lambda_,
aws_logs as logs,
aws_s3 as s3,
@ -17,138 +16,10 @@ from aws_cdk import (
aws_ses_actions as ses_actions,
aws_secretsmanager as secretsmanager,
aws_sns as sns,
aws_sqs as sqs,
)
from constructs import Construct
# Operations these tables actually issue (PutItem/UpdateItem/DeleteItem writes,
# GetItem/Query/BatchGetItem reads). DynamoDB emits ThrottledRequests/SystemErrors
# keyed by TableName + Operation only, so the CDK *_for_operations helpers (which
# render a SUM MathExpression across these per-operation metrics) are the correct,
# non-deprecated way to roll a table up to a single alarmable series.
_DDB_ALARM_OPERATIONS = [
dynamodb.Operation.GET_ITEM,
dynamodb.Operation.BATCH_GET_ITEM,
dynamodb.Operation.QUERY,
dynamodb.Operation.SCAN,
dynamodb.Operation.PUT_ITEM,
dynamodb.Operation.UPDATE_ITEM,
dynamodb.Operation.DELETE_ITEM,
dynamodb.Operation.BATCH_WRITE_ITEM,
]
def _add_ddb_alarms(scope, id_prefix, table, alarm_name_prefix, alarm_topic):
"""Add throttle + system-error alarms for a DynamoDB table.
Both fire on any non-zero datapoint in a 5-min window. ALARM-only SnsAction
to site-alerts (no OK action); TreatMissingData NOT_BREACHING.
"""
table.metric_throttled_requests_for_operations(
operations=_DDB_ALARM_OPERATIONS,
period=Duration.minutes(5),
statistic="Sum",
).create_alarm(
scope,
f"{id_prefix}ThrottlesAlarm",
alarm_name=f"{alarm_name_prefix}-throttles",
alarm_description=f"{alarm_name_prefix} DynamoDB throttled requests",
threshold=0,
evaluation_periods=1,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
table.metric_system_errors_for_operations(
operations=_DDB_ALARM_OPERATIONS,
period=Duration.minutes(5),
statistic="Sum",
).create_alarm(
scope,
f"{id_prefix}SystemErrorsAlarm",
alarm_name=f"{alarm_name_prefix}-system-errors",
alarm_description=f"{alarm_name_prefix} DynamoDB server-side (5xx) errors",
threshold=0,
evaluation_periods=1,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
# CloudWatch namespace for the log-derived sender-authentication metrics.
_SENDER_AUTH_METRIC_NAMESPACE = "Seahaven/ProcurementIngest"
def _add_sender_auth_rejected_alarm(scope, id_prefix, function_name, alarm_topic):
"""Metric-filter + alarm on ``sender_auth_rejected`` warnings (INFRA-107).
A rejected inbound email is skipped without erroring the invocation, so it
is invisible to the Errors/Throttles/DLQ alarms. This turns the structured
warning log into a CloudWatch metric and pages when rejections spike --
catching a silent false-reject storm (allowlist wrong, signing-domain
drift, SES header-format change) that would otherwise discard legitimate
mail while the pipeline reports healthy. This is the safety net for the WO
allowlist domain assumption (seahaven.com) -- if the real Gmail-forward
re-signing domain differs, this alarm surfaces it instead of a silent
work-order outage.
ALARM-only SnsAction to site-alerts; no OK action. The metric filter reads
the function's own log group (imported by the deterministic
``/aws/lambda/<fn>`` name, created by the function's log_retention). A plain
substring pattern is used because Lambda prefixes each line with its own
level/timestamp/request-id, so the JSON payload is not a standalone JSON
log event a `{$.event=...}` pattern could match.
"""
metric_name = f"{function_name}-sender-auth-rejected"
logs.MetricFilter(
scope,
f"{id_prefix}SenderAuthRejectedFilter",
log_group=logs.LogGroup.from_log_group_name(
scope,
f"{id_prefix}LogGroup",
f"/aws/lambda/{function_name}",
),
filter_pattern=logs.FilterPattern.literal('"sender_auth_rejected"'),
metric_namespace=_SENDER_AUTH_METRIC_NAMESPACE,
metric_name=metric_name,
metric_value="1",
default_value=0,
)
# Fire on a *sustained* reject condition rather than a volume spike. The
# earlier Sum>=3-over-15-min threshold had a blind spot that is exactly the
# failure this alarm exists to catch: a low-traffic pipeline in total
# drift outage (allowlist wrong / signing-domain changed) may only produce
# a trickle of rejects -- one every few minutes -- that never sums to 3 in
# any window, so the outage never pages. Instead: >=1 reject per 5-min
# period, alarming when 2 of the last 6 periods breach (evaluation_periods=6
# / datapoints_to_alarm=2). Six periods (30 min) with only 2 required
# datapoints closes the sparse-outage residual: even rejections >10-15 min
# apart can still place two breaching datapoints in a single 30-min
# evaluation window. A single stray spoof probe (one lone period) is
# tolerated and self-clears, but a sustained reject condition trips even
# at very low arrival rates. default_value=0 on the metric filter keeps the
# series continuous so NOT_BREACHING only applies before the first datapoint
# ever arrives.
cloudwatch.Metric(
namespace=_SENDER_AUTH_METRIC_NAMESPACE,
metric_name=metric_name,
period=Duration.minutes(5),
statistic="Sum",
).create_alarm(
scope,
f"{id_prefix}SenderAuthRejectedAlarm",
alarm_name=f"{function_name}-sender-auth-rejected",
alarm_description=(
f"{function_name} rejected inbound mail on sender authentication "
"(possible allowlist/DKIM-domain drift silently dropping real mail)"
),
threshold=1,
evaluation_periods=6,
datapoints_to_alarm=2,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_OR_EQUAL_TO_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
import common
class WorkorderIngestStack(Stack):
@ -167,15 +38,8 @@ class WorkorderIngestStack(Stack):
)
# --- S3 bucket for raw emails ---
email_bucket = s3.Bucket(
self,
"EmailBucket",
bucket_name=f"workorder-ingest-emails-{self.account}",
block_public_access=s3.BlockPublicAccess.BLOCK_ALL,
removal_policy=RemovalPolicy.RETAIN,
lifecycle_rules=[
s3.LifecycleRule(expiration=Duration.days(90)),
],
email_bucket = common.make_email_bucket(
self, "EmailBucket", "workorder-ingest-emails"
)
# --- DynamoDB tables ---
@ -222,12 +86,7 @@ class WorkorderIngestStack(Stack):
# parse (bad email, transient error) is silently dropped after Lambda's
# retries. CDK generates the queue name to avoid colliding with the
# interim CLI-created workorder-email-processor-dlq (removed post-deploy).
email_processor_dlq = sqs.Queue(
self,
"EmailProcessorDlq",
retention_period=Duration.days(14),
enforce_ssl=True,
)
email_processor_dlq = common.make_processor_dlq(self, "EmailProcessorDlq")
# --- Lambda function ---
email_processor = lambda_.Function(
@ -245,9 +104,16 @@ class WorkorderIngestStack(Stack):
command=[
"bash",
"-c",
"pip install --platform manylinux2014_aarch64 --only-binary=:all: "
"-r wo/email_processor/requirements.txt -t /asset-output && "
"cp wo/email_processor/*.py /asset-output/",
# pip step removed in Phase 7: requirements.txt is now empty
# (boto3 comes from the Lambda runtime), so nothing is installed
# and the manylinux pin has nothing to pin. cp-only is safe.
# shared/*.py ships the four modules extracted to
# lambdas/shared/ (Phase 3): ses_auth, web_ui_auth,
# email_parsing, emf. Flat cp keeps the bare-name
# imports (e.g. `from ses_auth import ...`) resolving
# unchanged in /asset-output.
"cp wo/email_processor/*.py /asset-output/ && "
"cp shared/*.py /asset-output/",
],
),
),
@ -283,20 +149,7 @@ class WorkorderIngestStack(Stack):
# (us-east-1/us-east-2/us-west-2). A profile-only grant AccessDenies at
# runtime whenever the profile routes to a region whose foundation-model
# ARN is not allowed.
email_processor.add_to_role_policy(
iam.PolicyStatement(
actions=[
"bedrock:InvokeModel",
"bedrock:InvokeModelWithResponseStream",
],
resources=[
"arn:aws:bedrock:us-east-1:328440206208:inference-profile/us.anthropic.claude-haiku-4-5-20251001-v1:0",
"arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-haiku-4-5-20251001-v1:0",
"arn:aws:bedrock:us-east-2::foundation-model/anthropic.claude-haiku-4-5-20251001-v1:0",
"arn:aws:bedrock:us-west-2::foundation-model/anthropic.claude-haiku-4-5-20251001-v1:0",
],
)
)
email_processor.add_to_role_policy(common.make_bedrock_invoke_statement(self))
# NOTE: The pre-emptive grant_encrypt_decrypt on the shared DynamoDB CMK
# (alias/seahaven-dynamodb) was removed (security sweep 2026-06-17). The
@ -308,22 +161,28 @@ class WorkorderIngestStack(Stack):
# point grant_read_write_data on the (then encrypted) tables would propagate
# the needed key permissions automatically.
# --- Errors alarm (INFRA-41 / audit H-8) ---
# ALARM-only (no OK action, per the CloudWatch-alarm preference) to the
# shared site-alerts topic. Any errored invocation in a 5-min window pages.
email_processor.metric_errors(
period=Duration.minutes(5),
statistic="Sum",
).create_alarm(
# --- Standard per-Lambda alarms: workorder-email-processor ---
# errors (INFRA-41 / audit H-8), throttles, DLQ-visible-messages (dropped
# emails), and a p95 duration alarm (orphan adoption of the CLI
# Lambda-Duration-workorder-email-processor under <fn>-duration naming,
# 45000 ms = 75% of the 60s timeout, eval 3 / dp 2). p95 (NOT p99) is the
# WO-specific duration statistic. All ALARM-only to site-alerts.
common.add_standard_lambda_alarms(
self,
"EmailProcessorErrorsAlarm",
alarm_name="workorder-email-processor-errors",
alarm_description="workorder-email-processor async invocation errors",
threshold=0,
evaluation_periods=1,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
"EmailProcessor",
email_processor,
"workorder-email-processor",
alarm_topic,
duration_statistic="p95",
errors=True,
dlq=email_processor_dlq,
descriptions={
"errors": "workorder-email-processor async invocation errors",
"throttles": "workorder-email-processor invocation throttles",
"dlq": "workorder-email-processor DLQ has visible messages (dropped emails)",
"duration": "workorder-email-processor p95 duration approaching the 60s timeout",
},
)
# --- Sender-auth rejection alarm (INFRA-107) ---
# A rejected email (bad/unaligned DKIM verdict) returns normally, so it
@ -334,68 +193,10 @@ class WorkorderIngestStack(Stack):
# different domain), 100% of legitimate work-order mail is silently
# dropped. This metric filter + alarm turns those warnings into a paging
# signal so a false-reject storm surfaces instead of a silent outage.
_add_sender_auth_rejected_alarm(
common.add_sender_auth_rejected_alarm(
self, "EmailProcessor", "workorder-email-processor", alarm_topic
)
# --- Throttles alarm: workorder-email-processor ---
# Any throttled invocation (concurrency cap hit) in a 5-min window pages.
# ALARM-only to site-alerts; no OK action; NOT_BREACHING when no data.
email_processor.metric_throttles(
period=Duration.minutes(5),
statistic="Sum",
).create_alarm(
self,
"EmailProcessorThrottlesAlarm",
alarm_name="workorder-email-processor-throttles",
alarm_description="workorder-email-processor invocation throttles",
threshold=0,
evaluation_periods=1,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
# --- Duration alarm: workorder-email-processor (orphan adoption) ---
# Adopts the orphaned CLI alarm Lambda-Duration-workorder-email-processor
# under the repo's <fn>-duration naming (NEW logical name → no deploy
# collision; delete the orphan post-deploy). p95 /
# 45000 ms (75% of the 60s timeout) / eval 3 of which 2 datapoints —
# tighter than the orphan's Maximum>=48000 / 1-of-1.
email_processor.metric_duration(
period=Duration.minutes(5),
statistic="p95",
).create_alarm(
self,
"EmailProcessorDurationAlarm",
alarm_name="workorder-email-processor-duration",
alarm_description="workorder-email-processor p95 duration approaching the 60s timeout",
threshold=45000,
evaluation_periods=3,
datapoints_to_alarm=2,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_OR_EQUAL_TO_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
# --- DLQ messages-present alarm (INFRA-41 / audit H-8) ---
# The errors alarm above fires on any errored invocation, but a message
# only lands in the DLQ after Lambda exhausts its async retries and gives
# up — i.e. a genuinely dropped email. ALARM-only (no OK action) to the
# same shared site-alerts topic. MAXIMUM over a 5-min window so a single
# visible message pages even if it is later consumed/redriven.
email_processor_dlq.metric_approximate_number_of_messages_visible(
period=Duration.minutes(5),
statistic="Maximum",
).create_alarm(
self,
"EmailProcessorDlqMessagesAlarm",
alarm_name="workorder-email-processor-dlq-messages",
alarm_description="workorder-email-processor DLQ has visible messages (dropped emails)",
threshold=0,
evaluation_periods=1,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
# --- Template fallback-rate alarm: workorder-email-processor ---
# The processor tries a deterministic template parse first and only calls
# the Bedrock AI extractor on a miss/invalid. A sustained rise in the
@ -406,64 +207,28 @@ class WorkorderIngestStack(Stack):
# ~760/day volume; FILL(0) + a >=10-sample volume floor prevent
# low-volume false pages and INSUFFICIENT_DATA. ALARM-only SnsAction to
# site-alerts, no OK action, NOT_BREACHING -- matching the stack idiom.
fb_metric = cloudwatch.Metric(
namespace="Seahaven/WorkorderIngest",
metric_name="ParseOutcome",
dimensions_map={"ParseMethod": "ai_fallback"},
statistic="Sum",
period=Duration.minutes(15),
)
# AI-fallback parses REJECTED by the validate_ai_fallback gate emit
# ParseMethod=ai_fallback_rejected (and nothing else), so they must
# count as fallback here too -- otherwise a drift outage whose AI
# output also fails the gate would LOWER the observed fallback rate
# while silently dropping mail.
fb_rej_metric = cloudwatch.Metric(
namespace="Seahaven/WorkorderIngest",
metric_name="ParseOutcome",
dimensions_map={"ParseMethod": "ai_fallback_rejected"},
statistic="Sum",
period=Duration.minutes(15),
)
tmpl_metric = cloudwatch.Metric(
namespace="Seahaven/WorkorderIngest",
metric_name="ParseOutcome",
dimensions_map={"ParseMethod": "template"},
statistic="Sum",
period=Duration.minutes(15),
)
fallback_rate = cloudwatch.MathExpression(
expression=(
# The IF volume floor (>=10) already guarantees the denominator
# is non-zero in the true branch, so divide directly. (An earlier
# MAX([...,1]) divide-by-zero guard used array syntax CloudWatch
# rejects at deploy: "Unsupported operand type(s) for MAX".)
"IF((FILL(fb,0)+FILL(rej,0)+FILL(tmpl,0))>=10, "
"100*(FILL(fb,0)+FILL(rej,0))"
"/(FILL(fb,0)+FILL(rej,0)+FILL(tmpl,0)), 0)"
),
using_metrics={
"fb": fb_metric,
"rej": fb_rej_metric,
"tmpl": tmpl_metric,
},
period=Duration.minutes(15),
label="TemplateFallbackRatePct",
)
fallback_rate.create_alarm(
# rejected_included=True: the WO expression folds ai_fallback_rejected
# (rej) into BOTH numerator and denominator -- a drift outage whose AI
# output also fails the gate must still count as fallback, otherwise it
# would LOWER the observed rate while silently dropping mail. (Contrast
# PO, which excludes rej to avoid a pre-call double-count.)
common.make_fallback_rate_alarm(
self,
"EmailProcessorTemplateFallbackRateAlarm",
namespace="Seahaven/WorkorderIngest",
alarm_topic=alarm_topic,
alarm_name="workorder-email-processor-template-fallback-rate",
alarm_description=(
"workorder-email-processor deterministic-template coverage "
"collapse: >15% of parses fell back to the Bedrock AI extractor"
),
rejected_included=True,
period=Duration.minutes(15),
threshold=15,
floor=10,
evaluation_periods=3,
datapoints_to_alarm=2,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
)
# --- AI-fallback rejected alarm: workorder-email-processor ---
# A parse rejected by the validate_ai_fallback gate is dropped without
@ -547,7 +312,31 @@ class WorkorderIngestStack(Stack):
architecture=lambda_.Architecture.ARM_64,
handler="handler.handler",
code=lambda_.Code.from_asset(
"../lambdas/wo/web_ui", exclude=["**/__pycache__/**"]
"../lambdas",
exclude=["**/__pycache__/**"],
bundling=cdk.BundlingOptions(
image=lambda_.Runtime.PYTHON_3_12.bundling_image,
command=[
"bash",
"-c",
# web_ui_auth.py is shared (lambdas/shared) and must land
# FLAT beside handler.py so
# `from web_ui_auth import is_authenticated` resolves at
# runtime. Only web_ui_auth is copied from shared/. WO
# baseline keeps __init__.py and requirements.txt, so the
# whole web_ui dir is copied; deployed file list becomes
# {__init__, handler, requirements.txt, web_ui_auth}.
# NOTE: the top-level `exclude=` on from_asset only
# filters the asset-hash fingerprint, NOT the directory
# Docker bundling actually mounts, so a local
# __pycache__ on disk at synth time WOULD otherwise leak
# into the bundled zip -- strip it explicitly post-cp
# instead of relying on exclude.
"cp -r wo/web_ui/. /asset-output/ && "
"cp shared/web_ui_auth.py /asset-output/ && "
"rm -rf /asset-output/__pycache__",
],
),
),
timeout=Duration.seconds(15),
memory_size=128,
@ -571,17 +360,43 @@ class WorkorderIngestStack(Stack):
# --- DynamoDB throttle + system-error alarms ---
# ThrottledRequests / SystemErrors emit at TableName + Operation only
# (verified against live CloudWatch: no TableName-only rollup exists, and
# metric_throttled_requests is deprecated/invalid in aws-cdk-lib 2.259.0).
# metric_throttled_requests is deprecated/invalid in aws-cdk-lib 2.261.0).
# Each table currently has zero throttle/error datapoints, so the series
# only materialise on first occurrence — NOT_BREACHING keeps them OK until
# then.
_add_ddb_alarms(
common.add_ddb_alarms(
self, "WorkOrdersTable", work_orders_table, "WorkOrders", alarm_topic
)
_add_ddb_alarms(
common.add_ddb_alarms(
self, "WorkOrderComments", comments_table, "WorkOrderComments", alarm_topic
)
# --- Function ARN + consumed-table-name outputs (Phase 4, additive) ---
cdk.CfnOutput(
self,
"EmailProcessorFunctionArn",
value=email_processor.function_arn,
description="ARN of the workorder-email-processor Lambda",
)
cdk.CfnOutput(
self,
"WebUiFunctionArn",
value=web_ui.function_arn,
description="ARN of the workorder-web-ui Lambda",
)
cdk.CfnOutput(
self,
"WorkOrdersTableName",
value=work_orders_table.table_name,
description="WorkOrders DynamoDB table",
)
cdk.CfnOutput(
self,
"WorkOrderCommentsTableName",
value=comments_table.table_name,
description="WorkOrderComments DynamoDB table",
)
# Public Function URL removed 2026-06-08 (INFRA-74 / audit C-5): the
# unauthenticated FunctionUrlAuthType.NONE URL was deleted out-of-band
# via CLI. Removing the construct (and its auto-generated Principal:*

View file

@ -0,0 +1,117 @@
# DLQ Recovery Runbook — Email-Processor Dead-Letter Queues
Operational procedure for draining an email-processor dead-letter queue (DLQ)
after a failed async parse. There is **no console redrive-to-source** for these
queues; recovery is a manual, targeted re-invoke via `scripts/reprocess.py`.
## Scope / mechanism
Both email processors set `dead_letter_queue=` on the `lambda_.Function`
construct. This is the legacy per-function **Lambda `DeadLetterConfig`**
(asynchronous-invocation DLQ), **not** an EventInvokeConfig on-failure
Destination — `aws lambda get-function-event-invoke-config` returns
`ResourceNotFoundException` for both functions (no destination config exists). A
failed async invocation lands on the DLQ only **after** Lambda exhausts its
automatic retries.
There is **no console redrive-to-source**: the SQS console's "redrive to source"
applies only to SQS-to-SQS DLQ relationships, not to a Lambda `DeadLetterConfig`
target. Recovery is manual, via targeted re-invoke.
## Resource inventory (acct 328440206208, us-east-1)
| Pipeline | Function | DLQ queue name | Raw-email bucket | DLQ alarm |
|---|---|---|---|---|
| PO | `po-email-processor` | `po-ingest-EmailProcessorDlqA753DED5-az8LUZE3ubtz` | `po-ingest-emails-328440206208` | `po-email-processor-dlq-messages` |
| WO | `workorder-email-processor` | `WorkorderIngestStack-EmailProcessorDlqA753DED5-Q8H555LrqSU1` | `workorder-ingest-emails-328440206208` | `workorder-email-processor-dlq-messages` |
DLQ URLs are `https://sqs.us-east-1.amazonaws.com/328440206208/<queue-name>`.
Both DLQs: 14-day retention, SSE, TLS-enforced, `VisibilityTimeout` 30s.
## Recovery procedure (no redrive — receive → extract key → targeted re-invoke → verify → purge)
1. **Trigger.** The `<fn>-dlq-messages` alarm fires
(`ApproximateNumberOfMessagesVisible` Maximum, 5 min, `> 0`, eval 1).
2. **RECEIVE** the message (do not purge yet):
```bash
aws sqs receive-message \
--queue-url <DLQ_URL> \
--max-number-of-messages 1 \
--visibility-timeout 120 \
--wait-time-seconds 5
```
Capture the `ReceiptHandle` from the response.
3. **EXTRACT the S3 key.** The `DeadLetterConfig` message `Body` is the original
async invocation payload — the S3 event JSON. Read
`Records[0].s3.bucket.name` and `Records[0].s3.object.key` from the `Body`.
This is the **raw** object key that reprocess/S3 emitted (no URL-decoding
applied).
4. **RE-INVOKE (targeted; dry-run first).** Confirm the key with a dry-run, then
execute:
```bash
# PO queue:
python scripts/reprocess.py --pipeline po --key '<key>' # dry-run
python scripts/reprocess.py --pipeline po --key '<key>' --execute # re-invoke
# WO queue:
python scripts/reprocess.py --pipeline wo --key '<key>' --execute
```
This re-invokes the **same** function with the same raw-key synthetic S3
event (a single object — **not** `--all`).
5. **VERIFY the write.** Confirm the downstream effect landed before proceeding:
the DynamoDB item exists / was updated (`purchase-orders` for PO,
`WorkOrders` for WO), and the function's log group shows a clean parse (no new
error, no new DLQ message). Do not proceed until verified.
6. **PURGE the one message.** Delete only the processed message by its
`ReceiptHandle`:
```bash
aws sqs delete-message --queue-url <DLQ_URL> --receipt-handle '<ReceiptHandle>'
```
Do **not** `purge-queue` — that would drop unexamined breadcrumbs. AWS access
is otherwise read-only; `delete-message` on a DLQ you are actively draining is
the one write this runbook performs.
## Recovery windows
- **DLQ breadcrumb retention: 14 days** (`MessageRetentionPeriod=1209600s`,
confirmed live on both queues). After 14 days the breadcrumb is gone.
- **Raw-email S3 retention: 90 days.** Both buckets have
`RemovalPolicy.RETAIN`, **but** a single Enabled lifecycle rule
(`Expiration.Days=90`, `Filter.Prefix=''`) expires objects bucket-wide. **The
lifecycle rule OVERRIDES the RETAIN policy — S3 is the real replay floor:** a
raw email is gone at ~90 days regardless of the table RETAIN policy.
Because 14 days (DLQ) < 90 days (S3), any object referenced by a live DLQ
breadcrumb is always still in S3, so DLQ replay within its 14-day window is never
blocked by S3 expiry. The 90-day floor binds only for replays reconstructed from
other sources (e.g. logs) after the breadcrumb has expired.
## Intentional non-DLQ drops (do NOT hunt for these in the DLQ)
Two drop classes **never** produce a DLQ message because they are fail-closed
**skips, not errors** — the handler returns normally (no raise, no retry, no DLQ
message):
- **Sender-auth rejections.** Logged as a structured `sender_auth_rejected`
warning and skipped; covered by the `<fn>-sender-auth-rejected` alarm
(log-metric filter), **not** the DLQ.
- **`ai_fallback_rejected` drops.** AI-fallback output that failed the
fail-closed `validate_ai_fallback()` gate, emitted as a
`ParseMethod=ai_fallback_rejected` EMF datapoint and dropped without a
DynamoDB write; covered by the `<fn>-ai-fallback-rejected` alarm, **not** the
DLQ.
If mail is missing but the DLQ is empty, check those two alarms / log filters —
the email was intentionally rejected. Re-invoking it via reprocess will just be
rejected again; fix the sender-auth config or the upstream email, not the DLQ.

View file

@ -7,17 +7,17 @@ back to Claude on Bedrock for structured extraction on a miss/invalid result,
then writes the result to the purchase-orders DynamoDB table.
"""
import email
import json
import logging
import os
import re
from datetime import datetime, timezone
from decimal import Decimal, InvalidOperation
from email import policy
import boto3
from derived_fields import derive_all
from email_parsing import parse_raw_email
from emf import emit_metric, emit_parse_outcome
from ses_auth import authenticate_inbound_email
from template_parser import try_deterministic_parse, validate_ai_fallback
@ -36,7 +36,6 @@ BEDROCK_MODEL_ID = os.environ.get(
# CloudWatch EMF namespace/metric for the parse-outcome metric (the PO
# fallback-rate alarm in cdk/po_stack.py reads the ["ParseMethod"] series).
METRIC_NAMESPACE = "Seahaven/PoIngest"
METRIC_NAME = "ParseOutcome"
# Shadow telemetry for the Python-derived classifier bake. One EMF record per
# derived field per email, emitted ONLY on the ai_fallback path (the template
@ -249,36 +248,6 @@ The Coupa commodity/category field if present in the email (e.g., "Maintenance -
"""
def parse_raw_email(raw_bytes: bytes) -> dict:
"""Parse a raw MIME email into subject, sender, and body text."""
msg = email.message_from_bytes(raw_bytes, policy=policy.default)
subject = msg.get("Subject", "")
sender = msg.get("From", "")
to = msg.get("To", "")
date = msg.get("Date", "")
body = ""
if msg.is_multipart():
for part in msg.walk():
content_type = part.get_content_type()
if content_type == "text/plain":
body = part.get_content()
break
elif content_type == "text/html" and not body:
body = part.get_content()
else:
body = msg.get_content()
return {
"subject": subject,
"sender": sender,
"to": to,
"date": date,
"body": body,
}
def extract_with_claude(email_data: dict) -> dict:
"""Send parsed email to Claude on Bedrock for structured extraction.
@ -347,26 +316,9 @@ def _emit_parse_method_metric(method, template_id, reason_code, po_number):
dashboards). CloudWatch materializes only the exact dimension sets listed
here and does NOT auto-aggregate, so the alarm's single-dimension query
would receive no data unless ["ParseMethod"] is emitted explicitly."""
emf = {
"_aws": {
# EMF requires Timestamp (epoch ms); without it CloudWatch may not
# extract the metric datapoint from the log event.
"Timestamp": int(datetime.now(timezone.utc).timestamp() * 1000),
"CloudWatchMetrics": [
{
"Namespace": METRIC_NAMESPACE,
"Dimensions": [["ParseMethod"], ["ParseMethod", "TemplateId"]],
"Metrics": [{"Name": METRIC_NAME, "Unit": "Count"}],
}
],
},
"ParseMethod": method,
"TemplateId": template_id or "unknown",
"ReasonCode": reason_code or "ok",
"po_number": po_number or "",
METRIC_NAME: 1,
}
print(json.dumps(emf))
emit_parse_outcome(
METRIC_NAMESPACE, method, template_id, reason_code, "po_number", po_number
)
def _derived_agreement(llm_value, python_value) -> str | None:
@ -403,29 +355,22 @@ def _emit_derived_agreement_metric(field, llm_value, python_value, po_number):
agreement = _derived_agreement(llm_value, python_value)
if agreement is None:
return
emf = {
"_aws": {
"Timestamp": int(datetime.now(timezone.utc).timestamp() * 1000),
"CloudWatchMetrics": [
{
"Namespace": METRIC_NAMESPACE,
"Dimensions": [["Field", "Agreement"]],
"Metrics": [{"Name": DERIVED_METRIC_NAME, "Unit": "Count"}],
}
],
emit_metric(
METRIC_NAMESPACE,
DERIVED_METRIC_NAME,
[["Field", "Agreement"]],
{
"Field": field,
"Agreement": agreement,
"po_number": po_number or "",
# Length-clamped: Python values are regex/enum-bounded by construction,
# but the LLM value is schema-unvalidated model output -- a hallucinated
# free-text field must not land unbounded in a 2-month log line
# (sh-security-review PO-DC-02, confirmed low).
"PythonValue": "" if python_value is None else str(python_value)[:64],
"LlmValue": "" if llm_value is None else str(llm_value)[:64],
},
"Field": field,
"Agreement": agreement,
"po_number": po_number or "",
# Length-clamped: Python values are regex/enum-bounded by construction,
# but the LLM value is schema-unvalidated model output -- a hallucinated
# free-text field must not land unbounded in a 2-month log line
# (sh-security-review PO-DC-02, confirmed low).
"PythonValue": "" if python_value is None else str(python_value)[:64],
"LlmValue": "" if llm_value is None else str(llm_value)[:64],
DERIVED_METRIC_NAME: 1,
}
print(json.dumps(emf))
)
def pad_zip(zip_code: str | None) -> str | None:

View file

@ -1 +1,2 @@
boto3>=1.43.47
# Per-function dependencies. Leave empty if the function uses only boto3 and stdlib.
# boto3 is provided by the Lambda Python runtime (lambda-template.md) — do not vendor it.

View file

@ -34,6 +34,9 @@ import moto # noqa: F401
_HERE = os.path.dirname(__file__)
_MODULE_DIR = os.path.abspath(os.path.join(_HERE, ".."))
# Phase 3 single-sourced ses_auth/email_parsing/emf under lambdas/shared/.
# _HERE = lambdas/po/email_processor/tests -> ../../.. = lambdas.
_SHARED_DIR = os.path.abspath(os.path.join(_HERE, "..", "..", "..", "shared"))
# The handler creates boto3 clients at import time; give it a region and dummy
# creds so import works offline/under CI.
@ -59,6 +62,24 @@ def _load_module(filename, module_name):
return module
def _load_shared_module(filename, module_name):
"""Load a Phase 3 shared sibling (ses_auth/email_parsing/emf) from
lambdas/shared/. Mirrors ``_load_module`` but resolves against
``_SHARED_DIR`` -- the handler ``from email_parsing import ...`` /
``from emf import ...`` / ``from ses_auth import ...`` bare imports must be
pre-bound in sys.modules because this support module does NOT put shared/
on sys.path."""
if module_name in sys.modules:
return sys.modules[module_name]
spec = importlib.util.spec_from_file_location(
module_name, os.path.join(_SHARED_DIR, filename)
)
module = importlib.util.module_from_spec(spec)
sys.modules[module_name] = module
spec.loader.exec_module(module)
return module
def load_template_parser():
return _load_module("template_parser.py", f"{_HANDLER_NAME}__template_parser")
@ -68,10 +89,14 @@ def load_po_handler():
return sys.modules[_HANDLER_NAME]
siblings = {
"template_parser": load_template_parser(),
"ses_auth": _load_module("ses_auth.py", f"{_HANDLER_NAME}__ses_auth"),
"ses_auth": _load_shared_module("ses_auth.py", f"{_HANDLER_NAME}__ses_auth"),
"derived_fields": _load_module(
"derived_fields.py", f"{_HANDLER_NAME}__derived_fields"
),
"email_parsing": _load_shared_module(
"email_parsing.py", f"{_HANDLER_NAME}__email_parsing"
),
"emf": _load_shared_module("emf.py", f"{_HANDLER_NAME}__emf"),
}
saved = {name: sys.modules.get(name) for name in siblings}
sys.modules.update(siblings)

View file

@ -633,10 +633,12 @@ def test_fixture_hygiene_ses_auth_and_scrub_markers():
from _po_parser_support import (
ADVERSARIAL_STEMS,
AI_FALLBACK_STEMS,
_load_module,
_load_shared_module,
)
ses_auth = _load_module("ses_auth.py", "po_email_processor_handler__ses_auth")
ses_auth = _load_shared_module(
"ses_auth.py", "po_email_processor_handler__ses_auth"
)
coupa = (
[("new-po", s) for s in NEW_PO_STEMS]
+ [("cancellation", s) for s in CANCELLATION_STEMS]

View file

@ -0,0 +1 @@
boto3==1.43.31

View file

@ -5,15 +5,14 @@ Serves a simple HTML dashboard for viewing purchase orders.
Accessed via Lambda Function URL.
"""
import hmac
import json
import logging
import os
import time
from decimal import Decimal
from html import escape as esc
import boto3
from web_ui_auth import is_authenticated
logger = logging.getLogger()
logger.setLevel(logging.INFO)
@ -30,73 +29,6 @@ PO_TABLE = os.environ.get("PO_TABLE", "purchase-orders")
# Manager to keep it out of CloudFormation templates and Lambda env vars. If the
# ARN is unset or the secret is missing, the handler fails closed and denies every
# request.
_WEB_UI_AUTH_TOKEN_SECRET_ARN = os.environ.get("WEB_UI_AUTH_TOKEN_SECRET_ARN")
# Refresh the cached token this often so a rotated secret propagates without
# waiting for the execution environment to recycle (emergency-rotation path).
_AUTH_TOKEN_CACHE_TTL_SECONDS = 300
_auth_token_cache = None
_auth_token_cached_at = 0.0
def _get_auth_token() -> str | None:
"""Fetch the shared web UI auth token from Secrets Manager.
Cached in the warm container for a short TTL so we don't hit Secrets Manager
on every request, while still picking up a rotated secret within the TTL
rather than only when the execution environment recycles. Returns None when
not configured or unreadable (the caller then fails closed).
"""
global _auth_token_cache, _auth_token_cached_at
now = time.monotonic()
if (
_auth_token_cache is not None
and now - _auth_token_cached_at < _AUTH_TOKEN_CACHE_TTL_SECONDS
):
return _auth_token_cache
if not _WEB_UI_AUTH_TOKEN_SECRET_ARN:
return None
secrets = boto3.client("secretsmanager")
try:
secret = secrets.get_secret_value(SecretId=_WEB_UI_AUTH_TOKEN_SECRET_ARN)
_auth_token_cache = secret["SecretString"]
_auth_token_cached_at = now
return _auth_token_cache
except Exception:
# Fail closed (return None -> caller 401s) but surface the failure: a
# Secrets Manager permission/config error would otherwise make every
# request 401 with no operational signal. The secret value is never
# logged.
logger.exception(
"Failed to fetch web UI auth token from Secrets Manager; "
"denying request (failing closed)"
)
return None
def _header(event: dict, name: str) -> str:
"""Case-insensitive header lookup from a Lambda Function URL / APIGW event."""
headers = event.get("headers") or {}
name_lower = name.lower()
for key, value in headers.items():
if key.lower() == name_lower:
return value or ""
return ""
def is_authenticated(event: dict) -> bool:
"""Constant-time check of the request's shared secret against the configured
token. Fails closed when no token is configured."""
token = _get_auth_token()
if not token:
return False
presented = _header(event, "x-auth-token")
if not presented:
auth = _header(event, "authorization")
if auth.lower().startswith("bearer "):
presented = auth[7:].strip()
if not presented:
return False
return hmac.compare_digest(presented, token)
def get_purchase_orders(limit=500):

View file

@ -0,0 +1 @@
boto3==1.43.31

View file

@ -0,0 +1,34 @@
import email
from email import policy
def parse_raw_email(raw_bytes: bytes) -> dict:
"""Parse a raw email into subject, sender, body text."""
msg = email.message_from_bytes(raw_bytes, policy=policy.default)
subject = msg.get("Subject", "")
sender = msg.get("From", "")
to = msg.get("To", "")
cc = msg.get("Cc", "")
date = msg.get("Date", "")
body = ""
if msg.is_multipart():
for part in msg.walk():
content_type = part.get_content_type()
if content_type == "text/plain":
body = part.get_content()
break
elif content_type == "text/html" and not body:
body = part.get_content()
else:
body = msg.get_content()
return {
"subject": subject,
"sender": sender,
"to": to,
"cc": cc,
"date": date,
"body": body,
}

82
lambdas/shared/emf.py Normal file
View file

@ -0,0 +1,82 @@
"""Shared CloudWatch EMF emitter for the procurement-ingest pipelines.
A single generic ``emit_metric`` builds the Embedded Metric Format envelope
(``_aws`` block + promoted dimension properties + the metric-value key) and
prints it to stdout, where the Lambda log subscription materializes the metric.
Both the PO and WO email processors emit their parse-outcome metric through
``emit_parse_outcome`` so the load-bearing dimension-set list
``[["ParseMethod"], ["ParseMethod", "TemplateId"]]`` is pinned in exactly ONE
place -- a one-sided dimension change to one pipeline is structurally
impossible.
Envelope byte-equivalence rests on CPython dict insertion order (preserved by
json.dumps with default separators): ``{**properties, metric_name: value}``
yields ``_aws`` first, then the caller's properties in order, then the metric
value last -- matching every inline emitter this module replaced.
"""
import json
from datetime import datetime, timezone
PARSE_METRIC_NAME = "ParseOutcome"
# Two dimension sets are published for every parse-outcome metric: ["ParseMethod"]
# (aggregated across all template ids -- the series the fallback-rate alarms
# query) AND ["ParseMethod", "TemplateId"] (per-template breakdown for Logs
# Insights / dashboards). CloudWatch materializes only the exact dimension sets
# listed here and does NOT auto-aggregate, so an alarm's single-dimension query
# would receive no data unless ["ParseMethod"] is emitted explicitly. Pinned
# ONCE here; both pipelines share it.
_PARSE_DIMENSION_SETS = [["ParseMethod"], ["ParseMethod", "TemplateId"]]
def emit_metric(
namespace, metric_name, dimension_sets, properties, *, value=1, unit="Count"
):
"""Print one CloudWatch EMF log line for ``metric_name`` in ``namespace``.
Zero-latency (no PutMetricData API call): the extraction path is async and
the role already has logs:PutLogEvents. ``properties`` are emitted in caller
order between the ``_aws`` block and the trailing metric-value key; the keys
named in ``dimension_sets`` are the promoted (dimensioned) fields, the rest
ride along as Logs-Insights-queryable properties.
"""
emf = {
"_aws": {
# EMF requires Timestamp (epoch ms); without it CloudWatch may not
# extract the metric datapoint from the log event.
"Timestamp": int(datetime.now(timezone.utc).timestamp() * 1000),
"CloudWatchMetrics": [
{
"Namespace": namespace,
"Dimensions": dimension_sets,
"Metrics": [{"Name": metric_name, "Unit": unit}],
}
],
},
**properties,
metric_name: value,
}
print(json.dumps(emf))
def emit_parse_outcome(namespace, method, template_id, reason_code, id_key, id_value):
"""Emit one parse-outcome EMF line shared by both email processors.
ParseMethod/TemplateId are the only promoted (dimensioned) fields to keep
cardinality low; ReasonCode and the pipeline id (``id_key``: ``po_number``
or ``work_order_id``) ride along as Logs-Insights-queryable properties. See
``_PARSE_DIMENSION_SETS`` for why both the single- and two-dimension sets
are published.
"""
emit_metric(
namespace,
PARSE_METRIC_NAME,
_PARSE_DIMENSION_SETS,
{
"ParseMethod": method,
"TemplateId": template_id or "unknown",
"ReasonCode": reason_code or "ok",
id_key: id_value or "",
},
)

View file

@ -0,0 +1,77 @@
import hmac
import logging
import os
import time
import boto3
logger = logging.getLogger()
_WEB_UI_AUTH_TOKEN_SECRET_ARN = os.environ.get("WEB_UI_AUTH_TOKEN_SECRET_ARN")
# Refresh the cached token this often so a rotated secret propagates without
# waiting for the execution environment to recycle (emergency-rotation path).
_AUTH_TOKEN_CACHE_TTL_SECONDS = 300
_auth_token_cache = None
_auth_token_cached_at = 0.0
def _get_auth_token() -> str | None:
"""Fetch the shared web UI auth token from Secrets Manager.
Cached in the warm container for a short TTL so we don't hit Secrets Manager
on every request, while still picking up a rotated secret within the TTL
rather than only when the execution environment recycles. Returns None when
not configured or unreadable (the caller then fails closed).
"""
global _auth_token_cache, _auth_token_cached_at
now = time.monotonic()
if (
_auth_token_cache is not None
and now - _auth_token_cached_at < _AUTH_TOKEN_CACHE_TTL_SECONDS
):
return _auth_token_cache
if not _WEB_UI_AUTH_TOKEN_SECRET_ARN:
return None
secrets = boto3.client("secretsmanager")
try:
secret = secrets.get_secret_value(SecretId=_WEB_UI_AUTH_TOKEN_SECRET_ARN)
_auth_token_cache = secret["SecretString"]
_auth_token_cached_at = now
return _auth_token_cache
except Exception:
# Fail closed (return None -> caller 401s) but surface the failure: a
# Secrets Manager permission/config error would otherwise make every
# request 401 with no operational signal. The secret value is never
# logged.
logger.exception(
"Failed to fetch web UI auth token from Secrets Manager; "
"denying request (failing closed)"
)
return None
def _header(event: dict, name: str) -> str:
"""Case-insensitive header lookup from a Lambda Function URL / APIGW event."""
headers = event.get("headers") or {}
name_lower = name.lower()
for key, value in headers.items():
if key.lower() == name_lower:
return value or ""
return ""
def is_authenticated(event: dict) -> bool:
"""Constant-time check of the request's shared secret against the configured
token. Fails closed when no token is configured."""
token = _get_auth_token()
if not token:
return False
presented = _header(event, "x-auth-token")
if not presented:
auth = _header(event, "authorization")
if auth.lower().startswith("bearer "):
presented = auth[7:].strip()
if not presented:
return False
return hmac.compare_digest(presented, token)

View file

@ -6,19 +6,18 @@ Parses the raw email, sends it to Claude for structured extraction,
then writes the result to DynamoDB.
"""
import email
import hashlib
import json
import logging
import os
import re
from datetime import datetime, timezone
from email import policy
from email.utils import parsedate_to_datetime
import boto3
from email_parsing import parse_raw_email
from emf import emit_parse_outcome
from ses_auth import authenticate_inbound_email
from template_parser import try_deterministic_parse, validate_ai_fallback
logger = logging.getLogger()
@ -36,7 +35,6 @@ BEDROCK_MODEL_ID = os.environ.get(
# CloudWatch EMF namespace + metric for parse-outcome observability.
METRIC_NAMESPACE = "Seahaven/WorkorderIngest"
METRIC_NAME = "ParseOutcome"
EXTRACTION_PROMPT = """\
You are an email parser for a facilities maintenance work order system.
@ -79,38 +77,6 @@ Rules:
"""
def parse_raw_email(raw_bytes: bytes) -> dict:
"""Parse a raw email into subject, sender, body text."""
msg = email.message_from_bytes(raw_bytes, policy=policy.default)
subject = msg.get("Subject", "")
sender = msg.get("From", "")
to = msg.get("To", "")
cc = msg.get("Cc", "")
date = msg.get("Date", "")
body = ""
if msg.is_multipart():
for part in msg.walk():
content_type = part.get_content_type()
if content_type == "text/plain":
body = part.get_content()
break
elif content_type == "text/html" and not body:
body = part.get_content()
else:
body = msg.get_content()
return {
"subject": subject,
"sender": sender,
"to": to,
"cc": cc,
"date": date,
"body": body,
}
# An <email>/</email> (or whitespace-padded variant) appearing INSIDE the
# untrusted email text could forge the data-block boundary, so any such
# sequence is neutralized before wrapping. A single [\s/]* class (not two
@ -185,26 +151,14 @@ def emit_parse_metric(method, template_id, reason_code, work_order_id):
dashboards). CloudWatch materializes only the exact dimension sets listed
here and does NOT auto-aggregate, so the alarm's single-dimension query
would receive no data unless ["ParseMethod"] is emitted explicitly."""
emf = {
"_aws": {
# EMF requires Timestamp (epoch ms); without it CloudWatch may not
# extract the metric datapoint from the log event (advisory A2).
"Timestamp": int(datetime.now(timezone.utc).timestamp() * 1000),
"CloudWatchMetrics": [
{
"Namespace": METRIC_NAMESPACE,
"Dimensions": [["ParseMethod"], ["ParseMethod", "TemplateId"]],
"Metrics": [{"Name": METRIC_NAME, "Unit": "Count"}],
}
],
},
"ParseMethod": method,
"TemplateId": template_id or "unknown",
"ReasonCode": reason_code or "ok",
"work_order_id": work_order_id or "",
METRIC_NAME: 1,
}
print(json.dumps(emf))
emit_parse_outcome(
METRIC_NAMESPACE,
method,
template_id,
reason_code,
"work_order_id",
work_order_id,
)
def save_work_order(parsed: dict, s3_key: str):

View file

@ -1 +1,2 @@
boto3>=1.43.47
# Per-function dependencies. Leave empty if the function uses only boto3 and stdlib.
# boto3 is provided by the Lambda Python runtime (lambda-template.md) — do not vendor it.

View file

@ -1,452 +0,0 @@
"""Fail-closed SES sender authentication (INFRA-107).
SES Email Receiving *prepends* its own trace headers -- including an
``Authentication-Results`` header whose authserv-id is ``amazonses.com`` --
to the top of the raw MIME it writes to S3. Everything below those
prepended headers (the From header, any additional Authentication-Results
copies) is attacker-controlled, so ONLY the topmost Authentication-Results
header is trusted, and only when its authserv-id is ``amazonses.com``.
An email is accepted only when that header carries ``dkim=pass`` for a
domain in the ``ALLOWED_DKIM_DOMAINS`` allowlist (a comma-separated Lambda
environment variable set by the CDK stack). Every other outcome fails
closed and the email is rejected:
- ``ALLOWED_DKIM_DOMAINS`` unset or empty
- no Authentication-Results header at all
- topmost header unparseable or from an authserv-id other than SES
- no ``dkim=pass`` clause
- ``dkim=pass`` only for domains outside the allowlist
Observed SES format (2026-07-15, both ingest buckets)::
Authentication-Results: amazonses.com;
spf=pass (spfCheck: ...) client-ip=...; envelope-from=...; helo=...;
dkim=pass header.i=@seahaven.com;
dmarc=none header.from=hxgnsmartcloud.com;
Note SES reports the passing DKIM identity as ``header.i=@<domain>``
(RFC 6376 AUID), not ``header.d=``; the parser accepts both, and treats
``header.d`` (the plain signing domain) as authoritative over ``header.i``
when both are present. The ``header.i`` domain is derived per RFC 6376 as
the part after the AUID's *last* ``@`` -- a ``@`` inside a quoted
local-part (e.g. ``i="@seahaven.com"@attacker.com``) is signer-controlled
label text, never the identity domain, and yields the true signer
(``attacker.com``).
**Clause injection defence (INFRA-107 hardening).** SES echoes several
attacker-controlled SMTP-session tokens into its own Authentication-Results
value as their own semicolon-delimited property clauses -- notably
``envelope-from=<MAIL FROM>``, ``helo=<HELO>`` and ``header.from=<From
domain>``. An RFC 5321 quoted-local-part MAIL FROM may legally contain
spaces and semicolons, e.g.::
MAIL FROM:<"x; dkim=pass header.i=@amazon.coupahost.com"@attacker.com>
which SES renders verbatim as ``envelope-from="x; dkim=pass
header.i=@amazon.coupahost.com"@attacker.com;``. A naive ``split(";")``
would tear the quoted string apart and manufacture a synthetic
``dkim=pass header.i=@amazon.coupahost.com`` clause out of attacker input.
The parser therefore tokenises per RFC 8601 / RFC 5322 structure: CFWS
comments ``(...)`` are stripped first, and the value is split into clauses
only on semicolons that sit *outside* a quoted-string. A ``;`` inside a
quoted ``pvalue`` stays part of that one property clause and can never be
read as the start of a ``dkim=`` methodspec.
"""
import email.parser
import email.policy
import json
import logging
import os
import re
logger = logging.getLogger()
ALLOWED_DKIM_DOMAINS_ENV = "ALLOWED_DKIM_DOMAINS"
SES_AUTHSERV_ID = "amazonses.com"
# One resinfo clause of an Authentication-Results value, e.g.
# "dkim=pass header.i=@seahaven.com". The clause must START with the
# method=result pair; header.d= / header.i= may appear anywhere after it.
# The result token must be terminated by end-of-clause or whitespace so
# "dkim=pass-anything" can never be read as "pass". (Comments are stripped
# before matching, so the pre-strip "dkim=pass(comment)" form arrives here
# as "dkim=pass ..." and still terminates on whitespace.)
_DKIM_RESULT_RE = re.compile(r"^dkim\s*=\s*([a-z0-9]+)(?=$|\s)", re.IGNORECASE)
def get_allowed_dkim_domains() -> frozenset:
"""Read the DKIM-domain allowlist from the environment (may be empty)."""
raw = os.environ.get(ALLOWED_DKIM_DOMAINS_ENV, "")
return frozenset(
d.strip().lower().lstrip("@").rstrip(".") for d in raw.split(",") if d.strip()
)
def _unfold(value: str) -> str:
"""Collapse RFC 5322 folding whitespace into single spaces."""
return re.sub(r"[\r\n\t ]+", " ", value).strip()
def _strip_comments(value: str) -> tuple:
"""Remove RFC 5322 CFWS comments ``(...)`` from an unfolded header value.
Comments may nest and may contain quoted pairs (``\\)``). A ``(`` that
appears *inside* a quoted-string is literal text, not a comment start,
so quoted-strings (which carry attacker-controlled ``pvalue`` content
such as a quoted MAIL FROM local part) are passed through untouched.
Dropping comments first means comment-embedded ``header.i=`` / ``;``
fakes can never influence clause splitting or domain extraction.
Every removed comment is replaced by a single space so comments act as
folding whitespace (RFC 5322 CFWS semantics). A ``)`` that appears
outside any comment (depth 0) is an unmatched close and makes the value
not well-formed, closing the ``)(`` clause-injection gap.
CFWS also means a comment abutting a token *ends* that token:
``header.d=seahaven.com(note)`` reads as domain ``seahaven.com``, where
the pre-CFWS gluing behaviour read ``seahaven.comnote`` (rejected).
This is deliberate -- ``header.d`` is written verbatim by SES from a
*verified* signature's ``d=`` tag, so a comment in that position is
never attacker-controlled.
Returns ``(stripped_text, well_formed)``. ``well_formed`` is False when
the value ends inside an unterminated comment or quoted-string, or when
an unmatched ``)`` appears at the top level. Callers reject on
``not well_formed`` so a malformed header (which could otherwise be
mis-tokenised) fails closed rather than being partially parsed.
"""
out = []
depth = 0 # comment nesting depth
in_quote = False # inside a quoted-string (only tracked at depth 0)
extra_close = False # SES-AR-01: ")" at depth 0 (no matching open)
i = 0
n = len(value)
while i < n:
c = value[i]
if depth > 0:
# Inside a comment: only quoted-pairs and nested parens matter.
if c == "\\":
i += 2
continue
if c == "(":
depth += 1
elif c == ")":
depth -= 1
if depth == 0:
# SES-AR-02: comment acts as folding whitespace (RFC 5322).
out.append(" ")
i += 1
continue
if in_quote:
out.append(c)
if c == "\\" and i + 1 < n:
out.append(value[i + 1])
i += 2
continue
if c == '"':
in_quote = False
i += 1
continue
# Normal context (outside any comment or quoted-string).
if c == "(":
depth += 1
i += 1
continue
if c == ")":
# SES-AR-01: ")" at depth 0 is an unmatched close -- refuse to
# tokenise rather than letting a ")(...)" pair manufacture a
# gap where attacker text leaks out at depth 0.
extra_close = True
i += 1
continue
if c == '"':
in_quote = True
out.append(c)
i += 1
well_formed = depth == 0 and not in_quote and not extra_close
return "".join(out), well_formed
def _split_clauses(value: str) -> list:
"""Split a comment-free header value into clauses on top-level ``;``.
A semicolon inside a quoted-string is preserved as part of the clause,
so an attacker-controlled quoted ``pvalue`` (e.g. a quoted MAIL FROM
echoed into ``envelope-from=``) cannot smuggle in a fake ``dkim=pass``
clause. Callers must run :func:`_strip_comments` first.
"""
clauses = []
buf = []
in_quote = False
i = 0
n = len(value)
while i < n:
c = value[i]
if in_quote:
buf.append(c)
if c == "\\" and i + 1 < n:
buf.append(value[i + 1])
i += 2
continue
if c == '"':
in_quote = False
i += 1
continue
if c == '"':
in_quote = True
buf.append(c)
i += 1
continue
if c == ";":
clauses.append("".join(buf))
buf = []
i += 1
continue
buf.append(c)
i += 1
clauses.append("".join(buf))
return clauses
def _split_properties(clause: str) -> list:
"""Split a clause into whitespace-separated ``name=pvalue`` tokens.
Whitespace *inside* a quoted-string does not split, so an RFC 8601 pvalue
that embeds a quoted-string (e.g. a DKIM AUID with a quoted local-part that
legally contains spaces) survives as a single token. This is the same
quoted-string discipline :func:`_split_clauses` applies at the ``;`` level,
reused here at the token level so ``header.d=`` / ``header.i=`` extraction
is quoted-string-aware rather than a naive regex grab.
"""
tokens = []
buf = []
in_quote = False
i = 0
n = len(clause)
while i < n:
c = clause[i]
if in_quote:
buf.append(c)
if c == "\\" and i + 1 < n:
buf.append(clause[i + 1])
i += 2
continue
if c == '"':
in_quote = False
i += 1
continue
if c == '"':
in_quote = True
buf.append(c)
i += 1
continue
if c.isspace():
if buf:
tokens.append("".join(buf))
buf = []
i += 1
continue
buf.append(c)
i += 1
if buf:
tokens.append("".join(buf))
return tokens
def _auid_domain(pvalue: str) -> str:
"""Domain of a DKIM AUID (``header.i``) per RFC 6376.
The identity domain is the part after the *last* ``@`` of the AUID -- but a
``@`` inside a quoted local-part is NOT the identity separator. So
``"@seahaven.com"@attacker.com`` yields ``attacker.com`` (the real signer),
not ``seahaven.com``: the ``@seahaven.com`` sits inside the quoted
local-part and is signer-controlled label text, never the domain. A bare
unquoted ``@seahaven.com`` still yields ``seahaven.com``. Returns "" when
there is no top-level ``@`` (no valid domain) or the domain looks malformed.
"""
last_at = -1
in_quote = False
i = 0
n = len(pvalue)
while i < n:
c = pvalue[i]
if in_quote:
if c == "\\":
i += 2
continue
if c == '"':
in_quote = False
i += 1
continue
if c == '"':
in_quote = True
elif c == "@":
last_at = i
i += 1
if last_at < 0:
return ""
domain = pvalue[last_at + 1 :].lower().rstrip(".")
# A DKIM domain-name is a plain dot-atom; anything with a residual quote is
# malformed (or a smuggling attempt) and must not be trusted.
if not domain or '"' in domain:
return ""
return domain
def _plain_domain(pvalue: str) -> str:
"""Domain of ``header.d`` -- the DKIM signing domain, a plain dot-atom.
SES writes ``header.d`` verbatim from the signature's ``d=`` tag, which is
never a quoted-string. A residual quote means malformed/smuggled input and
fails closed.
"""
domain = pvalue.lower().rstrip(".")
if not domain or '"' in domain:
return ""
return domain
def _clause_signer_domain(clause: str) -> str:
"""Authoritative DKIM signer domain of a ``dkim=pass`` clause, or "".
``header.d`` (the signing domain) is authoritative and is preferred when
present; only when it is absent does this fall back to ``header.i`` and
derive the domain from the AUID's post-final-``@`` part. Both lookups run
over :func:`_split_properties` tokens, so a ``header.d=``/``header.i=``
literal smuggled *inside* another property's quoted pvalue is confined to
that one token and can never be read as a top-level property.
"""
header_d = None
header_i = None
for token in _split_properties(clause):
name, sep, val = token.partition("=")
if not sep:
continue
key = name.strip().lower()
if key == "header.d" and header_d is None:
header_d = val
elif key == "header.i" and header_i is None:
header_i = val
if header_d is not None:
return _plain_domain(header_d)
if header_i is not None:
return _auid_domain(header_i)
return ""
def parse_authentication_results(value: str) -> tuple:
"""Parse one Authentication-Results header value.
Returns ``(authserv_id, passing_dkim_domains)`` where the domains are the
authoritative signer domains of every ``dkim=pass`` clause -- ``header.d``
when present, else the ``header.i`` AUID's post-final-``@`` domain (see
:func:`_clause_signer_domain`). Malformed input yields ``("", frozenset())``,
which callers treat as a rejection.
Tokenisation is comment- and quoted-string-aware (RFC 8601 / RFC 5322) at
the ``;`` (clause), whitespace (property) and ``@`` (AUID domain) levels, so
attacker-controlled tokens SES echoes into its header (envelope-from, helo,
header.from) cannot be split into a forged ``dkim=pass`` clause and a quoted
AUID local-part cannot masquerade as the identity domain.
"""
text, well_formed = _strip_comments(_unfold(value))
if not well_formed:
# Unbalanced quotes/comments: refuse to guess how to tokenise it.
return "", frozenset()
clauses = [c.strip() for c in _split_clauses(text)]
if not clauses or not clauses[0]:
return "", frozenset()
# First clause is the authserv-id, optionally followed by a version
# token ("amazonses.com 1"); take only the first token.
authserv_id = clauses[0].split()[0].strip('"').lower()
passing = set()
for clause in clauses[1:]:
match = _DKIM_RESULT_RE.match(clause)
if not match or match.group(1).lower() != "pass":
continue
domain = _clause_signer_domain(clause)
if domain:
passing.add(domain)
return authserv_id, frozenset(passing)
def evaluate_sender_authentication(raw_email: bytes, allowed_domains) -> tuple:
"""Evaluate the SES-stamped verdicts in a raw MIME message.
Returns ``(accepted, reason, detail)``. Pure function of its inputs so
it can be unit-tested without touching the environment.
"""
if not allowed_domains:
return False, "allowlist_not_configured", {}
try:
# compat32 keeps header values as raw strings (we unfold ourselves)
# and never raises on structurally odd headers; headersonly avoids
# parsing the body at all.
msg = email.parser.BytesParser(policy=email.policy.compat32).parsebytes(
raw_email, headersonly=True
)
except Exception:
return False, "unparseable_message", {}
ar_headers = msg.get_all("Authentication-Results") or []
if not ar_headers:
return False, "authentication_results_missing", {}
# SES prepends its trace headers, so index 0 is the SES-stamped copy.
# Any Authentication-Results header further down arrived inside the
# message (attacker-suppliable) and is deliberately ignored.
#
# The parse is wrapped so an unexpected parser exception fails CLOSED
# (rejected, structured reason) instead of propagating out of the handler
# into Lambda's async retries / DLQ on attacker-crafted input.
try:
authserv_id, passing = parse_authentication_results(str(ar_headers[0]))
except Exception:
return False, "authentication_results_unparseable", {}
detail = {
"authserv_id": authserv_id,
"passing_dkim_domains": sorted(passing),
}
if authserv_id != SES_AUTHSERV_ID:
return False, "untrusted_authserv_id", detail
if not passing:
return False, "no_passing_dkim_signature", detail
matched = passing & set(allowed_domains)
if not matched:
return False, "dkim_domain_not_allowlisted", detail
detail["matched_domains"] = sorted(matched)
return True, "authenticated", detail
def authenticate_inbound_email(raw_email: bytes, s3_key: str) -> bool:
"""Fail-closed gate used by the S3-triggered handlers.
On rejection: logs a structured warning with the reason and S3 key and
returns False. Callers skip the message and return normally, so
rejected mail never errors the invocation (no retries, no DLQ spam).
"""
allowed = get_allowed_dkim_domains()
accepted, reason, detail = evaluate_sender_authentication(raw_email, allowed)
if accepted:
logger.info(json.dumps({"event": "sender_auth_ok", "s3_key": s3_key, **detail}))
return True
logger.warning(
json.dumps(
{
"event": "sender_auth_rejected",
"reason": reason,
"s3_key": s3_key,
"allowed_dkim_domains": sorted(allowed),
**detail,
}
)
)
return False

View file

@ -16,6 +16,14 @@ _HERE = os.path.dirname(__file__)
_MODULE_DIR = os.path.abspath(os.path.join(_HERE, ".."))
if _MODULE_DIR not in sys.path:
sys.path.insert(0, _MODULE_DIR)
# Phase 3: handler.py now imports ses_auth/email_parsing/emf, single-sourced
# under lambdas/shared/ (no longer in _MODULE_DIR). Put shared/ on sys.path so
# the handler's bare `from email_parsing import ...` etc. resolve. These modules
# are single-copy, so bare-name caching is correct for both pipelines -- no
# collision guard needed (that only matters for the duplicated template_parser).
_SHARED_DIR = os.path.abspath(os.path.join(_HERE, "..", "..", "..", "shared"))
if _SHARED_DIR not in sys.path:
sys.path.insert(0, _SHARED_DIR)
os.environ.setdefault("AWS_DEFAULT_REGION", "us-east-1")
FIXTURES = os.path.join(_HERE, "fixtures")

View file

@ -5,14 +5,13 @@ Serves a simple HTML dashboard for viewing work orders and comments.
Accessed via Lambda Function URL.
"""
import hmac
import json
import logging
import os
import time
from html import escape as esc
import boto3
from web_ui_auth import is_authenticated
logger = logging.getLogger()
logger.setLevel(logging.INFO)
@ -29,73 +28,6 @@ COMMENTS_TABLE = os.environ.get("COMMENTS_TABLE", "WorkOrderComments")
# fetched at runtime from Secrets Manager to keep it out of CloudFormation
# templates and Lambda env vars. If the ARN is unset or the secret is missing, the
# handler fails closed and denies every request.
_WEB_UI_AUTH_TOKEN_SECRET_ARN = os.environ.get("WEB_UI_AUTH_TOKEN_SECRET_ARN")
# Refresh the cached token this often so a rotated secret propagates without
# waiting for the execution environment to recycle (emergency-rotation path).
_AUTH_TOKEN_CACHE_TTL_SECONDS = 300
_auth_token_cache = None
_auth_token_cached_at = 0.0
def _get_auth_token() -> str | None:
"""Fetch the shared web UI auth token from Secrets Manager.
Cached in the warm container for a short TTL so we don't hit Secrets Manager
on every request, while still picking up a rotated secret within the TTL
rather than only when the execution environment recycles. Returns None when
not configured or unreadable (the caller then fails closed).
"""
global _auth_token_cache, _auth_token_cached_at
now = time.monotonic()
if (
_auth_token_cache is not None
and now - _auth_token_cached_at < _AUTH_TOKEN_CACHE_TTL_SECONDS
):
return _auth_token_cache
if not _WEB_UI_AUTH_TOKEN_SECRET_ARN:
return None
secrets = boto3.client("secretsmanager")
try:
secret = secrets.get_secret_value(SecretId=_WEB_UI_AUTH_TOKEN_SECRET_ARN)
_auth_token_cache = secret["SecretString"]
_auth_token_cached_at = now
return _auth_token_cache
except Exception:
# Fail closed (return None -> caller 401s) but surface the failure: a
# Secrets Manager permission/config error would otherwise make every
# request 401 with no operational signal. The secret value is never
# logged.
logger.exception(
"Failed to fetch web UI auth token from Secrets Manager; "
"denying request (failing closed)"
)
return None
def _header(event: dict, name: str) -> str:
"""Case-insensitive header lookup from a Lambda Function URL / APIGW event."""
headers = event.get("headers") or {}
name_lower = name.lower()
for key, value in headers.items():
if key.lower() == name_lower:
return value or ""
return ""
def is_authenticated(event: dict) -> bool:
"""Constant-time check of the request's shared secret against the configured
token. Fails closed when no token is configured."""
token = _get_auth_token()
if not token:
return False
presented = _header(event, "x-auth-token")
if not presented:
auth = _header(event, "authorization")
if auth.lower().startswith("bearer "):
presented = auth[7:].strip()
if not presented:
return False
return hmac.compare_digest(presented, token)
def get_work_orders(limit=500):

View file

@ -1,39 +1,93 @@
"""
Re-invoke the po-email-processor Lambda for every object still
sitting under the inbound/ prefix in the email bucket.
Reprocess missed procurement-ingest emails by re-invoking an email-processor
Lambda with a synthetic S3 event.
Usage:
python scripts/reprocess.py # dry-run (list only)
python scripts/reprocess.py --execute # actually invoke
TARGETED replay is the default and preferred mode: point it at exactly the
object(s) you need to replay.
# dry-run a single object (default is dry-run — lists, does not invoke)
python scripts/reprocess.py --pipeline po --key inbound/2026/msg.eml
# actually re-invoke that one object
python scripts/reprocess.py --pipeline po --key inbound/2026/msg.eml --execute
# replay a prefix, or everything modified since a timestamp
python scripts/reprocess.py --pipeline wo --prefix inbound/2026/07/
python scripts/reprocess.py --pipeline po --since 2026-07-16T00:00:00Z --execute
FULL-PREFIX sweep is DEMOTED behind an explicit --all flag and carries five
load-bearing caveats (see the --all help text and the in-code comment above the
sweep branch):
python scripts/reprocess.py --pipeline po --all --execute # replays ALL of inbound/
1. Event-type (async) invocation is CONCURRENT, so sorting does NOT
serialize the replay.
2. Switch InvocationType to "RequestResponse" if ordering matters.
3. Metrics get double-counted on replay.
4. Bedrock is re-billed for every ai_fallback email replayed.
5. Out-of-order replay can REGRESS already-merged fields.
Dry-run is the default in ALL modes (targeted and --all alike); nothing is
invoked unless --execute is passed.
"""
import argparse
import json
from datetime import datetime, timezone
import boto3
sts = boto3.client("sts")
s3 = boto3.client("s3")
lambda_client = boto3.client("lambda")
# Per-pipeline resolution: the function to invoke and the raw-email bucket to
# list, keyed by --pipeline. Both are derived at runtime from --pipeline +
# the caller's account id; there is no PO-hardcoded default.
PIPELINES = {
"po": {"function": "po-email-processor", "bucket": "po-ingest-emails-{acct}"},
"wo": {
"function": "workorder-email-processor",
"bucket": "workorder-ingest-emails-{acct}",
},
}
FUNCTION_NAME = "po-email-processor"
DEFAULT_PREFIX = "inbound/"
def get_bucket_name() -> str:
account_id = sts.get_caller_identity()["Account"]
return f"po-ingest-emails-{account_id}"
def parse_since(value: str) -> datetime:
"""Parse an ISO-8601 UTC timestamp (accepts a trailing 'Z') into an
aware UTC datetime so it can be compared against S3 LastModified."""
normalized = value.replace("Z", "+00:00")
dt = datetime.fromisoformat(normalized)
if dt.tzinfo is None:
dt = dt.replace(tzinfo=timezone.utc)
return dt.astimezone(timezone.utc)
def list_inbound_keys(bucket: str) -> list[str]:
keys = []
def list_keys(s3, bucket: str, prefix: str, since: datetime | None) -> list[str]:
"""Paginate list_objects_v2 under prefix, returning raw object keys.
When since is set, only objects with LastModified >= since are kept.
The keys returned are the RAW list_objects_v2 obj["Key"] — no
URL-encoding or decoding is applied (see build_s3_event).
"""
keys: list[str] = []
paginator = s3.get_paginator("list_objects_v2")
for page in paginator.paginate(Bucket=bucket, Prefix="inbound/"):
for page in paginator.paginate(Bucket=bucket, Prefix=prefix):
for obj in page.get("Contents", []):
if since is not None and obj["LastModified"] < since:
continue
keys.append(obj["Key"])
return keys
def build_s3_event(bucket: str, key: str) -> dict:
"""Build the synthetic S3 notification event the handler consumes.
CONTRACT (pinned by tests/test_reprocess_contract.py): the shape is
exactly Records[0].s3.bucket.name / Records[0].s3.object.key, and the
key is emitted RAW — reprocess applies NO urllib.parse.quote/unquote.
A real S3 notification URL-encodes the object key; reprocess builds from
the raw list_objects_v2 key (or the raw --key arg), and the handler is
what would decode. Replaying a pre-decoded/pre-encoded key would not
match what s3.get_object(Bucket, Key) expects.
"""
return {
"Records": [
{
@ -47,28 +101,134 @@ def build_s3_event(bucket: str, key: str) -> dict:
def main():
parser = argparse.ArgumentParser(description="Reprocess missed PO emails")
parser = argparse.ArgumentParser(
description=(
"Reprocess missed procurement-ingest emails by re-invoking an "
"email-processor Lambda with a synthetic S3 event. TARGETED replay "
"(--key/--prefix/--since) is the default and preferred mode; "
"full-prefix sweep requires the explicit --all flag."
)
)
parser.add_argument(
"--pipeline",
required=True,
choices=["po", "wo"],
help=(
"Which pipeline to replay: 'po' (po-email-processor / "
"po-ingest-emails-<acct>) or 'wo' (workorder-email-processor / "
"workorder-ingest-emails-<acct>)."
),
)
# Targeting selectors — at least one of {--key, --prefix, --since, --all}
# is required (enforced below); --key is exclusive.
parser.add_argument(
"--key",
help="Replay exactly one raw object key (e.g. inbound/2026/msg.eml).",
)
parser.add_argument(
"--prefix",
help=(
"Replay every object under this S3 prefix "
"(default targeting prefix is inbound/)."
),
)
parser.add_argument(
"--since",
help=(
"Only objects with LastModified >= this ISO-8601 UTC timestamp "
"(e.g. 2026-07-16T00:00:00Z). Combines with --prefix."
),
)
parser.add_argument(
"--all",
action="store_true",
help=(
"DEMOTED full-prefix sweep: replay EVERY object under inbound/ "
"(not a targeted subset). "
"CAVEATS: (1) Event-type (async) invocation is CONCURRENT so "
"sorting does NOT serialize the replay; "
"(2) switch to RequestResponse if order matters; "
"(3) metrics get double-counted on replay; "
"(4) Bedrock is re-billed; "
"(5) out-of-order replay can REGRESS already-merged fields. "
"Prefer --key/--prefix/--since. Still dry-run unless --execute."
),
)
parser.add_argument(
"--execute",
action="store_true",
help="Actually invoke the Lambda (default is dry-run)",
help=(
"Actually invoke the Lambda. Default is dry-run (list only). "
"Applies to EVERY mode, including --all."
),
)
args = parser.parse_args()
bucket = get_bucket_name()
keys = list_inbound_keys(bucket)
# Exactly-one target gate: full sweep is ONLY reachable via --all, so an
# accidental whole-prefix sweep from running with no selector is impossible.
if not any([args.key, args.prefix, args.since, args.all]):
parser.error("choose a target: --key, --prefix, --since, or --all (full sweep)")
# --key is mutually exclusive with the listing selectors and with --all.
if args.key and (args.prefix or args.since or args.all):
parser.error("--key cannot be combined with --prefix, --since, or --all")
# --all is a distinct MODE (the demoted full-prefix sweep), not a modifier:
# reject it alongside the targeting selectors. Without this, the args.all
# clobber branch below would silently discard a narrower --prefix/--since
# (e.g. `--all --since 2026-07-01` would sweep the ENTIRE corpus, not the
# bounded window) and trigger every --all hazard on unintended objects.
if args.all and (args.prefix or args.since):
parser.error("--all cannot be combined with --prefix or --since")
account_id = boto3.client("sts").get_caller_identity()["Account"]
function_name = PIPELINES[args.pipeline]["function"]
bucket = PIPELINES[args.pipeline]["bucket"].format(acct=account_id)
if args.key:
# Single raw key — skip listing entirely. The key is passed through
# to build_s3_event untransformed (raw-key contract).
keys = [args.key]
else:
s3 = boto3.client("s3")
# --all is the DEMOTED full-prefix sweep. Five caveats, all load-bearing:
# 1. Event-type (async) invocation is CONCURRENT, so sorting the key
# list does NOT serialize the replay.
# 2. If ordering matters, switch InvocationType to "RequestResponse"
# (serial, waits per call).
# 3. Metrics (ParseOutcome EMF, alarms) get double-counted on replay.
# 4. Bedrock is re-billed for every ai_fallback email replayed.
# 5. Out-of-order replay can REGRESS already-merged fields (a stale
# email overwriting a newer merge).
# Targeted replay (--key / --prefix / --since) avoids all five at
# whole-corpus scale; prefer it.
if args.all:
prefix = DEFAULT_PREFIX
since = None
else:
# --prefix / --since target. --since without --prefix defaults the
# prefix to inbound/ so it filters the raw-email tree, not the bucket.
prefix = args.prefix if args.prefix else DEFAULT_PREFIX
since = parse_since(args.since) if args.since else None
keys = list_keys(s3, bucket, prefix, since)
if not keys:
print("No objects found under inbound/ — nothing to reprocess.")
print(f"No matching objects in s3://{bucket}/ — nothing to reprocess.")
return
print(f"Found {len(keys)} email(s) in s3://{bucket}/inbound/\n")
if args.all:
print(
f"[--all full sweep] {len(keys)} email(s) under "
f"s3://{bucket}/{DEFAULT_PREFIX}\n"
)
else:
print(f"{len(keys)} email(s) selected in s3://{bucket}/\n")
lambda_client = boto3.client("lambda") if args.execute else None
for key in keys:
if args.execute:
print(f" Invoking for {key} ... ", end="", flush=True)
print(f" Invoking {function_name} for {key} ... ", end="", flush=True)
resp = lambda_client.invoke(
FunctionName=FUNCTION_NAME,
FunctionName=function_name,
InvocationType="Event", # async — don't wait for each one
Payload=json.dumps(build_s3_event(bucket, key)),
)

View file

@ -19,15 +19,25 @@ os.environ.setdefault("AWS_SECRET_ACCESS_KEY", "testing")
os.environ.setdefault("AWS_SESSION_TOKEN", "testing")
REPO_ROOT = Path(__file__).resolve().parents[1]
_SHARED_DIR = REPO_ROOT / "lambdas" / "shared"
# Sibling modules that exist per-pipeline and are imported by bare name from
# the handlers (the Lambda runtime puts each function's own directory on
# sys.path). Both pipelines duplicate these filenames, so the bare names MUST
# be bound to the right pipeline's file around each handler exec -- relying on
# sys.path ordering (or on whatever a previously collected suite left in
# sys.modules) silently binds a handler to the OTHER pipeline's sibling.
_SIBLING_MODULES = ("ses_auth", "template_parser", "derived_fields")
# Sibling modules imported by bare name from the handlers (the Lambda runtime
# puts each function's own directory on sys.path; the CDK bundling then cp's the
# shared modules in flat beside handler.py so those bare imports resolve too).
# template_parser/derived_fields are duplicated PER PIPELINE, so their bare
# names MUST be bound to the right pipeline's file around each handler exec --
# relying on sys.path ordering (or on whatever a previously collected suite left
# in sys.modules) silently binds a handler to the OTHER pipeline's sibling.
# ses_auth/email_parsing/emf are now single-sourced under lambdas/shared/ (Phase
# 3); the loop below resolves them from there via a shared-dir fallback.
_SIBLING_MODULES = (
"ses_auth",
"template_parser",
"derived_fields",
"email_parsing",
"emf",
)
def _load_module(path, module_name):
@ -69,7 +79,14 @@ def load_handler(relative_path, module_name):
return _load_module(path, module_name)
saved = {}
for sibling in _SIBLING_MODULES:
# Per-pipeline siblings (template_parser/derived_fields) resolve next to
# the handler; the shared, single-sourced siblings (ses_auth/
# email_parsing/emf) fall back to lambdas/shared/. No ambiguity: post
# Phase 3 the shared names exist ONLY under shared/, the per-pipeline
# names ONLY next to the handler.
sibling_path = path.parent / f"{sibling}.py"
if not sibling_path.exists():
sibling_path = _SHARED_DIR / f"{sibling}.py"
if not sibling_path.exists():
continue
saved[sibling] = sys.modules.get(sibling)
@ -109,11 +126,12 @@ def email_handler(request):
return request.getfixturevalue(request.param)
@pytest.fixture(params=["wo", "po"])
def ses_auth(request):
"""The ses_auth module of each pipeline (duplicated file, kept in sync)."""
pipeline = request.param
return load_handler(
f"lambdas/{pipeline}/email_processor/ses_auth.py",
f"{pipeline}_ses_auth",
)
@pytest.fixture
def ses_auth():
"""The single-sourced ses_auth module (lambdas/shared/, Phase 3).
Previously parameterized over the two per-pipeline copies to prove they
stayed byte-identical; now there is exactly one copy, so this loads it
once -- halving the test_ses_auth run.
"""
return load_handler("lambdas/shared/ses_auth.py", "shared_ses_auth")

View file

@ -1 +1 @@
moto>=5.0.0
moto==5.2.2

View file

@ -23,16 +23,45 @@ PO_HANDLER = REPO_ROOT / "lambdas" / "po" / "email_processor" / "handler.py"
WO_HANDLER = REPO_ROOT / "lambdas" / "wo" / "email_processor" / "handler.py"
PO_STACK = REPO_ROOT / "cdk" / "po_stack.py"
WO_STACK = REPO_ROOT / "cdk" / "wo_stack.py"
# Phase 3: ses_auth/web_ui_auth/email_parsing/emf are single-sourced here and
# shipped into the email-processor bundles via `cp shared/*.py`.
SHARED_DIR = REPO_ROOT / "lambdas" / "shared"
# The complete set of top-level modules the PO email_processor ships via the
# non-recursive `cp ./*.py` glob. Pinned as an upper bound: a new top-level
# .py in that dir must be added here deliberately, which is the moment to
# decide whether it SHOULD ship (a real module) or must be excluded (a scratch
# or secrets file that from_asset would otherwise stage into the bundle on a
# local deploy). See test_po_bundle_ships_no_unexpected_top_level_modules.
PO_EXPECTED_TOP_LEVEL_MODULES = frozenset(
{"handler", "ses_auth", "template_parser", "derived_fields"}
)
# The names Phase 3 single-sourced under lambdas/shared/. After the move NONE
# of these may reappear as a top-level .py in either email-processor pipeline
# dir: every handler loader resolves a pipeline-local copy FIRST
# (tests/conftest.py load_handler checks path.parent before _SHARED_DIR;
# lambdas/wo/email_processor/tests/_wo_parser_support.py inserts the pipeline
# dir ahead of shared/ on sys.path), so a stray reappearance would silently
# SHADOW the single shared source in every handler-loaded test while
# test_ses_auth / test_parse_raw_email keep exercising shared/ -- divergence
# undetected. This is exactly the future-drift invariant the retired
# byte-identity ses_auth fixture used to guard; it is now enforced by policing
# the pipeline dirs and shared/ SEPARATELY (never as a union, which would let
# the same name already expected in shared/ mask a pipeline-dir stray) --
# see test_no_shared_module_shadow_in_pipeline_dirs and the per-dir exact-set
# pins below.
SHARED_MODULES = frozenset({"ses_auth", "email_parsing", "emf", "web_ui_auth"})
# Exact top-level .py stems each dir must hold. Pinned as exact sets (both
# bounds): the bundling globs (`cp <pipeline>/email_processor/*.py` +
# `cp shared/*.py`) ship every top-level .py in these dirs, and
# `Code.from_asset` stages untracked files too (it does not honor .gitignore),
# so a stray scratch/secrets .py -- or a shadow copy of a shared module -- would
# silently ship into the production zip on a local deploy. Any new top-level
# module must be added here deliberately, the moment to decide whether it SHOULD
# ship (a real module) or must be excluded.
PO_PIPELINE_MODULES = frozenset({"handler", "template_parser", "derived_fields"})
WO_PIPELINE_MODULES = frozenset({"__init__", "handler", "template_parser"})
# Full shipped set of each email-processor bundle (pipeline glob + shared glob).
# Derived from the per-dir pins above so it stays consistent with them; policed
# per-dir (NOT via this union) so a shared-name shadow in a pipeline dir cannot
# be masked. web_ui_auth is a deliberate, harmless ride-along of the pinned
# `cp shared/*.py` (constraint #8) -- never imported by the email handlers, but
# it ships, so it is part of the shipped set.
PO_EXPECTED_TOP_LEVEL_MODULES = PO_PIPELINE_MODULES | SHARED_MODULES
WO_EXPECTED_TOP_LEVEL_MODULES = WO_PIPELINE_MODULES | SHARED_MODULES
# Pipeline-scoped glob shapes the per-stack ships-all pins accept. Phase 2
# widened the bundling cwd to the shared ../lambdas asset root, so the executed
@ -47,6 +76,22 @@ PO_EXPECTED_TOP_LEVEL_MODULES = frozenset(
PO_SCOPED_GLOB_RE = r"(?:^|\s)(?:\./|po/email_processor/)?\*\.py(?:\s|$)"
WO_SCOPED_GLOB_RE = r"(?:^|\s)(?:\./|wo/email_processor/)?\*\.py(?:\s|$)"
# Phase 3: both email-processor bundles gain a second glob copying the
# single-sourced shared modules flat into the zip. This must be the EXECUTED
# form (`_executed_cp_commands` strips comments), so a commented-out shared cp
# cannot false-pass the pin.
SHARED_CP_RE = r"(?:^|\s)shared/\*\.py(?:\s|$)"
# Phase 3: each stack's web_ui function stages ONLY shared/web_ui_auth.py flat
# beside its handler (NOT all of shared/ -- the email-only modules must not
# bloat the web UI zip). The auth-gated web_ui handlers do a module-top-level
# `from web_ui_auth import is_authenticated`, so dropping/commenting this cp
# ImportErrors the Lambda at cold start with green CI -- the PR #105 ImportError
# class the AST test exists to prevent, but for a surface (web_ui) the
# email-processor selector deliberately excludes. Checked as the EXECUTED form
# so a commented-out cp cannot false-pass.
WEB_UI_AUTH_CP_RE = r"(?:^|\s)shared/web_ui_auth\.py(?:\s|$)"
def _first_party_sibling_imports(handler_path: Path) -> set[str]:
"""Top-level module names handler.py imports that are first-party siblings.
@ -67,7 +112,16 @@ def _first_party_sibling_imports(handler_path: Path) -> set[str]:
if node.module:
names.add(node.module.split(".")[0])
sibling_dir = handler_path.parent
return {name for name in names if (sibling_dir / f"{name}.py").exists()}
# A first-party sibling resolves either next to the handler (per-pipeline:
# template_parser/derived_fields) or under lambdas/shared/ (single-sourced:
# ses_auth/email_parsing/emf, Phase 3). Both must count, or the ships-all
# pin would silently drop the shared siblings (ses_auth was silently
# dropped, email_parsing/emf never seen, before this resolved shared/).
return {
name
for name in names
if (sibling_dir / f"{name}.py").exists() or (SHARED_DIR / f"{name}.py").exists()
}
def _extract_bundling_command(stack_path: Path) -> str:
@ -79,11 +133,13 @@ def _extract_bundling_command(stack_path: Path) -> str:
lines, with `#` comments interspersed) is folded by the parser itself,
so this returns the exact single command string CDK will hand to bash.
Each stack currently has exactly one bundled function; if a second one
is ever added, "the" bundling command becomes ambiguous and every
assertion in this file needs to pick its target explicitly — so demand
exactly one match rather than silently returning whichever ast.walk
happens to visit first.
Since Phase 3 each stack has TWO bundled functions -- the email processor
and the web_ui function (which now also stages a shared module). "The"
bundling command this test cares about is the EMAIL-processor one, so we
select the command whose text mentions ``email_processor`` (its first cp
is ``cp <pipeline>/email_processor/*.py``) and demand exactly one match --
the web_ui command (``cp -r <pipeline>/web_ui/.``) and site_extractor do
not match.
"""
tree = ast.parse(stack_path.read_text())
commands: list[str] = []
@ -94,13 +150,47 @@ def _extract_bundling_command(stack_path: Path) -> str:
last = list_node.elts[-1]
if isinstance(last, ast.Constant) and isinstance(last.value, str):
commands.append(last.value)
if len(commands) != 1:
email_cmds = [c for c in commands if "email_processor" in c]
if len(email_cmds) != 1:
raise AssertionError(
f"expected exactly one bundling command=[...] list in {stack_path}, "
f"found {len(commands)} — update this test to select the intended "
"bundling command explicitly"
f"expected exactly one email-processor bundling command=[...] list in "
f"{stack_path}, found {len(email_cmds)} (of {len(commands)} total "
"command lists) — update this test to select the intended bundling "
"command explicitly"
)
return commands[0]
return email_cmds[0]
def _extract_web_ui_command(stack_path: Path) -> str:
"""Extract the web_ui function's bash -c bundling command string.
Mirrors _extract_bundling_command but selects the command whose text
mentions ``web_ui`` (its first cp is ``cp -r <pipeline>/web_ui/.``). The
email-processor command (``cp <pipeline>/email_processor/*.py``, plus
``cp shared/*.py``) and site_extractor do not contain ``web_ui`` in the
folded command STRING (the explanatory ``# ... web_ui_auth ...`` comments
around the email command are Python comments, not string content, so they
are not part of the folded literal). Demands exactly one match so an
ambiguity surfaces loudly instead of silently checking the wrong command.
"""
tree = ast.parse(stack_path.read_text())
commands: list[str] = []
for node in ast.walk(tree):
if isinstance(node, ast.keyword) and node.arg == "command":
list_node = node.value
if isinstance(list_node, ast.List) and list_node.elts:
last = list_node.elts[-1]
if isinstance(last, ast.Constant) and isinstance(last.value, str):
commands.append(last.value)
web_ui_cmds = [c for c in commands if "web_ui" in c]
if len(web_ui_cmds) != 1:
raise AssertionError(
f"expected exactly one web_ui bundling command=[...] list in "
f"{stack_path}, found {len(web_ui_cmds)} (of {len(commands)} total "
"command lists) — update this test to select the intended bundling "
"command explicitly"
)
return web_ui_cmds[0]
def _executed_cp_commands(command: str) -> list[str]:
@ -127,39 +217,38 @@ def _bundling_ships_all(command: str, sibling_names: set[str]) -> bool:
"""True if `command` is guaranteed to ship every name in `sibling_names`.
Inspects only the `cp` commands bash actually executes (comments stripped
via `_executed_cp_commands`). An executed copy ships every top-level .py
sibling unconditionally in two shapes:
- a non-recursive glob copy whose (optional) path prefix resolves, under
the ../lambdas bundling cwd, to a directory that ACTUALLY contains
every required sibling, e.g. `cp po/email_processor/*.py /asset-output/`
(PO, Phase 2). The directory is filesystem-checked, not merely
pattern-matched: a wrong-pipeline or arbitrary-directory glob
(`cp wo/email_processor/*.py` in po_stack, `cp venv/lib/*.py`) does NOT
qualify, because that directory does not hold every required sibling --
so it can never short-circuit to True while dropping a module;
via `_executed_cp_commands`) and ACCUMULATES the set of shipped module
stems across ALL of them -- because since Phase 3 the required siblings
ship via TWO globs, not one: `cp <pipeline>/email_processor/*.py` covers
the per-pipeline siblings and `cp shared/*.py` covers the single-sourced
ones (ses_auth/email_parsing/emf). A single-cp "one glob ships everything"
check would wrongly report False now that coverage is split.
Each executed cp contributes to the shipped set as follows:
- a non-recursive `*.py` glob ships every top-level .py in the globbed
directory -- but ONLY that directory. Its (optional) path prefix is
resolved against the ../lambdas asset root (the Phase 2 bundling cwd)
and every `.py` stem actually present there is added. A wrong-pipeline
or arbitrary-directory glob (`cp wo/email_processor/*.py` in po_stack,
`cp venv/lib/*.py`) therefore contributes only what that dir really
holds (or nothing, if it does not exist) and can never cover a sibling
that lives elsewhere;
- a recursive copy of the WHOLE source dir, e.g. `cp -r . /asset-output/`
-- the source operand must be `.`/`./`, so a narrowed recursive
copy like `cp -r ./package /asset-output/` does NOT qualify.
Any other executed `cp` is treated as an explicit filename allowlist (the
legacy PO form) and is safe only if every required `<name>.py` literally
appears among the copied filenames -- the branch that must reject a
reverted allowlist missing a sibling.
(`.`/`./` source only), ships everything -> short-circuit True;
- any other executed `cp` is an explicit filename allowlist (the legacy
PO form): each copied `<name>.py` stem is added, so a reverted
allowlist missing a sibling leaves that sibling out of the set.
Returns True only if every required sibling ended up in the accumulated set.
"""
cp_cmds = _executed_cp_commands(command)
if not cp_cmds:
return False
copied_files: set[str] = set()
shipped: set[str] = set()
for cp in cp_cmds:
glob_match = re.search(r"(?:^|\s)((?:[\w./-]+/)?)\*\.py(?:\s|$)", cp)
if glob_match:
# A `*.py` glob ships every top-level .py in the globbed directory
# -- but ONLY that directory. Resolve its prefix against the
# ../lambdas asset root (the Phase 2 bundling cwd) and confirm it
# actually holds every required sibling, so a wrong-pipeline glob
# cannot pass the ships-all check on the mere presence of a `*.py`.
glob_dir = (REPO_ROOT / "lambdas" / glob_match.group(1)).resolve()
if all((glob_dir / f"{name}.py").exists() for name in sibling_names):
return True
shipped.update(p.stem for p in glob_dir.glob("*.py"))
continue
if re.search(r"cp\s+-r\s+\.\/?\s+/asset-output", cp):
return True
@ -169,8 +258,8 @@ def _bundling_ships_all(command: str, sibling_names: set[str]) -> bool:
r"cp\s+(?:-\S+\s+)*(.*?)\s*/asset-output/?\"?\s*$", cp.strip()
)
if match:
copied_files.update(match.group(1).split())
return all(f"{name}.py" in copied_files for name in sibling_names)
shipped.update(Path(f).stem for f in match.group(1).split())
return all(name in shipped for name in sibling_names)
def test_po_bundling_ships_all_first_party_siblings():
@ -194,6 +283,16 @@ def test_po_bundling_ships_all_first_party_siblings():
"arbitrary-directory glob is rejected here on purpose. "
f"Executed cp commands: {executed_cps}"
)
# Phase 3: the shared siblings (ses_auth/email_parsing/emf) ship via a
# SECOND executed glob, `cp shared/*.py /asset-output/`. Require it to be
# actually executed (comment-stripped), so commenting it out fails here;
# combined with ships-all below (those siblings resolve only under shared/)
# a removed/commented shared cp fails BOTH assertions.
assert any(re.search(SHARED_CP_RE, cp) for cp in executed_cps), (
"cdk/po_stack.py email-processor bundling command must EXECUTE "
"'cp shared/*.py /asset-output/' so the single-sourced shared modules "
f"ship flat beside handler.py. Executed cp commands: {executed_cps}"
)
assert _bundling_ships_all(command, siblings), (
f"cdk/po_stack.py bundling command does not ship all of {sorted(siblings)}: "
f"{command!r}"
@ -221,36 +320,115 @@ def test_wo_bundling_ships_all_first_party_siblings():
"wrong-pipeline or arbitrary-directory glob is rejected here on purpose. "
f"Executed cp commands: {executed_cps}"
)
# Phase 3: shared siblings ship via the second executed glob `cp shared/*.py`.
assert any(re.search(SHARED_CP_RE, cp) for cp in executed_cps), (
"cdk/wo_stack.py email-processor bundling command must EXECUTE "
"'cp shared/*.py /asset-output/' so the single-sourced shared modules "
f"ship flat beside handler.py. Executed cp commands: {executed_cps}"
)
assert _bundling_ships_all(command, siblings), (
f"cdk/wo_stack.py bundling command does not ship all of {sorted(siblings)}: "
f"{command!r}"
)
def test_po_bundle_ships_no_unexpected_top_level_modules():
"""Upper bound: the PO glob ships EXACTLY the expected top-level modules.
def test_po_email_processor_dir_ships_no_unexpected_top_level_modules():
"""Upper bound on the PO pipeline dir alone (NOT unioned with shared/).
The sibling-import tests above prove the glob ships every module the
handler needs (shipped >= required). This proves the other direction
(shipped <= expected): because `cp ./*.py` copies every top-level .py in
the dir -- and `Code.from_asset` stages untracked files too (it does not
honor .gitignore) -- a stray scratch/secrets .py left in this dir would
silently ship into the production zip on a local deploy. Pinning the set
forces any new top-level module to be added to
PO_EXPECTED_TOP_LEVEL_MODULES deliberately, at which point the author
decides whether it should ship or be excluded from bundling.
handler needs (shipped >= required). This proves the other direction for
the pipeline dir (shipped <= expected): because `cp po/email_processor/*.py`
copies every top-level .py in that dir -- and `Code.from_asset` stages
untracked files too (it does not honor .gitignore) -- a stray scratch or
secrets .py, OR a shadow copy of a moved shared module, would silently ship
into the zip on a local deploy. Policing this dir SEPARATELY from shared/
(rather than as a union with it) is what makes a strayed-back ses_auth.py /
email_parsing.py / emf.py FAIL here instead of being masked by the same name
already being expected in shared/.
"""
top_level = {p.stem for p in PO_HANDLER.parent.glob("*.py")}
assert top_level == set(PO_EXPECTED_TOP_LEVEL_MODULES), (
assert top_level == set(PO_PIPELINE_MODULES), (
"unexpected top-level .py set in lambdas/po/email_processor -- the "
"'cp ./*.py' glob would ship exactly these into the Lambda zip. "
f"Found {sorted(top_level)}, expected "
f"{sorted(PO_EXPECTED_TOP_LEVEL_MODULES)}. If a new module is "
"intended, add it to PO_EXPECTED_TOP_LEVEL_MODULES; if it is a scratch "
"or secrets file, remove it (or exclude it from bundling) before deploy."
"'cp po/email_processor/*.py' glob would ship exactly these into the "
f"Lambda zip. Found {sorted(top_level)}, expected "
f"{sorted(PO_PIPELINE_MODULES)}. A moved shared module "
f"({sorted(SHARED_MODULES)}) reappearing here would SHADOW the single "
"shared source in every handler-loaded test -- remove it. If a new "
"per-pipeline module is intended, add it to PO_PIPELINE_MODULES."
)
def test_wo_email_processor_dir_ships_no_unexpected_top_level_modules():
"""Upper bound on the WO pipeline dir alone (NOT unioned with shared/).
The WO equivalent of the PO exact-set pin -- previously MISSING entirely,
so a stray .py (or a shadow copy of a moved shared module) in
lambdas/wo/email_processor was policed by nothing. Same rationale as the PO
pin: `cp wo/email_processor/*.py` ships every top-level .py in this dir, and
a strayed-back ses_auth/email_parsing/emf would shadow the shared source in
the WO handler-loaded tests.
"""
top_level = {p.stem for p in WO_HANDLER.parent.glob("*.py")}
assert top_level == set(WO_PIPELINE_MODULES), (
"unexpected top-level .py set in lambdas/wo/email_processor -- the "
"'cp wo/email_processor/*.py' glob would ship exactly these into the "
f"Lambda zip. Found {sorted(top_level)}, expected "
f"{sorted(WO_PIPELINE_MODULES)}. A moved shared module "
f"({sorted(SHARED_MODULES)}) reappearing here would SHADOW the single "
"shared source in every handler-loaded test -- remove it. If a new "
"per-pipeline module is intended, add it to WO_PIPELINE_MODULES."
)
def test_shared_dir_ships_no_unexpected_top_level_modules():
"""Upper bound on lambdas/shared/ alone (NOT unioned with a pipeline dir).
`cp shared/*.py` ships every top-level .py under lambdas/shared/ into BOTH
email-processor bundles, so a stray scratch/secrets .py here would leak into
both production zips. Pinned to exactly the four single-sourced modules.
"""
top_level = {p.stem for p in SHARED_DIR.glob("*.py")}
assert top_level == set(SHARED_MODULES), (
"unexpected top-level .py set in lambdas/shared -- the 'cp shared/*.py' "
"glob would ship exactly these into BOTH email-processor Lambda zips. "
f"Found {sorted(top_level)}, expected {sorted(SHARED_MODULES)}. If a new "
"shared module is intended, add it to SHARED_MODULES; if it is a scratch "
"or secrets file, remove it before deploy."
)
def test_no_shared_module_shadow_in_pipeline_dirs():
"""The moved shared names must live ONLY under lambdas/shared/.
Replaces the future-drift guard the retired byte-identity ses_auth fixture
used to provide. A stray reappearance of a moved shared module in either
email-processor pipeline dir would be resolved FIRST by every handler loader
-- tests/conftest.py load_handler checks `path.parent / f"{sibling}.py"`
before the _SHARED_DIR fallback, and
lambdas/wo/email_processor/tests/_wo_parser_support.py inserts the pipeline
dir ahead of shared/ on sys.path -- silently SHADOWING the single shared
source in every handler-loaded test, while test_ses_auth / test_parse_raw_email
keep exercising shared/. Divergence would go undetected. Police the pipeline
dirs and shared/ SEPARATELY so no union can mask the shadow.
"""
for name in sorted(SHARED_MODULES):
assert (SHARED_DIR / f"{name}.py").exists(), (
f"{name}.py must exist under lambdas/shared/ (single source of truth)"
)
assert not (PO_HANDLER.parent / f"{name}.py").exists(), (
f"{name}.py reappeared in lambdas/po/email_processor -- it would "
"SHADOW lambdas/shared/{name}.py in every PO handler-loaded test "
"(the loader resolves the pipeline-local copy first). Delete it; "
"the single source lives under lambdas/shared/."
)
assert not (WO_HANDLER.parent / f"{name}.py").exists(), (
f"{name}.py reappeared in lambdas/wo/email_processor -- it would "
"SHADOW lambdas/shared/{name}.py in every WO handler-loaded test "
"(_wo_parser_support inserts the pipeline dir ahead of shared/ on "
"sys.path). Delete it; the single source lives under lambdas/shared/."
)
def test_detection_logic_catches_allowlist_missing_a_sibling():
"""Unit-level check on `_bundling_ships_all` itself.
@ -272,13 +450,15 @@ def test_detection_logic_catches_allowlist_missing_a_sibling():
assert not _bundling_ships_all(reverted_allowlist_command, siblings)
# Control: the same allowlist shape WITH derived_fields.py added back in
# is correctly recognized as complete, proving the failure above is
# about the missing file and not a false-positive-prone regex.
# Control: the same allowlist shape listing ALL required siblings (including
# the Phase 3 shared ones -- derived_fields, email_parsing, emf added back)
# is correctly recognized as complete under the accumulate model, proving
# the failure above is about the missing files and not a regex artifact.
complete_allowlist_command = (
"pip install --platform manylinux2014_aarch64 --only-binary=:all: "
"-r requirements.txt -t /asset-output && "
"cp handler.py ses_auth.py template_parser.py derived_fields.py /asset-output/"
"cp handler.py ses_auth.py template_parser.py derived_fields.py "
"email_parsing.py emf.py /asset-output/"
)
assert _bundling_ships_all(complete_allowlist_command, siblings)
@ -325,6 +505,26 @@ def test_detection_logic_rejects_commented_out_scoped_glob():
assert not _bundling_ships_all(commented_out_command, siblings)
def test_detection_logic_rejects_commented_out_shared_cp():
"""A `cp shared/*.py` surviving only in a `#` comment must not count as run.
Simulates a half-finished revert that drops the executed shared cp but
leaves its text trailing as a comment. `_executed_cp_commands` strips `#`
comments before any regex sees the segment, so the shared-cp text alone --
never executed by bash -- is not among the executed cp's. Combined with the
fixed ships-all (ses_auth/email_parsing/emf resolve ONLY under shared/), a
removed or commented shared cp fails both the SHARED_CP_RE pin and ships-all.
"""
commented_out_command = (
"cp po/email_processor/*.py /asset-output/ # cp shared/*.py /asset-output/"
)
executed = _executed_cp_commands(commented_out_command)
assert not any(re.search(SHARED_CP_RE, cp) for cp in executed)
# And ships-all is False: the shared siblings never got shipped.
po_siblings = _first_party_sibling_imports(PO_HANDLER)
assert not _bundling_ships_all(commented_out_command, po_siblings)
def test_detection_logic_rejects_wrong_pipeline_glob():
"""A glob pointing at the WRONG pipeline (or any other dir) must not pass.
@ -361,3 +561,58 @@ def test_detection_logic_rejects_wrong_pipeline_glob():
assert not re.search(PO_SCOPED_GLOB_RE, "cp wo/email_processor/*.py /asset-output/")
assert re.search(WO_SCOPED_GLOB_RE, "cp wo/email_processor/*.py /asset-output/")
assert not re.search(WO_SCOPED_GLOB_RE, "cp po/email_processor/*.py /asset-output/")
def test_po_web_ui_bundle_stages_shared_auth_module():
"""The PO web_ui bundle must EXECUTE `cp shared/web_ui_auth.py`.
Phase 3 removed the inline auth block from lambdas/po/web_ui/handler.py,
which now does a module-top-level `from web_ui_auth import is_authenticated`;
web_ui_auth.py lives ONLY under lambdas/shared/. No test imports the web_ui
handler, and the email-processor AST pins deliberately exclude the web_ui
command -- so without this pin, dropping/commenting the web_ui cp would
ImportError the auth-gated Lambda at cold start with green CI. Checked
against the comment-stripped executed cp so the old text surviving only in a
comment cannot satisfy it.
"""
command = _extract_web_ui_command(PO_STACK)
executed_cps = _executed_cp_commands(command)
assert any(re.search(WEB_UI_AUTH_CP_RE, cp) for cp in executed_cps), (
"cdk/po_stack.py web_ui bundling command must EXECUTE "
"'cp shared/web_ui_auth.py /asset-output/' so the shared auth module "
"ships flat beside handler.py and `from web_ui_auth import "
f"is_authenticated` resolves at cold start. Executed cp commands: "
f"{executed_cps}"
)
def test_wo_web_ui_bundle_stages_shared_auth_module():
"""The WO web_ui bundle must EXECUTE `cp shared/web_ui_auth.py`.
WO equivalent of test_po_web_ui_bundle_stages_shared_auth_module -- same
ImportError-at-cold-start hazard for lambdas/wo/web_ui/handler.py.
"""
command = _extract_web_ui_command(WO_STACK)
executed_cps = _executed_cp_commands(command)
assert any(re.search(WEB_UI_AUTH_CP_RE, cp) for cp in executed_cps), (
"cdk/wo_stack.py web_ui bundling command must EXECUTE "
"'cp shared/web_ui_auth.py /asset-output/' so the shared auth module "
"ships flat beside handler.py and `from web_ui_auth import "
f"is_authenticated` resolves at cold start. Executed cp commands: "
f"{executed_cps}"
)
def test_detection_logic_rejects_commented_out_web_ui_auth_cp():
"""A `cp shared/web_ui_auth.py` surviving only in a `#` comment must not pass.
Mirror of test_detection_logic_rejects_commented_out_shared_cp for the
web_ui staging: a half-finished revert that drops the executed cp but leaves
its text trailing as a comment must NOT register as executed, since bash
would never run it.
"""
commented_out_command = (
"cp -r po/web_ui/. /asset-output/ # cp shared/web_ui_auth.py /asset-output/"
)
executed = _executed_cp_commands(commented_out_command)
assert not any(re.search(WEB_UI_AUTH_CP_RE, cp) for cp in executed)

View file

@ -0,0 +1,124 @@
"""Synthetic-event-shape contract test for scripts/reprocess.py (refactor §4.7).
Pins the exact S3 event shape reprocess emits and asserts the object key is
the RAW list_objects_v2 key — reprocess applies NO URL-encoding or decoding.
Why this matters: a real S3 event notification URL-encodes the object key, and
the handler is where any decode would live. reprocess builds its synthetic
event from the RAW list_objects_v2 obj["Key"] (or the raw --key arg), so it
must emit that key byte-for-byte. Replaying a pre-decoded (or pre-encoded) key
would not match what s3.get_object(Bucket, Key) expects downstream.
Recon note (honored narrowly): neither the PO nor the WO handler imports
urllib.parse or calls unquote/unquote_plus today — the raw event key goes
straight into s3.get_object. So the "handler decodes -> replaying a decoded
key double-decodes" failure mode is NOT reproducible against today's handlers
(they never decode once). This test therefore guards ONLY reprocess's own
contract: emit the raw list_objects_v2 key with no transformation. That keeps
replay single-decode-correct if a future handler ever adds unquote_plus.
"""
import importlib.util
import os
import pathlib
import sys
import pytest
# reprocess.py builds its boto3 clients inside main()/functions, so importing
# it is side-effect-free (no AWS calls, no clients at module scope). Set dummy
# region/creds defensively anyway so import stays offline-safe even if that
# ever changes. build_s3_event itself makes no AWS calls.
os.environ.setdefault("AWS_DEFAULT_REGION", "us-east-1")
os.environ.setdefault("AWS_ACCESS_KEY_ID", "testing")
os.environ.setdefault("AWS_SECRET_ACCESS_KEY", "testing")
os.environ.setdefault("AWS_SESSION_TOKEN", "testing")
# scripts/ is not a package (no __init__.py) and is not on sys.path, and
# tests/conftest.py only wires up per-handler dirs — so load reprocess.py by
# file path with importlib rather than `import scripts.reprocess`.
_p = pathlib.Path(__file__).resolve().parents[1] / "scripts" / "reprocess.py"
_spec = importlib.util.spec_from_file_location("reprocess", _p)
reprocess = importlib.util.module_from_spec(_spec)
_spec.loader.exec_module(reprocess)
def test_build_s3_event_shape_and_raw_key():
bucket = "po-ingest-emails-328440206208"
# Deliberately contains a SPACE, a literal '+', and a literal '%41'.
# A real S3 event notification would deliver this key encoded as
# "inbound/2026/AB+12%2B34+%2541.eml"
# (space->'+', '+'->'%2B', '%'->'%25'); the handler is where any decode
# would live. reprocess builds from the RAW list_objects_v2 key, so it
# must emit "inbound/2026/AB 12+34 %41.eml" unchanged.
raw_key = "inbound/2026/AB 12+34 %41.eml"
event = reprocess.build_s3_event(bucket, raw_key)
# 1. Exact top-level shape: a single Records entry.
assert list(event.keys()) == ["Records"]
assert len(event["Records"]) == 1
# 2. Bucket name is nested at Records[0].s3.bucket.name.
assert event["Records"][0]["s3"]["bucket"]["name"] == bucket
# 3. Object key is nested at Records[0].s3.object.key, byte-for-byte.
assert event["Records"][0]["s3"]["object"]["key"] == raw_key
# 4. RAW / no-URL-encoding, no-URL-decoding on the emitted key.
emitted = event["Records"][0]["s3"]["object"]["key"]
assert emitted == raw_key # reprocess applies NO transformation
# space NOT percent-encoded (a real S3 notification would send %20):
assert " " in emitted and "%20" not in emitted
# literal '+' preserved, not turned into a space:
assert "+" in emitted
# '%41' NOT decoded to 'A':
assert "%41" in emitted and "A .eml" not in emitted
# --- Selector mutual-exclusion: --all is a MODE, not a modifier ---------------
# reprocess.main() reaches its first AWS call (sts get_caller_identity) only
# AFTER the argparse mutual-exclusion gate, so an invalid selector combination
# raises SystemExit(2) with no AWS access needed. This pins that `--all` cannot
# be silently combined with a narrower --prefix/--since (which the args.all
# clobber branch would otherwise discard -> unintended whole-corpus sweep).
def _expect_argparse_reject(monkeypatch, capsys, argv):
monkeypatch.setattr(sys, "argv", ["reprocess.py", *argv])
with pytest.raises(SystemExit) as exc:
reprocess.main()
assert exc.value.code == 2 # argparse error exit code
return capsys.readouterr().err
def test_all_rejects_prefix(monkeypatch, capsys):
err = _expect_argparse_reject(
monkeypatch,
capsys,
["--pipeline", "po", "--all", "--prefix", "inbound/2026/07/"],
)
assert "--all cannot be combined with --prefix or --since" in err
def test_all_rejects_since(monkeypatch, capsys):
err = _expect_argparse_reject(
monkeypatch,
capsys,
["--pipeline", "po", "--all", "--since", "2026-07-01T00:00:00Z"],
)
assert "--all cannot be combined with --prefix or --since" in err
def test_all_rejects_prefix_and_since_together(monkeypatch, capsys):
_expect_argparse_reject(
monkeypatch,
capsys,
[
"--pipeline",
"po",
"--all",
"--prefix",
"inbound/2026/",
"--since",
"2026-07-01T00:00:00Z",
],
)