adam/procurement-ingest
1
0
Fork 0
mirror of https://github.com/Sea-Haven-Industries/procurement-ingest.git synced 2026-10-07 16:08:54 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Compare commits

base: adam:df80a2e3ec3ac076487f972f0a5cd7dfb8431aa0
Branches Tags
adam:main
adam:chore/enqueue-as-seahaven-bot
adam:fix/wo-dlq-age-alarm
adam:feature/wo-table-kebab-rename
adam:fix/suppress-codeql-false-positive-shoc-rotator
adam:chore/INFRA-114-logretention-migration
..
compare: adam:64f74c243aec62b650e4c11a5b916377d6dc052c
Branches Tags
adam:main
adam:chore/enqueue-as-seahaven-bot
adam:fix/wo-dlq-age-alarm
adam:feature/wo-table-kebab-rename
adam:fix/suppress-codeql-false-positive-shoc-rotator
adam:chore/INFRA-114-logretention-migration

6 commits
df80a2e3ec ... 64f74c243a

Author SHA1 Message Date
Adam Moussa
64f74c243a
test: lock attribute-context quote escaping in web_ui hostile-field test 
The escaping regression lock asserted only the element-context vector
(raw <script> absent, &lt;script&gt; present) while its docstring claimed
quotes were covered -- the payload's " and ' were never asserted on, so
a quote-escaping regression on the onclick row-link sink (attribute
breakout -> event-handler injection) would have passed green.
/sh-security-review finding WC-01 (confirmed medium, test-integrity).

Add assertions that the onclick sink's JSON string renders its opening
quote as &quot; (raw " after window.location= fails), that the
payload's quote characters appear only entity-escaped, and that the
raw payload never appears anywhere in the body. Mutation-verified: the
test now fails when the sink's quote-escaping is dropped.
 2026-07-20 14:53:17 -04:00
Adam Moussa
46c6f27190
test: consolidate test roots — one repo-root loader, shared support package, missing-scenario suites, enforced ruff/coverage floor (refactor phase 8) 
tests/conftest.py only loads for the tests/ root, not a standalone
`pytest lambdas/po/email_processor/tests` run, so it could never carry
session invariants like the dummy AWS env or the moto stubber
registration. Add a single repo-root conftest.py (pytest.ini pins
rootdir there, so it loads for every invocation) that sets the dummy
AWS credentials/region, imports moto BEFORE any handler module so
boto3 sessions pick up its stubber hook (carrying the explanatory
comment verbatim from the old _po_parser_support.py), and exposes one
load_lambda_module(pipeline, name) — the sys.modules save/restore
dance stays, since template_parser is still a duplicated bare name
across pipelines needing per-exec sibling binding.

Add tests/support/ as the shared package both pipelines' local
_*_parser_support.py modules delegate to: a superset FakeTable (PO's
update_item recording + WO's put_item and keyed single-row store),
FakeDynamoResource, load_email, and load_golden with parse_float=Decimal
kept (load-bearing for exact money comparison at PO magnitudes — WO's
prior load_golden had no parse_float and must not regress PO by losing
it). Rewrite _wo_parser_support.py off the bare `import handler` /
`from handler import parse_raw_email` strategy that was the source of
the bare-name sys.modules collision the other two loaders defend
against.

Move test_po_merge.py and test_pad_zip.py into
lambdas/po/email_processor/tests/ (PO-specific, belongs beside the
code) via git mv so history follows; test_parse_raw_email.py and
test_ses_auth.py stay at the repo root since they're genuinely
cross-pipeline, parameterized over both handlers. Delete
tests/test_local.py: it globs a nonexistent samples/ dir, is WO-only,
and imports a handler at collection time, bypassing the loader gate
entirely — the golden suites already cover its role. Its pytest.ini
exclusion comment goes with it.

New scenario coverage, all built on the single loader + support
package:
- PO+WO Bedrock transport errors (ThrottlingException, missing
  'content' key, empty content list, non-JSON model text), asserting
  PO's pre-call ai_fallback metric survives with no partial write and
  the exception propagates; WO's no-datapoint-on-throttle behavior is
  pinned with a documenting test rather than "fixed" by reordering.
- Handler-level SES-auth reject seam per pipeline: no auth
  monkeypatch + empty ALLOWED_DKIM_DOMAINS asserts zero Bedrock calls,
  zero writes, no raise — closing the hole where deleting the gate
  line today still passes every test.
- web_ui coverage for both PO and WO (0% before this): fail-closed on
  unset ARN and on a Secrets Manager exception, TTL cache refresh,
  Bearer/X-Auth-Token/header-case-insensitivity, wrong-token 401 with
  no table scan, non-ASCII token, and a hostile-field-escaping
  regression lock. PO web_ui has no __init__.py, so these go through
  the loader rather than package imports.
- A moto-backed mirror of test_po_merge for WO merge semantics
  (table 'WorkOrders'): null-status never clobbers wo_status,
  created_at immutable via if_not_exists, status->wo_status mapping,
  None fields absent from SET, record_type only-when-present.
- Small pins: the PO-DC-02 64-char EMF clamp regression and
  per-pipeline multi-record failure-isolation (all-or-retry contract).
  The reprocess.py synthetic-event-shape contract test already landed
  in Phase 7, so it isn't duplicated here.

Two WO product-code fixes ride along, since this is the phase that
exercises them: (a) the invalid_status reason-code fix in
template_parser.py's status check, which previously returned
malformed_site_code for the same failure validate_ai_fallback already
labels invalid_status, making one failure surface two codes depending
on path (grepped the dashboards/metric filters for
malformed_site_code first — no external references found, safe to
diverge the two codes); (b) wrapping the WO Bedrock call in
handler.py so a transport failure emits ai_fallback/bedrock_error in
an except-and-reraise. This is deliberately not a naive reorder: the
emit sits in the except block, not pre-call, so a gate-rejected email
still emits only ai_fallback_rejected and wo_stack's "a rejected
email emits nothing else" alarm contract doesn't double-count. A test
computes the emitted series by hand to pin the no-double-count
behavior. Neither change touches the handler event/return contract.

_validate_new_po_values in the PO template_parser.py is split into
per-rule helpers, and the V4 anchor-frame dataclass now carries
summary_matches/price so V13 can consume them; extract_new_po
(C901=35) is included in the split. Add ruff.toml enabling C901/PLR
so the mccabe/complexity suppressions scattered through the tree stop
being decorative; derived_fields.py is under the shadow-bake freeze
so its violations are silenced via a per-file ignore with a
justification comment instead of an in-file edit, and the handful of
other pre-existing violations surfaced by turning the config on get
the same per-file-ignore treatment with a reason, or a fix where the
file isn't frozen. scripts/ is added to the CI lint scope.

CI gains an explicit --cov module list (lambdas/po and wo
email_processor + web_ui, po/site_extractor, lambdas/shared) plus
--cov-fail-under=80, since web_ui and site_extractor lack __init__.py
markers and a bare --cov=lambdas silently skips them for the missing
package marker; .coveragerc omits the test dirs themselves from the
count. The Phase 0 AST bundle-consistency test stays in the standard
pytest run. .gitignore picks up the resulting .coverage data file.

docs/po-template-parser.md gets a small correction: the EXTRACTION_PROMPT
declares quantity/price as "number or null", not JSON strings, so
parse_float=Decimal already handles a conforming Bedrock response —
the doc previously implied the coercion path was the primary
mechanism rather than a defensive net for non-conforming responses.
 2026-07-20 14:14:30 -04:00
Adam Moussa
bb5b6c1964
feat: decompose email-processor handlers into flat siblings + lazy boto3 clients (refactor phase 5) 
Both email-processor God-handlers split along the seams that already
work in the flat-sibling pattern established by lambdas/shared/, so
bare-name imports keep working under the existing bundling glob.

PO (5-way split): handler.py keeps only the event loop, fail-closed
auth, and email_type routing. extraction.py holds extract_with_claude
and _EMAIL_TAG_RE, importing EXTRACTION_PROMPT from prompts.py and
parse_raw_email from shared/email_parsing.py rather than recreating a
PO-local copy. enrichment.py is a pure code move of enrich_parsed and
pad_zip (PO-only; WO has no enrichment stage) with zero behavior
change. telemetry.py holds the EMF ParseMethod emit wrappers.
persistence.py holds _write_fields/_merge_update/save_*, collapsing
the byte-identical save_new_po/save_revision bodies into one
_save_merge helper that both now call through, preserving the sticky
Cancelled ConditionExpression guard for both callers; save_cancellation
stays separate.

WO (5 concerns, no enrichment stage): the handler loop keeps
validate_ai_fallback and the re.fullmatch(r"[0-9]+", work_order_id)
key guard ahead of both save_work_order and save_event, since the
guard protects the DynamoDB partition key and the '#'-delimited
comment_id range-key segment. _header_date_iso and comment_id
determinism stay colocated with persistence.py's save_event for the
retry-idempotent event_id key.

EXTRACTION_PROMPT (PO) moves to prompts.py with cross-reference
headers to derived_fields.py's authoritative trade/site/fiscal rule
tables; handler.py re-exports it (from prompts import
EXTRACTION_PROMPT) since four tests dereference handler.EXTRACTION_
PROMPT directly. WO's prompt moves the same way.

I/O modules (extraction.py's bedrock client, persistence.py's
dynamodb resource, handler.py's s3 client) get lazy cached boto3
accessors; pure modules (enrichment.py, prompts.py, telemetry.py)
import no boto3. Test monkeypatch surfaces move to the module that
now owns the client (e.g. persistence.dynamodb) everywhere tests
patch it, and the moto-before-handler-import ordering in
_po_parser_support.py is preserved so the moto-backed suites don't
hit real AWS.

Behavior-preservation pins, verified with tests: PO still emits
ParseMethod=ai_fallback before the Bedrock call, with
ai_fallback_rejected as the additive second datapoint on rejection.
WO still emits after its gate with mutually-exclusive ai_fallback /
ai_fallback_rejected. Shadow DerivedFieldAgreement telemetry stays
ai_fallback-only. derived_fields.py is untouched (diff against
feature/phase-3-shared-extraction is empty). handler(event, context)
signatures and the save_* public contract are unchanged on both
pipelines; goldens unchanged.

PO_EXPECTED_TOP_LEVEL_MODULES and its WO equivalent in
tests/test_bundle_consistency.py are updated for the new sibling
modules so the AST bundle-consistency test still fails on an
unshipped or uncommented-out sibling.
 2026-07-17 21:00:31 -04:00
Adam Moussa
d3e49c12ef
feat: extract lambdas/shared/ — single-source ses_auth, web_ui auth, email parsing, EMF emitter (refactor phase 3) 
Four modules move into the handbook-mandated lambdas/shared/ location,
collapsing duplicated logic that had to be kept in sync by hand across
the PO and WO pipelines:

- ses_auth.py: the PO and WO copies were verified sha256-identical
  against the feature/phase-7-ops-recovery baseline before the move
  (no drift since the last audit). shared/ses_auth.py is the exact
  bytes of that one copy; both originals are git rm'd (the PO copy
  via rename, the WO copy as a straight delete). Bundling lands the
  module flat in /asset-output for both email processors, so the
  handlers keep `from ses_auth import authenticate_inbound_email`
  unchanged — zero handler diff for this move, which is what keeps
  fail-closed auth byte-identical through the change.

- web_ui_auth.py: extracts the byte-identical _get_auth_token /
  _header / is_authenticated block plus the four token-cache globals
  out of both web_ui handlers. The per-stack INFRA-74 comments stay
  in each handler as-is (deliberately drifted wording, stack-specific)
  rather than being unified into the shared module. Fail-closed
  semantics (unset ARN or Secrets Manager exception -> deny) are
  unchanged.

- email_parsing.py: parse_raw_email ships as the superset version that
  returns cc unconditionally. WO's output is bit-identical to before;
  PO simply ignores the cc field rather than being "cleaned up" to
  consume it. No second variant is kept.

- emf.py: a generic emitter parameterized by namespace, dimension
  sets, and properties. Every call site's emitted EMF envelope is
  unchanged, including the load-bearing
  [["ParseMethod"],["ParseMethod","TemplateId"]] dimension-set shape
  the alarms and metric filters depend on. Emission ordering is
  untouched: PO still emits ai_fallback before the Bedrock call, WO
  still emits its mutually-exclusive ai_fallback/ai_fallback_rejected
  after its gate. The deliberate-double-count comments survive.
  _emit_derived_agreement_metric was found living inside
  derived_fields.py, so per the DERIVED-FIELDS exception it is left
  as a third, unconverted copy (derived_fields.py and the shadow
  DerivedFieldAgreement telemetry stay untouchable while that bake
  runs) — a comment there points at shared/emf.py for the eventual
  follow-up.

Bundling: both email-processor cdk bundling commands gain a trailing
`cp shared/*.py /asset-output/` (they were already cp-only post-Phase
7, so no pip step or manylinux pin is reintroduced). Both web_ui
functions gain the same widened-root staging so web_ui_auth.py ships
beside their handler; site_extractor's from_asset is untouched.

Tests: PO_EXPECTED_TOP_LEVEL_MODULES gains the shared modules that now
ship, the AST sibling-import check resolves imports whose source now
lives under shared/, and the new shared cp line has its own
revert/mutation detection. _SIBLING_MODULES resolution and
_po_parser_support.py now load ses_auth/email_parsing/emf from
shared/; the two-copy ses_auth byte-identity fixture-hygiene test is
retired as obsolete now that there is one copy, and the ses_auth
fixture parameterization over two identical copies is dropped. The
sys.modules save/restore dance for template_parser (still duplicated
per-pipeline) is left in place.
 2026-07-17 18:39:53 -04:00
Adam Moussa
88805d906d
Reject --all combined with --prefix/--since in reprocess.py 
--all is a distinct mode (the demoted full-prefix sweep), but the args.all
branch unconditionally set prefix=inbound/ and since=None, so passing it
alongside a narrower selector silently discarded that selector. `--all
--since 2026-07-01` swept the entire corpus instead of the bounded window,
triggering every documented --all hazard (Bedrock re-bill, metric double-
count, merged-field regression) on objects the operator never targeted --
contradicting the tool's safety goal. Add the missing mutual-exclusion
guard alongside the existing --key one, and pin --all+--prefix,
--all+--since, and all three together as argparse rejections.
 2026-07-17 17:03:44 -04:00
Adam Moussa
7765111781
feat: ops/recovery tooling + dependency hygiene (refactor phase 7) 
Generalize scripts/reprocess.py from a PO-only full-sweep script into a
pipeline-general recovery tool. Targeted replay (--key/--prefix/--since)
is now the default, and the full inbound/ sweep is demoted behind an
explicit --all that documents its five hazards (async concurrency does
not serialize, use RequestResponse if order matters, metric double-count,
Bedrock re-bill, out-of-order field regression). --pipeline po|wo resolves
the correct function + bucket; dry-run-by-default / --execute is preserved.
A new tests/test_reprocess_contract.py pins the synthetic S3 event shape
and asserts the raw list_objects_v2 key is emitted untransformed (the
handler is the single decode point; a pre-decoded key would corrupt keys
containing spaces or '+').

Add docs/runbook-dlq-recovery.md: the async on-failure DLQ has no console
redrive-to-source, so it documents the receive -> extract key -> targeted
reprocess --key -> verify -> purge procedure, the real recovery windows
(14-day DLQ breadcrumb, 90-day raw-email S3 that overrides the table
RETAIN policy and is the true replay floor), and that sender-auth and
ai_fallback_rejected drops are fail-closed skips that never reach the DLQ.
Linked from the README alarms and scripts sections.

Drop the vendored boto3 floor pin from both email-processor requirements
(the Lambda runtime provides boto3; lambda-template.md empty-with-comment
form). With nothing left to install, the email-processor bundling becomes
cp-only -- the whole pip step is removed, which is the only acceptable way
the manylinux2014_aarch64 pin disappears (removing the pin while keeping a
pip install caused the PR #34 x86-wheel outage). Exact-pin moto==5.2.2 and
add pinned po/web_ui + po/site_extractor manifests (excluded from their
bundles, so hash-neutral) so their new Dependabot entries have something
to act on; add Dependabot entries for /tests, /lambdas/po/web_ui, and
/lambdas/po/site_extractor.

cdk diff is confined to exactly the two email processors' asset hashes on
both stacks. The wo/web_ui dead-manifest reduction was deliberately left
out: that manifest already ships inside the plain (non-bundled) WebUI
asset on main, so reducing or excluding it would redeploy workorder-web-ui
for no functional change -- deferred to keep the blast radius to the two
intended targets.

The untracked 44 MB lambdas/po/email_processor/package/ dir was removed
from the filesystem (asset-hash-neutral given Phase 2's package/ exclude);
it is untracked, so there is nothing to commit for it.
 2026-07-17 16:32:59 -04:00
9 changed files with 637 additions and 1227 deletions
Show stats Expand all files Collapse all files

684
.claude/workflows/phase-4-cdk-common.js
View file

@ -1,684 +0,0 @@
export const meta = {
name: 'phase-4-cdk-common',
description: 'Phase 4 of the procurement-ingest refactor (docs/refactor-evaluation.md): collapse the 379 identical CDK lines into cdk/common.py as PLAIN FUNCTIONS taking (scope, id, ...) — called with the SAME Stack scope and the SAME construct ids the stacks use today, so every logical ID is byte-stable (Construct-subclass wrapping is forbidden: it would reparent the tree and attempt REPLACEMENT of the RETAIN-protected purchase-orders/WorkOrders tables and named buckets = data loss). Extracts add_ddb_alarms, add_sender_auth_rejected_alarm, add_standard_lambda_alarms, make_bedrock_invoke_statement (account/region-derived ARN, not hardcoded 328440206208), make_email_bucket, make_processor_dlq, make_fallback_rate_alarm — preserving every per-function alarm variance byte-for-byte. Same PR: account on both cdk.Environment, constructs== exact pin, stale-comment fix, CfnOutputs for five function ARNs + consumed table names. ZERO cdk diff on both stacks is the acceptance test. The make_bedrock_invoke_statement IAM PolicyStatement move triggers mandatory GPT-4.1 cross-family review even though semantics are identical. Committed locally, never pushed.',
phases: [
{ title: 'Setup', detail: 'verify Phases 0+1+2+3 on base, branch feature/phase-4-cdk-common', model: 'haiku' },
{ title: 'Recon', detail: '4 mappers: the 379 duplicated CDK lines + per-function alarm variance, the two inline Bedrock PolicyStatements, the fallback-rate + rejected alarm math (post-Phase-1), app.py/pins/outputs surface' },
{ title: 'Spec', detail: 'serial fable spec: pin common.py contents + every function signature, per-stack call-site rewrites, alarm-variance table, Bedrock IAM equivalence, fallback-rate call params, app/pins/CfnOutputs, zero-diff judging rules' },
{ title: 'Implement', detail: 'opus: cdk/common.py; opus: both stacks (rewire + CfnOutputs); sonnet: app.py + requirements pin + README — disjoint files', model: 'opus' },
{ title: 'Verify', detail: 'mechanical gates + zero-cdk-diff verifier (the load-bearing gate) + 3 fable lenses (logical-ID safety, alarm variance, IAM equivalence)' },
{ title: 'Fix', detail: 'opus fixer, full re-verify, max 3 rounds', model: 'opus' },
{ title: 'Package', detail: 'single commit via -F (no push); runs cross_review.py inline on the IAM diff', model: 'sonnet' },
],
}
// ---------------------------------------------------------------- constants
const REPO = '/Users/adammoussa/Documents/repositories/seahaven/procurement-ingest'
const BRANCH = 'feature/phase-4-cdk-common'
let _args = args
if (typeof _args === 'string') {
try { _args = JSON.parse(_args) } catch (e) { _args = null }
}
const BASE = (_args && _args.base) || 'main'
const CONSTRAINTS = `
PINNED BEHAVIORAL CONSTRAINTS (docs/refactor-evaluation.md Phase 4 — violating any is a build failure):
1. EXTRACT AS PLAIN FUNCTIONS taking (scope, id, ...), called with the SAME
scope (the Stack instance) and the SAME construct ids the stacks use
today -> 100% logical-ID-safe. NEVER wrap in Construct subclasses: a
subclass inserts a tree node, changes EVERY child logical ID, and would
attempt REPLACEMENT of the RETAIN-protected purchase-orders / WorkOrders
tables and the named buckets = DATA LOSS. This is THE load-bearing rule
of the phase — every other check exists to defend it.
2. New module cdk/common.py. Extract exactly:
- _DDB_ALARM_OPERATIONS + add_ddb_alarms
- add_sender_auth_rejected_alarm
- add_standard_lambda_alarms(scope, id_prefix, fn, name_prefix, topic, *,
duration_statistic, errors=True, dlq=None, descriptions=...)
- make_bedrock_invoke_statement (DERIVE the inference-profile ARN + the
per-region foundation-model ARNs from Stack.of(scope).account /
Stack.of(scope).region — NOT hardcoded 328440206208)
- make_email_bucket
- make_processor_dlq
- make_fallback_rate_alarm(namespace, rejected_included, period, threshold,
floor, evaluation_periods, datapoints_to_alarm) reproducing the
expression strings / FILL / labels BYTE-FOR-BYTE.
3. PER-FUNCTION ALARM VARIANCE — preserve EXACTLY, do NOT homogenize:
PO email-processor duration p99 vs wo-email-processor p95; po-web-ui
throttles+duration only; site_extractor no-DLQ; wo web_ui has ZERO alarms
(do NOT let the shared helper silently add any); every bespoke alarm
DESCRIPTION string is passed through verbatim.
4. FALLBACK-RATE RECONCILIATION (post-Phase-1): PO's template-fallback-rate
EXCLUDES the rejected series (byte-identical to today, a pre-call
double-count would result otherwise) -> call make_fallback_rate_alarm with
rejected_included=False; WO's INCLUDES rejected -> rejected_included=True.
The two REJECTED alarms are a DIFFERENT shape and are NOT
make_fallback_rate_alarm: PO's ai-fallback-rejected is a 6h count-floor
IF(FILL(rej,0)>=1,...) alarm (added in Phase 1); WO's is the 5-min /
2-of-6 sparse idiom. Keep those as DISTINCT call sites (or a separate
dedicated helper) — do NOT force them through make_fallback_rate_alarm.
NO element-wise MAX anywhere (post-#102 rule).
5. Do NOT import stack-specific services (kms / ssm / event_sources) into
common.py — only the constructs the shared helpers actually need.
6. SAME PR, net-new & logical-ID-safe additions:
- add account='328440206208' to BOTH cdk.Environment calls
- pin constructs== to the exact installed version (not a floor >=)
- fix the stale "2.259.0" version comments
- add CfnOutputs for the FIVE function ARNs + the consumed table names.
These are additive; CfnOutputs and account are ID-safe. Verify none of
them perturbs an existing logical ID.
7. ZERO lambdas/ diff: git diff ${BASE}...HEAD -- lambdas/ must be EMPTY.
This phase is CDK-ONLY. tests/ may gain a cdk-diff / synth-only test for
the acceptance gate, but NO other tests/ change and NO lambdas/ change.
8. ZERO cdk diff on BOTH stacks is the acceptance test: npx cdk diff
po-ingest and npx cdk diff workorder-ingest must show ZERO resource
changes (no logical-ID, alarm, IAM, table, bucket, env, or metadata
delta beyond CDK-tooling noise). The CfnOutputs are the ONLY net-new
resources allowed to appear, and only as additions.
9. The wo artifact id is 'workorder-ingest' (the construct id / 2nd
positional arg), NOT 'WorkorderIngestStack' (that is stack_name). ALWAYS
drive synth/diff by the artifact id: npx cdk synth workorder-ingest,
npx cdk diff workorder-ingest. Using the stack_name selector fails.
10. NO cdk deploy, NO invoke, NO AWS mutation. Read-only AWS only if needed
(e.g. confirming deployed alarm names) — the diff gate is a pure local
synth-vs-synth comparison.
`
const PREAMBLE = `
You are one of several agents building refactor Phase 4 in the git repo at
${REPO} on branch ${BRANCH} (already checked out — do NOT switch branches,
do NOT create branches, do NOT commit, NEVER push, do NOT run cdk deploy or
touch AWS resources beyond read-only calls).
Authoritative spec: docs/refactor-evaluation.md, section "Phase 4".
Work ONLY in the files you are told you own; other agents are concurrently
editing other files in this same working tree.
${CONSTRAINTS}
Your final message is consumed by an orchestrator script, not a human —
return only the structured data requested.
`
// ------------------------------------------------------------------ schemas
const RECON = {
type: 'object',
required: ['summary', 'facts'],
properties: {
summary: { type: 'string' },
facts: { type: 'array', items: { type: 'string' } },
blockers: { type: 'array', items: { type: 'string' } },
},
}
const SPEC = {
type: 'object',
required: ['commonModule', 'poStackEdits', 'woStackEdits', 'alarmVariance', 'bedrockStatement', 'fallbackRateCalls', 'appAndPins', 'diffRules', 'notes'],
properties: {
commonModule: { type: 'string', description: 'the full cdk/common.py: every function (add_ddb_alarms + _DDB_ALARM_OPERATIONS, add_sender_auth_rejected_alarm, add_standard_lambda_alarms, make_bedrock_invoke_statement, make_email_bucket, make_processor_dlq, make_fallback_rate_alarm) with its EXACT signature, and the exact imports it needs (no stack-specific kms/ssm/event_sources per constraint 5)' },
poStackEdits: { type: 'string', description: 'cdk/po_stack.py: every inline block replaced by a common.* call, file:line, with the exact scope + construct-id + kwargs each call passes so the emitted resource is byte-identical; plus the PO CfnOutput additions (function ARNs + consumed table names)' },
woStackEdits: { type: 'string', description: 'cdk/wo_stack.py: same — call-site rewrites file:line preserving construct ids, plus WO CfnOutput additions; explicitly note wo web_ui gets NO alarms (constraint 3)' },
alarmVariance: { type: 'string', description: 'the per-function alarm-variance table proving each helper call reproduces exactly what the inline code emits today: PO p99 vs wo-email-processor p95, po-web-ui throttles+duration only, site_extractor no-DLQ, wo web_ui ZERO alarms, every bespoke description string mapped verbatim' },
bedrockStatement: { type: 'string', description: 'make_bedrock_invoke_statement: the exact actions + resources, showing how the inference-profile ARN and per-region FM ARNs are derived from Stack.of(scope).account/.region, and PROVING the derived strings resolve in-account to the SAME ARNs the two inline PolicyStatements hardcode today' },
fallbackRateCalls: { type: 'string', description: 'the make_fallback_rate_alarm call params for PO (rejected_included=False) and WO (rejected_included=True) reproducing the expression/FILL/label strings byte-for-byte; PLUS how the two DISTINCT rejected alarms stay distinct call sites (PO 6h count-floor IF(FILL(rej,0)>=1,...); WO 5-min/2-of-6 sparse) — NOT folded into make_fallback_rate_alarm, NO element-wise MAX' },
appAndPins: { type: 'string', description: 'app.py account= additions to both cdk.Environment calls; the exact constructs== pin (installed version); the stale "2.259.0" comment fix locations + new text; confirmation none perturbs a logical ID' },
diffRules: { type: 'string', description: 'exactly how Verify proves zero cdk diff: synth BASE and HEAD into separate temp dirs, diff the two stacks templates, ANY resource/logical-ID/property delta = FAIL, the ONLY allowed additions are the net-new CfnOutputs' },
notes: { type: 'string' },
},
}
const IMPL = {
type: 'object',
required: ['filesChanged', 'summary', 'checksRun'],
properties: {
filesChanged: { type: 'array', items: { type: 'string' } },
summary: { type: 'string' },
checksRun: { type: 'string' },
blockers: { type: 'array', items: { type: 'string' } },
},
}
const CHECKS = {
type: 'object',
required: ['passed', 'details'],
properties: {
passed: { type: 'boolean' },
details: { type: 'string' },
scopeViolations: { type: 'array', items: { type: 'string' } },
},
}
const DIFF = {
type: 'object',
required: ['passed', 'poDiffVerdict', 'woDiffVerdict', 'details'],
properties: {
passed: { type: 'boolean' },
poDiffVerdict: { type: 'string', description: 'po-ingest BASE-synth vs HEAD-synth: ZERO resource/logical-ID/property changes (CfnOutputs the only allowed net-new additions) — full template-diff evidence' },
woDiffVerdict: { type: 'string', description: 'workorder-ingest (artifact id, NOT WorkorderIngestStack): same zero-change evidence' },
details: { type: 'string' },
},
}
const FINDINGS = {
type: 'object',
required: ['findings'],
properties: {
findings: {
type: 'array',
items: {
type: 'object',
required: ['title', 'severity', 'confirmed', 'evidence', 'fix'],
properties: {
title: { type: 'string' },
severity: { enum: ['critical', 'high', 'medium', 'low'] },
confirmed: { type: 'boolean' },
evidence: { type: 'string' },
fix: { type: 'string' },
},
},
},
},
}
// ------------------------------------------------------------------- setup
phase('Setup')
const setup = await agent(`
In ${REPO}:
1. SEQUENCING GATE — Phases 0, 1, 2 AND 3 must all be on ${BASE} (Phase 4
dedups BOTH cdk stacks, which Phases 1 (po_stack alarms), 2 (bundling
roots) and 3 (shared cp) all edited — building against a pre-Phase-3
stack file guarantees a conflict and a wrong diff baseline). git fetch
origin, then pick the base ref: origin/${BASE} if that remote ref
exists, otherwise the local branch ${BASE} (a stacked local-only base is
expected and fine). Verify on the base ref:
(a) Phase 3: lambdas/shared/ exists
(git ls-tree <baseref> -- lambdas/shared | head);
(b) Phase 2: BOTH cdk/po_stack.py and cdk/wo_stack.py contain
Code.from_asset("../lambdas") for the email processors
(git show <baseref>:cdk/po_stack.py | grep -n '\\.\\./lambdas', same
for wo_stack.py);
(c) Phase 1: the po-email-processor-ai-fallback-rejected alarm /
EmailProcessorAiFallbackRejectedAlarm construct exists in po_stack.py
(git show <baseref>:cdk/po_stack.py | grep -n 'ai-fallback-rejected\\|AiFallbackRejected').
If Phase 3 is not on ${BASE}, STOP with a blocker (Phase 4 needs the
post-Phase-3 stack files as its zero-diff baseline). If any other phase
is missing, STOP with a blocker naming the unmet phase and do nothing
else.
2. Verify clean working tree (untracked .coverage / .claude/ / the local
lambdas/po/email_processor/package/ dir are fine; any OTHER dirt =
blocker, never stash or discard).
3. git checkout ${BASE}; then git pull --ff-only ONLY if the branch has an
upstream (a local-only base skips the pull — not a blocker); then
git checkout -b ${BRANCH}
4. gh pr list --state open --json number,title,headRefName (overlap check).
Return facts: HEAD sha, per-phase gate evidence, open PRs, blockers.
`, { label: 'setup:branch', model: 'haiku', schema: RECON })
if (!setup || (setup.blockers && setup.blockers.length)) {
return { aborted: 'setup blockers', blockers: setup ? setup.blockers : ['setup agent died'], facts: setup ? setup.facts : [] }
}
log(`Branch ${BRANCH} ready off ${BASE}. ${setup.summary}`)
// ------------------------------------------------------------------- recon
phase('Recon')
const recon = await parallel([
() => agent(`${PREAMBLE}
Read-only recon of the ~379 duplicated CDK lines and the PER-FUNCTION ALARM
VARIANCE that MUST survive the dedup (constraint 3). In cdk/po_stack.py and
cdk/wo_stack.py, quote verbatim with file:line:
1. _DDB_ALARM_OPERATIONS + add_ddb_alarms (both copies — are they
byte-identical? diff them).
2. add_sender_auth_rejected_alarm (both copies).
3. Every add_standard_lambda_alarms-shaped block: for EACH function
(po email-processor, wo email-processor, po web_ui, wo web_ui,
po site_extractor) list the exact alarm set, the duration statistic
(prove PO email p99 vs wo email p95), whether throttles/errors/dlq
alarms are present, and QUOTE every bespoke alarm description string.
CRITICALLY: confirm wo web_ui has ZERO alarms today.
4. make_email_bucket / make_processor_dlq shaped blocks (both copies), and
which functions get a DLQ (site_extractor has none).
5. The exact construct ids (2nd positional arg to every alarm / bucket /
dlq / statement construct) — these are the logical-ID roots that must be
passed UNCHANGED into the shared helpers.
30-40 precise facts. Any block that is NOT actually identical between
stacks (genuine drift) is a blocker to report, not to silently reconcile.`,
{ label: 'recon:duplicated-cdk', model: 'sonnet', phase: 'Recon', schema: RECON }),
() => agent(`${PREAMBLE}
Read-only recon of the two inline Bedrock IAM PolicyStatements (the
make_bedrock_invoke_statement source — constraint 2, the cross-family-review
surface). In both stacks quote verbatim with file:line: the full
iam.PolicyStatement (effect, actions, resources, conditions). For EACH
resource ARN record whether the account (328440206208) and region are
hardcoded or referenced, and enumerate every inference-profile ARN and every
per-region foundation-model ARN. Determine the EXACT list of regions / model
ids baked into the resources so the derived form (Stack.of(scope).account /
.region) can be proven to resolve to the identical strings in-account. Note
which principal/role each statement is attached to and how (add_to_role_policy
vs inline policy). 15-25 facts.`,
{ label: 'recon:bedrock-iam', model: 'sonnet', phase: 'Recon', schema: RECON }),
() => agent(`${PREAMBLE}
Read-only recon of the fallback-rate + rejected alarm math AS IT STANDS
POST-PHASE-1 (constraint 4). In both stacks quote verbatim with file:line:
1. PO's template-fallback-rate alarm: the full metric-math expression
string(s), every FILL(), every label, the period/threshold/
evaluation_periods/datapoints_to_alarm — and CONFIRM it EXCLUDES the
rejected series today (rejected_included=False).
2. WO's template-fallback-rate alarm: same, and CONFIRM it INCLUDES the
rejected series (rejected_included=True).
3. PO's ai-fallback-rejected alarm (added in Phase 1): confirm it is the
6h count-floor IF(FILL(rej,0)>=1,...) shape — quote the expression.
4. WO's rejected alarm: confirm it is the 5-min / 2-of-6 sparse idiom —
quote it. These two are DIFFERENT shapes and must stay distinct call
sites, NOT folded into make_fallback_rate_alarm.
5. Confirm NO element-wise MAX exists anywhere in either expression
(post-#102 rule).
Return the exact parameter values each make_fallback_rate_alarm call must
carry so Verify can prove byte-identity. 15-25 facts.`,
{ label: 'recon:fallback-alarms', model: 'sonnet', phase: 'Recon', schema: RECON }),
() => agent(`${PREAMBLE}
Read-only recon of the app.py / pins / outputs surface (constraint 6):
1. cdk/app.py: quote both cdk.Environment(...) calls verbatim with line
numbers (region-only today; account must be added).
2. The constructs dependency pin: quote cdk/requirements.txt line(s) and
find the stale "2.259.0" version comment(s) wherever they live (app.py,
stacks, requirements — grep the whole cdk/ tree) with file:line and the
actual installed constructs / aws-cdk-lib versions.
3. The five Lambda function construct variables in the stacks whose ARNs
need CfnOutputs (po email-processor, po web_ui, po site_extractor,
wo email-processor, wo web_ui) and the table constructs whose names are
consumed (purchase-orders, WorkOrders, and any comments table) — quote
the construct handles + ids so the CfnOutputs reference them correctly.
4. Any existing CfnOutput in either stack (WO reportedly has zero).
5. Confirm the wo artifact id is 'workorder-ingest' (the 2nd positional
arg to the Stack constructor in app.py), distinct from
stack_name='WorkorderIngestStack' (constraint 9).
15-25 facts.`,
{ label: 'recon:app-pins-outputs', model: 'haiku', phase: 'Recon', schema: RECON }),
])
const reconOk = recon.filter(Boolean)
const pack = reconOk.map(r => `## ${r.summary}\n${r.facts.join('\n')}`).join('\n\n')
const reconBlockers = reconOk.flatMap(r => r.blockers || [])
.filter(b => b && !/^\s*(none|n\/a)\b/i.test(b))
log(`Recon complete: ${reconOk.length}/4 mappers, ${reconBlockers.length} advisory notes`)
// Recon "blockers" for this phase are advisory design-notes / intended
// constraint-2 & 6 work items (derive the Bedrock ARN from Stack.of(scope),
// add account= to the environments, exact-pin constructs, fix the stale
// aws-cdk-lib version comment, decide the one common sender-auth docstring),
// NOT stop conditions — main-loop verified. The real gate is the ZERO cdk diff
// acceptance test in Verify. Fold the notes into the spec context instead of
// aborting so the spec agent must address each.
const reconAdvisories = reconBlockers.length
? `\n\nRECON ADVISORIES (recon flagged these; they are the intended constraint-2/6 work + a docstring choice, NOT drift that blocks dedup — resolve each per the constraints; the zero-cdk-diff test is the real acceptance gate):\n- ${reconBlockers.join('\n- ')}`
: ''
// -------------------------------------------------------------------- spec
phase('Spec')
const spec = await agent(`${PREAMBLE}
You are the SPEC agent — the single authority that pins every contested
decision BEFORE parallel implementation (parallel leaves cannot see each
other's choices). Using the recon pack below plus your own reads of the
actual files, produce the binding implementation spec:
- commonModule: the full cdk/common.py — every function per constraint 2
with its EXACT signature (add_standard_lambda_alarms keyword-only params
exactly as the doc pins them), and ONLY the imports the helpers need
(constraint 5 — no kms/ssm/event_sources). Every function is a PLAIN
function taking (scope, id, ...) — NO Construct subclass anywhere
(constraint 1).
- poStackEdits / woStackEdits: for each stack, the exact call-site rewrites
(file:line) mapping each inline block to a common.* call, passing the
SAME scope (the Stack) and the SAME construct id it uses today so every
logical ID is byte-stable; plus the CfnOutput additions (five function
ARNs split across the two stacks + consumed table names). State
explicitly that wo web_ui receives NO alarm call (constraint 3).
- alarmVariance: the per-function variance table (constraint 3) proving each
helper call reproduces today's emitted alarms EXACTLY — PO p99 vs wo-email
p95, po-web-ui throttles+duration only, site_extractor no-DLQ, wo web_ui
zero alarms, every bespoke description string mapped verbatim.
- bedrockStatement: make_bedrock_invoke_statement's actions + resources and
the derivation of the inference-profile / per-region FM ARNs from
Stack.of(scope).account/.region, with a proof table showing each derived
string equals the inline hardcoded ARN in-account (this is the
cross-family-review surface — be exhaustive).
- fallbackRateCalls: the make_fallback_rate_alarm call params for PO
(rejected_included=False) and WO (rejected_included=True) reproducing the
expression/FILL/label strings byte-for-byte, PLUS the plan for keeping the
two DISTINCT rejected alarms as separate call sites (PO 6h count-floor,
WO 5-min/2-of-6) — NOT folded, NO element-wise MAX (constraint 4).
- appAndPins: app.py account= on both Environment calls; the exact
constructs== pin; the stale "2.259.0" comment fix (file:line + new text);
confirmation each is logical-ID-neutral.
- diffRules: exactly how Verify proves ZERO cdk diff — synth ${BASE} and
HEAD each into a separate temp dir, diff both stacks' templates, ANY
resource/logical-ID/property delta = FAIL, the ONLY allowed net-new is
the CfnOutputs; drive synth/diff by artifact id (po-ingest /
workorder-ingest, constraint 9).
Recon pack:\n${pack}${reconAdvisories}`,
{ label: 'spec:pin-common', phase: 'Spec', schema: SPEC })
if (!spec) return { aborted: 'spec agent died — rerun workflow', reconBlockers }
const specBlock = `BINDING SPEC (from the spec agent — implement EXACTLY this):\n${JSON.stringify(spec, null, 2)}`
log('Spec pinned: common.py contents, per-stack rewrites, alarm variance, Bedrock IAM, fallback-rate calls, app/pins/outputs')
// --------------------------------------------------------------- implement
phase('Implement')
const impl = await parallel([
() => agent(`${PREAMBLE}
YOU OWN: cdk/common.py ONLY (create it). Do not touch po_stack.py,
wo_stack.py, app.py, requirements.txt, README, tests, or anything under
lambdas/.
Task: author cdk/common.py per spec.commonModule EXACTLY — every function
(add_ddb_alarms + _DDB_ALARM_OPERATIONS, add_sender_auth_rejected_alarm,
add_standard_lambda_alarms with the pinned keyword-only signature,
make_bedrock_invoke_statement deriving ARNs from Stack.of(scope).account/
.region, make_email_bucket, make_processor_dlq, make_fallback_rate_alarm)
as a PLAIN function taking (scope, id, ...) — NEVER a Construct subclass
(constraint 1). Import ONLY what the helpers need — no kms/ssm/
event_sources (constraint 5). Match the fallback-rate expression/FILL/label
strings byte-for-byte (constraint 4). Do NOT put the two rejected alarms in
make_fallback_rate_alarm.
Run before returning: ruff check cdk/common.py && ruff format cdk/common.py
--check, plus a py_compile import smoke (python3 -c "import common" from
cdk/ with the CDK venv). Full synth is the stacks agent's job — but if you
can import common cleanly, report it.
${specBlock}`,
{ label: 'impl:common-module', model: 'opus', phase: 'Implement', schema: IMPL }),
() => agent(`${PREAMBLE}
YOU OWN: cdk/po_stack.py and cdk/wo_stack.py ONLY. Do not touch
cdk/common.py (another agent authors it), app.py, requirements.txt, README,
or lambdas/.
Task: apply spec.poStackEdits + spec.woStackEdits — replace each inline
block with the matching common.* call, passing the SAME scope (the Stack
instance, NOT a new Construct) and the SAME construct id used today so every
logical ID is byte-stable (constraint 1). Preserve every per-function alarm
variance (constraint 3): PO email p99 / wo email p95, po-web-ui throttles+
duration only, site_extractor no-DLQ, wo web_ui gets NO alarm call, bespoke
descriptions passed verbatim. Wire the fallback-rate calls per
spec.fallbackRateCalls (PO rejected_included=False, WO True) and keep the
two rejected alarms as distinct call sites (constraint 4). Add the CfnOutputs
per spec (function ARNs + consumed table names) — additive, ID-safe.
Import from common (bare 'import common' / 'from common import ...' — cdk/
is on sys.path via app.py's imports). Touch nothing else (tables, KMS, SSM,
event sources, the RETAIN policies stay byte-identical).
Run before returning: ruff check cdk && cd cdk &&
npx cdk synth po-ingest -q -o /tmp/phase4-synth &&
npx cdk synth workorder-ingest -q -o /tmp/phase4-synth (artifact-id
selectors, NOT stack_name — constraint 9). If cdk/common.py has not landed
yet the synth fails on the missing import — poll by re-running up to ~10 min
before reporting a blocker. Confirm both stacks synth and note that the
ONLY template delta vs ${BASE} is the net-new CfnOutputs (spot-check a
couple of alarm logical IDs are unchanged).
${specBlock}`,
{ label: 'impl:cdk-stacks', model: 'opus', phase: 'Implement', schema: IMPL }),
() => agent(`${PREAMBLE}
YOU OWN: cdk/app.py, cdk/requirements.txt and README.md ONLY. Do not touch
common.py, the stacks, tests, or lambdas/.
Task A: apply spec.appAndPins — add account='328440206208' to BOTH
cdk.Environment calls in app.py, pin constructs== to the exact installed
version in cdk/requirements.txt, and fix the stale "2.259.0" comment(s) at
the file:line spec.appAndPins gives (only those in files you own — if a
stale comment lives in a stack file, note it for the stacks agent, do NOT
edit their file). Every edit here must be logical-ID-neutral (account on
Environment does not change resource logical IDs; verify in your summary).
Task B: README — document cdk/common.py in the CDK/architecture section
(the shared plain-function helpers, the logical-ID-safety rule, the
account/region-derived Bedrock ARN, the new CfnOutputs), and note the
account is now explicit on both stacks. Match existing README style. If the
README documents the stacks' resource inventory, keep it accurate.
Run before returning: ruff check cdk (app.py) and a py_compile of app.py;
you cannot run the full synth without the stacks agent's edits — if you want
to smoke-test, poll cd cdk && npx cdk synth po-ingest -q up to ~10 min, but
a clean app.py parse is sufficient for your scope.
${specBlock}`,
{ label: 'impl:app-pins-readme', model: 'sonnet', phase: 'Implement', schema: IMPL }),
])
const implOk = impl.filter(Boolean)
const implBlockers = implOk.flatMap(r => r.blockers || [])
log(`Implement complete: ${implOk.length}/3 agents, blockers: ${implBlockers.length}`)
// ---------------------------------------------------- verify + fix loop
const EXPECTED_SCOPE = [
'cdk/common.py',
'cdk/po_stack.py',
'cdk/wo_stack.py',
'cdk/app.py',
'cdk/requirements.txt',
'README.md',
'tests/',
]
const mechanicalPrompt = `${PREAMBLE}
Independent re-verification — trust nothing self-reported. Run ALL gates,
quoting failures verbatim:
1. pytest -q --no-cov (repo root — all suites green; a synth-only cdk-diff
test may now exist under tests/)
2. ruff check . && ruff format --check .
3. cd cdk && npx cdk synth po-ingest -q && npx cdk synth workorder-ingest -q
(artifact-id selectors, NOT stack_name — constraint 9)
4. ZERO-CODE invariant (constraint 7): git diff ${BASE}...HEAD -- lambdas/
must output NOTHING.
5. NO CONSTRUCT SUBCLASS (constraint 1): grep cdk/common.py for
'class .*Construct' / 'class .*(Construct)' — there must be NONE; every
extracted symbol is a plain 'def'. Report any subclass as a hard FAIL.
6. cdk/common.py imports NO kms/ssm/event_sources (constraint 5) — grep and
confirm.
7. Both cdk.Environment calls in app.py carry account='328440206208';
constructs== is an exact pin (not >=); no "2.259.0" stale comment
remains (grep the cdk/ tree).
8. CfnOutputs: five function ARNs + the consumed table names are present
across the two stacks (grep CfnOutput).
9. git status --porcelain scope check: every modified/added path under
${EXPECTED_SCOPE.join(', ')} (untracked .coverage/.claude/package/
tolerated). No lambdas/ or non-cdk-diff tests/ change.
passed=true only if all green. YOU MAY NOT edit files.`
const diffPrompt = `${PREAMBLE}
You are the ZERO-CDK-DIFF verifier — the load-bearing gate of this phase
(the doc names it the acceptance test). Everything is local synth-vs-synth.
1. From a clean worktree state, synth the BASE templates: check out (via
git worktree add or git stash-free 'git show'-based synth — prefer
'git worktree add /tmp/phase4-base ${BASE}' so HEAD is untouched), then
in that BASE tree cd cdk && npx cdk synth po-ingest -q -o
/tmp/phase4-diff-base && npx cdk synth workorder-ingest -q -o
/tmp/phase4-diff-base.
2. Synth the HEAD templates: in the working tree cd cdk && npx cdk synth
po-ingest -q -o /tmp/phase4-diff-head && npx cdk synth workorder-ingest
-q -o /tmp/phase4-diff-head. (Both by ARTIFACT ID, constraint 9.)
3. Diff the two CloudFormation templates per stack
(/tmp/phase4-diff-base/<stack>.template.json vs
/tmp/phase4-diff-head/<stack>.template.json). Normalize only CDK-tooling
noise (the CDKMetadata Analytics string, asset-hash-derived S3Key values
that were ALSO equal on BASE). Then judge:
- ZERO logical-ID changes (no renamed/removed/added Resources except the
net-new CfnOutputs).
- ZERO alarm property deltas (thresholds, statistics p99/p95, expression
strings, FILL, labels, evaluation_periods, datapoints_to_alarm).
- ZERO IAM deltas: the Bedrock PolicyStatement actions/resources must be
byte-identical after the account/region derivation resolves in-account.
- ZERO DynamoDB table / bucket / DLQ / env / runtime deltas.
- The ONLY allowed net-new is the Outputs block (the five function ARNs
+ consumed table names).
ANY delta outside that allowance = FAIL. Paste the actual per-stack
template diff (or 'identical' with the normalized-noise list).
4. Cross-check with the live tool: cd cdk && npx cdk diff po-ingest ;
npx cdk diff workorder-ingest against the deployed state must also show
only the CfnOutput additions (network permitting; if AWS creds are
read-only-absent, the BASE-vs-HEAD template diff in steps 1-3 is
authoritative).
5. Clean up any /tmp worktrees you created (git worktree remove).
passed=true only if BOTH stacks show zero resource change beyond the
CfnOutput additions.`
const lenses = [
{ key: 'logical-id-safety', prompt: `${PREAMBLE}
ADVERSARIAL REVIEW — logical-ID-safety lens (the load-bearing rule,
constraint 1). Try to prove a construct-id or tree-shape change slipped in.
(1) Read cdk/common.py: is EVERY extracted symbol a plain 'def' taking
(scope, id, ...)? Any 'class X(Construct)' / Construct subclass / nested
Construct is an automatic CRITICAL — a subclass reparents the tree and
would attempt REPLACEMENT of the RETAIN-protected purchase-orders /
WorkOrders tables and the named buckets. (2) For every rewritten call site
in both stacks, prove the scope passed is the Stack instance (self), NOT a
new intermediate construct, and the construct id string is IDENTICAL to the
BASE inline id (git diff ${BASE} the id strings). (3) Synth BASE and HEAD
and diff the full Resources logical-ID SET — it must be identical (only
Outputs added). Any renamed/removed logical ID = confirmed CRITICAL.
(4) Confirm the RETAIN removal policies on the tables and the bucket
names/policies are byte-identical post-refactor. confirmed=true only with a
concrete logical-ID delta or a subclass-smell file:line.` },
{ key: 'alarm-variance', prompt: `${PREAMBLE}
ADVERSARIAL REVIEW — alarm-variance lens (constraint 3). The shared helpers
must NOT homogenize the deliberate per-function differences. Read
cdk/common.py + every helper call site + the synthesized templates' alarms.
Prove each of these survived EXACTLY: (1) PO email-processor duration alarm
uses p99, wo-email-processor uses p95 — quote both from the synthesized
templates. (2) po-web-ui has ONLY throttles+duration alarms (no errors/dlq
beyond what it had). (3) site_extractor has NO DLQ alarm. (4) wo web_ui has
ZERO alarms — prove the shared helper did NOT silently add any (search the
wo template for any alarm whose dimensions point at the wo web_ui
function). (5) Every bespoke alarm description string is byte-identical to
BASE (diff the AlarmDescription fields). (6) Fallback-rate: PO excludes the
rejected series (rejected_included=False), WO includes it; the two rejected
alarms kept their distinct shapes (PO 6h count-floor, WO 5-min/2-of-6); NO
element-wise MAX anywhere. confirmed=true only with a template-level diff
showing a homogenized or dropped alarm.` },
{ key: 'iam-equivalence', prompt: `${PREAMBLE}
ADVERSARIAL REVIEW — IAM-equivalence lens (the cross-family-review surface).
make_bedrock_invoke_statement moved the PolicyStatement construction and
swapped hardcoded 328440206208 for Stack.of(scope).account/.region. Prove
the produced IAM is IDENTICAL. (1) Synth both stacks and extract the Bedrock
PolicyStatement from each template; diff actions and resources against
${BASE}'s synthesized statements — the resolved ARN strings (account +
region substituted) must be byte-identical in-account. (2) Confirm the
resources still enumerate the SAME inference-profile ARN and the SAME
per-region foundation-model ARNs (no region dropped/added, no wildcard
broadening). (3) Confirm effect/conditions/principal attachment unchanged
and the statement is attached to the SAME role. (4) Flag any broadening
(e.g. a Ref/Sub that resolves to a wildcard, or Stack.region producing a
different region than the hardcoded one) as confirmed HIGH. confirmed=true
only with the two synthesized statements diffed.` },
]
let round = 0
let checks = null
let diff = null
let confirmed = []
while (round < 3) {
phase('Verify')
const results = await parallel([
() => agent(mechanicalPrompt, { label: `verify:mechanical-r${round}`, model: 'sonnet', phase: 'Verify', schema: CHECKS }),
() => agent(diffPrompt, { label: `verify:cdk-diff-r${round}`, model: 'opus', phase: 'Verify', schema: DIFF }),
...lenses.map(l => () =>
agent(l.prompt, { label: `verify:${l.key}-r${round}`, phase: 'Verify', schema: FINDINGS })),
])
checks = results[0]
diff = results[1]
confirmed = results.slice(2).filter(Boolean)
.flatMap(r => r.findings || [])
.filter(f => f.confirmed && f.severity !== 'low')
const green = checks && checks.passed && diff && diff.passed
log(`Verify round ${round}: mechanical ${checks && checks.passed ? 'GREEN' : 'RED'}, cdk-diff ${diff && diff.passed ? 'GREEN' : 'RED'}, confirmed findings: ${confirmed.length}`)
if (green && confirmed.length === 0) break
round += 1
if (round >= 3) break
phase('Fix')
await agent(`${PREAMBLE}
You are the fix agent — you may edit files under: ${EXPECTED_SCOPE.join(', ')}.
Fix EVERY item below minimally; the binding spec and 10 pinned constraints
still hold (a finding that conflicts with a constraint is reported, not
"fixed" — the constraint wins, esp. constraint 1's plain-function /
no-Construct-subclass rule, constraint 3's alarm variance, and constraint 4's
distinct rejected alarms / no element-wise MAX). Re-run the specific failing
gate per fix (the zero-cdk-diff check is authoritative — a fix that
introduces ANY logical-ID or property delta is worse than the finding).
MECHANICAL:\n${checks ? checks.details : '(agent died — rerun all gates)'}
CDK-DIFF:\n${diff ? diff.details : '(agent died — rerun all)'}
CONFIRMED FINDINGS:\n${JSON.stringify(confirmed, null, 2)}
${specBlock}`,
{ label: `fix:round-${round}`, model: 'opus', phase: 'Fix', schema: IMPL })
}
const verifyClean = checks && checks.passed && diff && diff.passed && confirmed.length === 0
if (!verifyClean) {
return {
status: 'NEEDS ATTENTION — verify not clean after 3 rounds; branch left uncommitted',
branch: BRANCH,
mechanical: checks,
cdkDiff: diff,
unresolvedFindings: confirmed,
implBlockers,
reconBlockers,
spec,
}
}
// ----------------------------------------------------------------- package
phase('Package')
const commit = await agent(`${PREAMBLE.replace('do NOT commit, ', '')}
YOU are the commit agent:
1. MANDATORY GPT-4.1 CROSS-FAMILY REVIEW (the doc mandates it for the
make_bedrock_invoke_statement IAM PolicyStatement move, even though
semantics are identical). Capture the Bedrock-statement diff first:
git diff ${BASE}...HEAD -- cdk/common.py cdk/po_stack.py cdk/wo_stack.py
(isolate the make_bedrock_invoke_statement + its two call sites), then
RUN inline:
python3 ~/Documents/repositories/seahaven/security-review/cross_review.py
"Review this CDK IAM PolicyStatement move for breaking changes: the two
inline Bedrock invoke PolicyStatements (hardcoded account 328440206208)
were consolidated into make_bedrock_invoke_statement in cdk/common.py,
deriving the inference-profile + per-region foundation-model ARNs from
Stack.of(scope).account/.region. Confirm the produced actions/resources
are byte-identical in-account and no privilege broadening. <paste diff>"
Record the verdict verbatim in your summary. If cross_review.py is
unavailable, DO NOT block the local commit but flag the review as
OUTSTANDING in your summary (it must be run before merge).
2. Read ~/Documents/repositories/seahaven/engineering-handbook/commit-messages.md
and follow it exactly.
3. git add only paths under: ${EXPECTED_SCOPE.join(', ')} and
.claude/workflows/phase-4-cdk-common.js. NOT .coverage, NOT package/,
NOT cdk.out. Verify the staged set with git status.
4. ONE commit; write the message to /tmp/phase4-commit-msg.txt and use
git commit -F /tmp/phase4-commit-msg.txt (backticks in -m get eaten by
zsh). Suggested subject:
"feat: extract cdk/common.py — dedup 379 CDK lines as logical-ID-safe plain helpers (refactor phase 4)"
Body: the plain-function (scope, id, ...) approach and WHY no Construct
subclass (RETAIN-table replacement), the account/region-derived Bedrock
ARN, the zero-cdk-diff acceptance evidence for both stacks, the
account=/constructs pin/stale-comment/CfnOutput net-new additions, and
the cross_review.py verdict one-liner. NO AI attribution / Co-Authored-By
lines.
5. Do NOT push. Return commit sha + shortstat + the cross_review.py verdict
in summary.`,
{ label: 'package:commit', model: 'sonnet', phase: 'Package', schema: IMPL })
return {
status: 'BUILT — committed locally, NOT pushed',
branch: BRANCH,
base: BASE,
commit: commit ? commit.summary : 'commit agent died — commit manually',
spec: { commonModule: spec.commonModule, bedrockStatement: spec.bedrockStatement, fallbackRateCalls: spec.fallbackRateCalls, appAndPins: spec.appAndPins, notes: spec.notes },
diffEvidence: diff ? { po: diff.poDiffVerdict, wo: diff.woDiffVerdict } : null,
implementation: implOk.map(r => r.summary),
filesChanged: implOk.flatMap(r => r.filesChanged),
verifyRounds: round + 1,
blockers: implBlockers.concat(reconBlockers),
outstandingGates: [
'MANDATORY cross-family GPT-4.1 review (cross_review.py) on the make_bedrock_invoke_statement IAM PolicyStatement move — the Package agent runs it inline and records the verdict; confirm that verdict is clean (or re-run) before merge. If cross_review.py was unavailable at commit time, this review is OUTSTANDING — do not merge without it.',
'/sh-security-review NOT required (pure CDK refactor — no untrusted-input/auth-logic change; the pre-push scanners still run as the unattended backstop)',
'push + PR + gh pr checks green',
'deploy-then-merge with cdk diff zero-change on BOTH stacks as the live acceptance signal: deploy from branch, confirm cdk diff po-ingest / cdk diff workorder-ingest show only the CfnOutput additions, both stacks reach UPDATE_COMPLETE, all alarms still OK, THEN merge (a no-op resource redeploy is itself the verification signal). Drive synth/diff by artifact id workorder-ingest, NOT stack_name WorkorderIngestStack (constraint 9).',
'update the Confluence "AWS Architecture Map" if the new CfnOutputs / account-explicit envs change the documented resource inventory',
],
}

19
README.md
View file

@ -88,7 +88,7 @@ Amazon APM work order emails (from Hexagon EAM / HxGN SmartCloud) are received a
## Architecture
**IaC:** AWS CDK (Python), two stacks in one app, region `us-east-1`. The `cdk.Environment` is deliberately **account-agnostic** (region-only, no `account=`): the stacks deploy to whichever account the deploy credentials target (`328440206208` today), and every account-derived template value — bucket names, `Lambda::Permission` source account, the site-alerts SNS action ARN, the Bedrock ARN below — renders as the CloudFormation `AWS::AccountId` pseudo-parameter rather than a literal. Pinning `account=` was evaluated in Phase 4 and rejected: it would resolve those tokens to literals, and against the deployed (account-agnostic) templates CloudFormation flags the `RemovalPolicy.RETAIN` email buckets as requiring replacement — a data-loss risk — for no functional gain.
**IaC:** AWS CDK (Python), two stacks in one app, region `us-east-1`.
All Lambdas: Python 3.12, ARM64, 60-day log retention.
@ -98,18 +98,6 @@ All Lambdas: Python 3.12, ARM64, 60-day log retention.
**SES:** Both stacks add rules to the shared `INBOUND_MAIL` receipt rule set on `int.seahaven.com`.
### Shared CDK helpers (`cdk/common.py`, Phase 4)
The ~379 lines that `po_stack.py` and `wo_stack.py` both defined identically (DynamoDB alarms, the sender-auth-rejected metric filter + alarm, the standard per-Lambda alarm set, the Bedrock `InvokeModel` grant, the raw-email bucket, the async-invoke DLQ, and the template-fallback-rate math alarm) are collapsed into `cdk/common.py`.
**Logical-ID-safety rule (load-bearing).** Every helper is a **plain function** taking `(scope, id, ...)` — never a `Construct` subclass. Each stack calls a helper with its **own Stack instance as `scope`** and the **exact same literal construct id** it used inline before the extraction, so every synthesized logical ID is byte-stable. A `Construct` subclass would insert an extra tree node, reparent every child's logical ID, and CloudFormation would attempt to **replace** the `RemovalPolicy.RETAIN`-protected `purchase-orders` / `WorkOrders` / `WorkOrderComments` tables and the named S3 buckets — a data-loss event. Nothing in `common.py` subclasses `Construct`, and it imports only the CDK constructs its helpers touch (`dynamodb`, `cloudwatch`/`cw_actions`, `iam`, `logs`, `s3`, `sqs` — no `kms`, `ssm`, or Lambda event-source imports).
Extracted helpers: `add_ddb_alarms`, `add_sender_auth_rejected_alarm`, `add_standard_lambda_alarms` (bespoke `descriptions=` dict passed through verbatim per call site — no alarm text is generated or homogenized), `make_bedrock_invoke_statement`, `make_email_bucket`, `make_processor_dlq`, `make_fallback_rate_alarm`. Per-function alarm variance is preserved exactly through call-site arguments, not baked into the helpers: PO's email-processor duration alarm uses `p99`, WO's uses `p95`; `po-web-ui` gets throttles + duration only (no errors, no DLQ); `po-ingest-site-extractor` has no DLQ alarm (it's a DynamoDB-stream consumer, not async-invoked); `workorder-web-ui` gets **zero** alarms — the helper is simply never called for it, so it cannot silently add any. The two "AI-fallback-rejected" alarms (PO's 6-hour count-floor `IF(FILL(rej,0)>=1,...)`, WO's 5-minute/2-of-6 sparse idiom) are a different shape from `make_fallback_rate_alarm` and stay as distinct inline call sites in each stack rather than being forced through the shared helper.
**Bedrock ARN, now account/region-derived.** `make_bedrock_invoke_statement(scope)` builds the inference-profile ARN and the `us-east-1` foundation-model ARN from `Stack.of(scope).account` / `.region` instead of the previous hardcoded `328440206208`/`us-east-1` literals (the `us-east-2`/`us-west-2` foundation-model ARNs stay literal — they're fixed cross-region reach targets, not the stack's own region). Because the environment is account-agnostic (above), `stack.account` is the `AWS::AccountId` pseudo-parameter, so the derived ARN synthesizes as an `Fn::Sub`/`Ref` that **resolves at deploy time to the same ARN** the hardcoded literal named in-account. Versus the deployed stack this is a benign **in-place** IAM policy update (IAM policies never require replacement) — and it makes the grant account-portable instead of pinned to the management account. This IAM PolicyStatement change is the surface the mandatory GPT-4.1 cross-family review covers.
**New `CfnOutput`s.** Each stack now exports its Lambda function ARNs and the DynamoDB table names it owns/consumes, all net-new/additive (no existing output changes): `po-ingest` — `EmailProcessorFunctionArn`, `WebUiFunctionArn`, `SiteExtractorFunctionArn`, `PurchaseOrdersTableName`, `PendingSiteReviewTableName` (the existing `VerifiedSitesTableName` output is unchanged); `workorder-ingest` — `EmailProcessorFunctionArn`, `WebUiFunctionArn`, `WorkOrdersTableName`, `WorkOrderCommentsTableName`.
### Sender authentication (INFRA-107)
The `From` header and any `Authentication-Results` header inside the raw MIME are attacker-forgeable, so neither is trusted. Instead, both email processors authenticate the sender against the verdicts SES itself stamps at delivery time, failing closed. The authenticator is **single-sourced** at `lambdas/shared/ses_auth.py` (Phase 3) — previously duplicated byte-for-byte in each pipeline's `email_processor/` dir and kept in sync by a fixture-hygiene test; now one copy, so a future hardening fix to the fail-closed logic lands **once** instead of needing two identical edits. The CDK bundling `cp`s it flat beside each handler so the handlers' unchanged `from ses_auth import authenticate_inbound_email` still resolves at runtime (see [`lambdas/shared/`](#deploy-pipeline-guards-phase-0)). The gate:
@ -378,10 +366,7 @@ python scripts/backfill_sites.py
```
cdk/
app.py # Two stacks: po-ingest + WorkorderIngestStack (region-only env)
common.py # Phase 4: shared plain-function CDK helpers (alarms, Bedrock grant,
# email bucket, processor DLQ, fallback-rate alarm) -- called with
# each stack's own scope + literal construct ids, logical-ID-safe
app.py # Two stacks: po-ingest + WorkorderIngestStack
po_stack.py # Purchase order pipeline resources
wo_stack.py # Work order pipeline resources
lambdas/ # Phase 2: shared Code.from_asset("../lambdas") bundling root for

331
cdk/common.py
View file

@ -1,331 +0,0 @@
"""Shared CDK helpers for the procurement-ingest stacks (Phase 4).
Plain free functions extracted from po_stack.py / wo_stack.py. Each takes the
same Stack `scope` and the same literal construct `id` the stacks passed inline,
so every synthesized logical ID is byte-stable. NOTHING here is a Construct
subclass: a subclass would insert a tree node and reparent/replace the
RETAIN-protected tables and named buckets.
"""
from aws_cdk import (
Duration,
RemovalPolicy,
Stack,
aws_cloudwatch as cloudwatch,
aws_cloudwatch_actions as cw_actions,
aws_dynamodb as dynamodb,
aws_iam as iam,
aws_logs as logs,
aws_s3 as s3,
aws_sqs as sqs,
)
# --- DynamoDB alarm operations (moved verbatim from both stacks) ---
_DDB_ALARM_OPERATIONS = [
dynamodb.Operation.GET_ITEM,
dynamodb.Operation.BATCH_GET_ITEM,
dynamodb.Operation.QUERY,
dynamodb.Operation.SCAN,
dynamodb.Operation.PUT_ITEM,
dynamodb.Operation.UPDATE_ITEM,
dynamodb.Operation.DELETE_ITEM,
dynamodb.Operation.BATCH_WRITE_ITEM,
]
# CloudWatch namespace for the log-derived sender-authentication metrics.
_SENDER_AUTH_METRIC_NAMESPACE = "Seahaven/ProcurementIngest"
def add_ddb_alarms(scope, id_prefix, table, alarm_name_prefix, alarm_topic):
"""Add throttle + system-error alarms for a DynamoDB table.
Both fire on any non-zero datapoint in a 5-min window. ALARM-only SnsAction
to site-alerts (no OK action); TreatMissingData NOT_BREACHING.
"""
table.metric_throttled_requests_for_operations(
operations=_DDB_ALARM_OPERATIONS,
period=Duration.minutes(5),
statistic="Sum",
).create_alarm(
scope,
f"{id_prefix}ThrottlesAlarm",
alarm_name=f"{alarm_name_prefix}-throttles",
alarm_description=f"{alarm_name_prefix} DynamoDB throttled requests",
threshold=0,
evaluation_periods=1,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
table.metric_system_errors_for_operations(
operations=_DDB_ALARM_OPERATIONS,
period=Duration.minutes(5),
statistic="Sum",
).create_alarm(
scope,
f"{id_prefix}SystemErrorsAlarm",
alarm_name=f"{alarm_name_prefix}-system-errors",
alarm_description=f"{alarm_name_prefix} DynamoDB server-side (5xx) errors",
threshold=0,
evaluation_periods=1,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
def add_sender_auth_rejected_alarm(scope, id_prefix, function_name, alarm_topic):
"""Metric-filter + alarm on ``sender_auth_rejected`` warnings (INFRA-107).
A rejected inbound email is skipped without erroring the invocation, so it
is invisible to the Errors/Throttles/DLQ alarms. This turns the structured
warning log into a CloudWatch metric and pages when rejections spike --
catching a silent false-reject storm (allowlist wrong, signing-domain
drift, SES header-format change) that would otherwise discard legitimate
mail while the pipeline reports healthy.
ALARM-only SnsAction to site-alerts; no OK action. The metric filter reads
the function's own log group (imported by the deterministic
``/aws/lambda/<fn>`` name, created by the function's log_retention). A plain
substring pattern is used because Lambda prefixes each line with its own
level/timestamp/request-id, so the JSON payload is not a standalone JSON
log event a `{$.event=...}` pattern could match.
"""
metric_name = f"{function_name}-sender-auth-rejected"
logs.MetricFilter(
scope,
f"{id_prefix}SenderAuthRejectedFilter",
log_group=logs.LogGroup.from_log_group_name(
scope,
f"{id_prefix}LogGroup",
f"/aws/lambda/{function_name}",
),
filter_pattern=logs.FilterPattern.literal('"sender_auth_rejected"'),
metric_namespace=_SENDER_AUTH_METRIC_NAMESPACE,
metric_name=metric_name,
metric_value="1",
default_value=0,
)
cloudwatch.Metric(
namespace=_SENDER_AUTH_METRIC_NAMESPACE,
metric_name=metric_name,
period=Duration.minutes(5),
statistic="Sum",
).create_alarm(
scope,
f"{id_prefix}SenderAuthRejectedAlarm",
alarm_name=f"{function_name}-sender-auth-rejected",
alarm_description=(
f"{function_name} rejected inbound mail on sender authentication "
"(possible allowlist/DKIM-domain drift silently dropping real mail)"
),
threshold=1,
evaluation_periods=6,
datapoints_to_alarm=2,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_OR_EQUAL_TO_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
def add_standard_lambda_alarms(
scope,
id_prefix,
fn,
name_prefix,
topic,
*,
duration_statistic,
errors=True,
dlq=None,
descriptions,
):
"""Standard per-Lambda alarm set: errors (optional), throttles, dlq
(optional), duration. Every alarm bespoke-described via `descriptions`
(keys: errors/throttles/dlq/duration passed through VERBATIM). Construct
ids are f"{id_prefix}<Kind>Alarm", alarm names f"{name_prefix}-<kind>",
exactly the inline literals. Order of creation is irrelevant to logical IDs
(ids are explicit) so the fixed errors->throttles->dlq->duration order here
reproduces both PO (dlq before duration) and WO (duration before dlq)
templates identically.
"""
if errors:
fn.metric_errors(period=Duration.minutes(5), statistic="Sum").create_alarm(
scope,
f"{id_prefix}ErrorsAlarm",
alarm_name=f"{name_prefix}-errors",
alarm_description=descriptions["errors"],
threshold=0,
evaluation_periods=1,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(topic))
fn.metric_throttles(period=Duration.minutes(5), statistic="Sum").create_alarm(
scope,
f"{id_prefix}ThrottlesAlarm",
alarm_name=f"{name_prefix}-throttles",
alarm_description=descriptions["throttles"],
threshold=0,
evaluation_periods=1,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(topic))
if dlq is not None:
dlq.metric_approximate_number_of_messages_visible(
period=Duration.minutes(5),
statistic="Maximum",
).create_alarm(
scope,
f"{id_prefix}DlqMessagesAlarm",
alarm_name=f"{name_prefix}-dlq-messages",
alarm_description=descriptions["dlq"],
threshold=0,
evaluation_periods=1,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(topic))
fn.metric_duration(
period=Duration.minutes(5),
statistic=duration_statistic,
).create_alarm(
scope,
f"{id_prefix}DurationAlarm",
alarm_name=f"{name_prefix}-duration",
alarm_description=descriptions["duration"],
threshold=45000,
evaluation_periods=3,
datapoints_to_alarm=2,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_OR_EQUAL_TO_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(topic))
def make_bedrock_invoke_statement(scope):
"""Return the Bedrock InvokeModel PolicyStatement for the email processor.
The us.* inference profile can route cross-region, so the grant covers the
inference-profile ARN plus the per-region foundation-model ARNs
(us-east-1/us-east-2/us-west-2). ARN #1 account and ARN #1/#2 region are
DERIVED from Stack.of(scope).account/.region (not hardcoded 328440206208);
ARN #3/#4 regions (us-east-2/us-west-2) are cross-region reach targets, NOT
the stack's own region, so they stay literal. Caller attaches via
fn.add_to_role_policy(...) on the SAME function -> logical-ID-safe.
"""
stack = Stack.of(scope)
account = stack.account
region = stack.region
return iam.PolicyStatement(
actions=[
"bedrock:InvokeModel",
"bedrock:InvokeModelWithResponseStream",
],
resources=[
f"arn:aws:bedrock:{region}:{account}:inference-profile/us.anthropic.claude-haiku-4-5-20251001-v1:0",
f"arn:aws:bedrock:{region}::foundation-model/anthropic.claude-haiku-4-5-20251001-v1:0",
"arn:aws:bedrock:us-east-2::foundation-model/anthropic.claude-haiku-4-5-20251001-v1:0",
"arn:aws:bedrock:us-west-2::foundation-model/anthropic.claude-haiku-4-5-20251001-v1:0",
],
)
def make_email_bucket(scope, id, name_prefix):
"""Raw-email S3 bucket: BLOCK_ALL public, RETAIN, 90-day expiration.
bucket_name = f"{name_prefix}-{account}" (account derived from the stack),
reproducing f"po-ingest-emails-{self.account}" /
f"workorder-ingest-emails-{self.account}" exactly.
"""
return s3.Bucket(
scope,
id,
bucket_name=f"{name_prefix}-{Stack.of(scope).account}",
block_public_access=s3.BlockPublicAccess.BLOCK_ALL,
removal_policy=RemovalPolicy.RETAIN,
lifecycle_rules=[
s3.LifecycleRule(expiration=Duration.days(90)),
],
)
def make_processor_dlq(scope, id):
"""Async-invoke DLQ: 14-day retention, enforce_ssl. CDK-generated name."""
return sqs.Queue(
scope,
id,
retention_period=Duration.days(14),
enforce_ssl=True,
)
def make_fallback_rate_alarm(
scope,
id,
*,
namespace,
alarm_topic,
alarm_name,
alarm_description,
rejected_included,
period,
threshold,
floor,
evaluation_periods,
datapoints_to_alarm,
):
"""Template-fallback-rate MathExpression alarm.
rejected_included=False (PO): numerator FILL(fb,0), denom fb+tmpl.
rejected_included=True (WO): numerator (fb+rej), denom fb+rej+tmpl.
Expression string, FILL, label reproduced BYTE-FOR-BYTE. GREATER_THAN,
NOT_BREACHING. NO element-wise MAX (post-#102). The two DISTINCT rejected
alarms are NOT built here.
"""
fb_metric = cloudwatch.Metric(
namespace=namespace,
metric_name="ParseOutcome",
dimensions_map={"ParseMethod": "ai_fallback"},
statistic="Sum",
period=period,
)
tmpl_metric = cloudwatch.Metric(
namespace=namespace,
metric_name="ParseOutcome",
dimensions_map={"ParseMethod": "template"},
statistic="Sum",
period=period,
)
if rejected_included:
fb_rej_metric = cloudwatch.Metric(
namespace=namespace,
metric_name="ParseOutcome",
dimensions_map={"ParseMethod": "ai_fallback_rejected"},
statistic="Sum",
period=period,
)
using_metrics = {"fb": fb_metric, "rej": fb_rej_metric, "tmpl": tmpl_metric}
sum_terms = "FILL(fb,0)+FILL(rej,0)+FILL(tmpl,0)"
numerator = "(FILL(fb,0)+FILL(rej,0))"
else:
using_metrics = {"fb": fb_metric, "tmpl": tmpl_metric}
sum_terms = "FILL(fb,0)+FILL(tmpl,0)"
numerator = "FILL(fb,0)"
expression = f"IF(({sum_terms})>={floor}, 100*{numerator}/({sum_terms}), 0)"
fallback_rate = cloudwatch.MathExpression(
expression=expression,
using_metrics=using_metrics,
period=period,
label="TemplateFallbackRatePct",
)
fallback_rate.create_alarm(
scope,
id,
alarm_name=alarm_name,
alarm_description=alarm_description,
threshold=threshold,
evaluation_periods=evaluation_periods,
datapoints_to_alarm=datapoints_to_alarm,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))

449
cdk/po_stack.py
View file

@ -8,6 +8,7 @@ from aws_cdk import (
aws_cloudwatch as cloudwatch,
aws_cloudwatch_actions as cw_actions,
aws_dynamodb as dynamodb,
aws_iam as iam,
aws_kms as kms,
aws_lambda as lambda_,
aws_lambda_event_sources as lambda_event_sources,
@ -18,11 +19,136 @@ from aws_cdk import (
aws_ses_actions as ses_actions,
aws_secretsmanager as secretsmanager,
aws_sns as sns,
aws_sqs as sqs,
aws_ssm as ssm,
)
from constructs import Construct
import common
# Operations these tables actually issue (PutItem/UpdateItem/DeleteItem writes,
# GetItem/Query/BatchGetItem reads). DynamoDB emits ThrottledRequests/SystemErrors
# keyed by TableName + Operation only, so the CDK *_for_operations helpers (which
# render a SUM MathExpression across these per-operation metrics) are the correct,
# non-deprecated way to roll a table up to a single alarmable series.
_DDB_ALARM_OPERATIONS = [
dynamodb.Operation.GET_ITEM,
dynamodb.Operation.BATCH_GET_ITEM,
dynamodb.Operation.QUERY,
dynamodb.Operation.SCAN,
dynamodb.Operation.PUT_ITEM,
dynamodb.Operation.UPDATE_ITEM,
dynamodb.Operation.DELETE_ITEM,
dynamodb.Operation.BATCH_WRITE_ITEM,
]
def _add_ddb_alarms(scope, id_prefix, table, alarm_name_prefix, alarm_topic):
"""Add throttle + system-error alarms for a DynamoDB table.
Both fire on any non-zero datapoint in a 5-min window. ALARM-only SnsAction
to site-alerts (no OK action); TreatMissingData NOT_BREACHING.
"""
table.metric_throttled_requests_for_operations(
operations=_DDB_ALARM_OPERATIONS,
period=Duration.minutes(5),
statistic="Sum",
).create_alarm(
scope,
f"{id_prefix}ThrottlesAlarm",
alarm_name=f"{alarm_name_prefix}-throttles",
alarm_description=f"{alarm_name_prefix} DynamoDB throttled requests",
threshold=0,
evaluation_periods=1,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
table.metric_system_errors_for_operations(
operations=_DDB_ALARM_OPERATIONS,
period=Duration.minutes(5),
statistic="Sum",
).create_alarm(
scope,
f"{id_prefix}SystemErrorsAlarm",
alarm_name=f"{alarm_name_prefix}-system-errors",
alarm_description=f"{alarm_name_prefix} DynamoDB server-side (5xx) errors",
threshold=0,
evaluation_periods=1,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
# CloudWatch namespace for the log-derived sender-authentication metrics.
_SENDER_AUTH_METRIC_NAMESPACE = "Seahaven/ProcurementIngest"
def _add_sender_auth_rejected_alarm(scope, id_prefix, function_name, alarm_topic):
"""Metric-filter + alarm on ``sender_auth_rejected`` warnings (INFRA-107).
A rejected inbound email is skipped without erroring the invocation, so it
is invisible to the Errors/Throttles/DLQ alarms. This turns the structured
warning log into a CloudWatch metric and pages when rejections spike --
catching a silent false-reject storm (allowlist wrong, signing-domain
drift, SES header-format change) that would otherwise discard legitimate
mail while the pipeline reports healthy.
ALARM-only SnsAction to site-alerts; no OK action. The metric filter reads
the function's own log group (imported by the deterministic
``/aws/lambda/<fn>`` name, created by the function's log_retention). A plain
substring pattern is used because Lambda prefixes each line with its own
level/timestamp/request-id, so the JSON payload is not a standalone JSON
log event a `{$.event=...}` pattern could match.
"""
metric_name = f"{function_name}-sender-auth-rejected"
logs.MetricFilter(
scope,
f"{id_prefix}SenderAuthRejectedFilter",
log_group=logs.LogGroup.from_log_group_name(
scope,
f"{id_prefix}LogGroup",
f"/aws/lambda/{function_name}",
),
filter_pattern=logs.FilterPattern.literal('"sender_auth_rejected"'),
metric_namespace=_SENDER_AUTH_METRIC_NAMESPACE,
metric_name=metric_name,
metric_value="1",
default_value=0,
)
# Fire on a *sustained* reject condition rather than a volume spike. The
# earlier Sum>=3-over-15-min threshold had a blind spot that is exactly the
# failure this alarm exists to catch: a low-traffic pipeline in total
# drift outage (allowlist wrong / signing-domain changed) may only produce
# a trickle of rejects -- one every few minutes -- that never sums to 3 in
# any window, so the outage never pages. Instead: >=1 reject per 5-min
# period, alarming when 2 of the last 6 periods breach (evaluation_periods=6
# / datapoints_to_alarm=2). Six periods (30 min) with only 2 required
# datapoints closes the sparse-outage residual: even rejections >10-15 min
# apart can still place two breaching datapoints in a single 30-min
# evaluation window. A single stray spoof probe (one lone period) is
# tolerated and self-clears, but a sustained reject condition trips even
# at very low arrival rates. default_value=0 on the metric filter keeps the
# series continuous so NOT_BREACHING only applies before the first datapoint
# ever arrives.
cloudwatch.Metric(
namespace=_SENDER_AUTH_METRIC_NAMESPACE,
metric_name=metric_name,
period=Duration.minutes(5),
statistic="Sum",
).create_alarm(
scope,
f"{id_prefix}SenderAuthRejectedAlarm",
alarm_name=f"{function_name}-sender-auth-rejected",
alarm_description=(
f"{function_name} rejected inbound mail on sender authentication "
"(possible allowlist/DKIM-domain drift silently dropping real mail)"
),
threshold=1,
evaluation_periods=6,
datapoints_to_alarm=2,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_OR_EQUAL_TO_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
class PoIngestStack(Stack):
@ -41,7 +167,16 @@ class PoIngestStack(Stack):
)
# --- S3 bucket for raw emails ---
email_bucket = common.make_email_bucket(self, "EmailBucket", "po-ingest-emails")
email_bucket = s3.Bucket(
self,
"EmailBucket",
bucket_name=f"po-ingest-emails-{self.account}",
block_public_access=s3.BlockPublicAccess.BLOCK_ALL,
removal_policy=RemovalPolicy.RETAIN,
lifecycle_rules=[
s3.LifecycleRule(expiration=Duration.days(90)),
],
)
# --- Shared customer-managed CMK for sensitive DynamoDB tables ---
# Owned by the account-baseline app (alias/seahaven-dynamodb, INFRA-95 /
@ -88,7 +223,12 @@ class PoIngestStack(Stack):
# parse (bad email, transient error) is silently dropped after Lambda's
# retries. CDK generates the queue name to avoid colliding with the
# interim CLI-created po-email-processor-dlq (removed post-deploy).
email_processor_dlq = common.make_processor_dlq(self, "EmailProcessorDlq")
email_processor_dlq = sqs.Queue(
self,
"EmailProcessorDlq",
retention_period=Duration.days(14),
enforce_ssl=True,
)
# --- Lambda function ---
email_processor = lambda_.Function(
@ -161,30 +301,38 @@ class PoIngestStack(Stack):
# (us-east-1/us-east-2/us-west-2). A profile-only grant AccessDenies at
# runtime whenever the profile routes to a region whose foundation-model
# ARN is not allowed.
email_processor.add_to_role_policy(common.make_bedrock_invoke_statement(self))
# --- Standard per-Lambda alarms: po-email-processor ---
# errors (INFRA-41 / audit H-8), throttles, DLQ-messages (dropped PO
# emails), and a p99 duration alarm (orphan adoption of the CLI
# Lambda-Duration-po-email-processor under <fn>-duration naming, 45000 ms
# = 75% of the 60s timeout, eval 3 / dp 2). All ALARM-only to site-alerts.
common.add_standard_lambda_alarms(
self,
"EmailProcessor",
email_processor,
"po-email-processor",
alarm_topic,
duration_statistic="p99",
errors=True,
dlq=email_processor_dlq,
descriptions={
"errors": "po-email-processor async invocation errors",
"throttles": "po-email-processor invocation throttles",
"dlq": "po-email-processor DLQ has messages (dropped PO emails)",
"duration": "po-email-processor p99 duration approaching the 60s timeout",
},
email_processor.add_to_role_policy(
iam.PolicyStatement(
actions=[
"bedrock:InvokeModel",
"bedrock:InvokeModelWithResponseStream",
],
resources=[
"arn:aws:bedrock:us-east-1:328440206208:inference-profile/us.anthropic.claude-haiku-4-5-20251001-v1:0",
"arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-haiku-4-5-20251001-v1:0",
"arn:aws:bedrock:us-east-2::foundation-model/anthropic.claude-haiku-4-5-20251001-v1:0",
"arn:aws:bedrock:us-west-2::foundation-model/anthropic.claude-haiku-4-5-20251001-v1:0",
],
)
)
# --- Errors alarm (INFRA-41 / audit H-8) ---
# ALARM-only (no OK action, per the CloudWatch-alarm preference) to the
# shared site-alerts topic. Any errored invocation in a 5-min window pages.
email_processor.metric_errors(
period=Duration.minutes(5),
statistic="Sum",
).create_alarm(
self,
"EmailProcessorErrorsAlarm",
alarm_name="po-email-processor-errors",
alarm_description="po-email-processor async invocation errors",
threshold=0,
evaluation_periods=1,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
# --- Sender-auth rejection alarm (INFRA-107) ---
# A rejected email (bad/unaligned DKIM verdict) returns normally, so it
# produces NO Lambda error, NO DLQ message and NO retry -- only a
@ -195,10 +343,70 @@ class PoIngestStack(Stack):
# A CloudWatch Logs metric filter turns those warnings into a metric so a
# false-reject storm pages instead of vanishing. default_value=0 keeps the
# series populated (alarm stays OK, never INSUFFICIENT_DATA) between events.
common.add_sender_auth_rejected_alarm(
_add_sender_auth_rejected_alarm(
self, "EmailProcessor", "po-email-processor", alarm_topic
)
# --- Throttles alarm: po-email-processor ---
# Any throttled invocation (concurrency cap hit) in a 5-min window pages.
# ALARM-only to site-alerts; no OK action; NOT_BREACHING when no data.
email_processor.metric_throttles(
period=Duration.minutes(5),
statistic="Sum",
).create_alarm(
self,
"EmailProcessorThrottlesAlarm",
alarm_name="po-email-processor-throttles",
alarm_description="po-email-processor invocation throttles",
threshold=0,
evaluation_periods=1,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
# --- DLQ messages-present alarm ---
# Pages when any message lands in the EmailProcessorDlq: a message here
# means a PO email was permanently dropped after Lambda exhausted its
# async retries. Maximum over a single 5-min window > 0 fires; missing
# data (no messages metric emitted) is not breaching. Reuses the shared
# site-alerts topic, ALARM-only, like the errors alarm above. The metric
# helper derives the QueueName dimension from the queue construct, so the
# alarm tracks the CDK-generated queue name without hardcoding it.
email_processor_dlq.metric_approximate_number_of_messages_visible(
period=Duration.minutes(5),
statistic="Maximum",
).create_alarm(
self,
"EmailProcessorDlqMessagesAlarm",
alarm_name="po-email-processor-dlq-messages",
alarm_description="po-email-processor DLQ has messages (dropped PO emails)",
threshold=0,
evaluation_periods=1,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
# --- Duration alarm: po-email-processor (orphan adoption) ---
# Adopts the orphaned CLI alarm Lambda-Duration-po-email-processor under
# the repo's <fn>-duration naming (NEW logical name → no deploy collision;
# delete the orphan post-deploy). p99 / 45000 ms
# (75% of the 60s timeout) / eval 3 of 3 — tighter than the orphan's
# Maximum>=48000 / 1-of-1.
email_processor.metric_duration(
period=Duration.minutes(5),
statistic="p99",
).create_alarm(
self,
"EmailProcessorDurationAlarm",
alarm_name="po-email-processor-duration",
alarm_description="po-email-processor p99 duration approaching the 60s timeout",
threshold=45000,
evaluation_periods=3,
datapoints_to_alarm=2,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_OR_EQUAL_TO_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
# --- Template fallback-rate alarm: po-email-processor ---
# The processor tries a deterministic template parse first and only calls
# the Bedrock AI extractor on a miss/invalid. A sustained rise in the
@ -244,23 +452,43 @@ class PoIngestStack(Stack):
# falsely page this template-drift alarm on top of the dedicated
# rejected alarm below. The rejected series gets its own alarm
# instead (EmailProcessorAiFallbackRejectedAlarm, below).
common.make_fallback_rate_alarm(
fb_metric = cloudwatch.Metric(
namespace="Seahaven/PoIngest",
metric_name="ParseOutcome",
dimensions_map={"ParseMethod": "ai_fallback"},
statistic="Sum",
period=Duration.hours(6),
)
tmpl_metric = cloudwatch.Metric(
namespace="Seahaven/PoIngest",
metric_name="ParseOutcome",
dimensions_map={"ParseMethod": "template"},
statistic="Sum",
period=Duration.hours(6),
)
fallback_rate = cloudwatch.MathExpression(
expression=(
"IF((FILL(fb,0)+FILL(tmpl,0))>=8, "
"100*FILL(fb,0)/(FILL(fb,0)+FILL(tmpl,0)), 0)"
),
using_metrics={"fb": fb_metric, "tmpl": tmpl_metric},
period=Duration.hours(6),
label="TemplateFallbackRatePct",
)
fallback_rate.create_alarm(
self,
"EmailProcessorTemplateFallbackRateAlarm",
namespace="Seahaven/PoIngest",
alarm_topic=alarm_topic,
alarm_name="po-email-processor-template-fallback-rate",
alarm_description=(
"po-email-processor deterministic-template coverage collapse: "
">20% of parses fell back to the Bedrock AI extractor"
),
rejected_included=False,
period=Duration.hours(6),
threshold=20,
floor=8,
evaluation_periods=4,
datapoints_to_alarm=2,
)
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
# --- AI-fallback rejected alarm: po-email-processor (Phase 1) ---
# The validate_ai_fallback gate (template_parser.py) fail-closes Bedrock
@ -424,24 +652,38 @@ class PoIngestStack(Stack):
po_table.grant_read_data(web_ui)
web_ui_auth_secret.grant_read(web_ui)
# --- Standard per-Lambda alarms: po-web-ui ---
# Throttles + p99 duration only (no errors alarm, no DLQ -- web_ui is a
# synchronous read path with no async DLQ). p99 / 45000 ms (75% of the
# 60s timeout) / eval 3, datapoints 2.
common.add_standard_lambda_alarms(
# --- Throttles alarm: po-web-ui ---
web_ui.metric_throttles(
period=Duration.minutes(5),
statistic="Sum",
).create_alarm(
self,
"WebUi",
web_ui,
"po-web-ui",
alarm_topic,
duration_statistic="p99",
errors=False,
dlq=None,
descriptions={
"throttles": "po-web-ui invocation throttles",
"duration": "po-web-ui p99 duration approaching the 60s timeout",
},
)
"WebUiThrottlesAlarm",
alarm_name="po-web-ui-throttles",
alarm_description="po-web-ui invocation throttles",
threshold=0,
evaluation_periods=1,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
# --- Duration alarm: po-web-ui ---
# Net-new (no orphan exists for this function).
# p99 / 45000 ms (75% of the 60s timeout) / eval 3, datapoints 2.
web_ui.metric_duration(
period=Duration.minutes(5),
statistic="p99",
).create_alarm(
self,
"WebUiDurationAlarm",
alarm_name="po-web-ui-duration",
alarm_description="po-web-ui p99 duration approaching the 60s timeout",
threshold=45000,
evaluation_periods=3,
datapoints_to_alarm=2,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_OR_EQUAL_TO_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
# Public Function URL removed 2026-06-08 (INFRA-74 / audit C-5): the
# unauthenticated FunctionUrlAuthType.NONE URL was deleted out-of-band
@ -503,27 +745,56 @@ class PoIngestStack(Stack):
)
)
# --- Standard per-Lambda alarms: po-ingest-site-extractor ---
# errors + throttles + p99 duration. NO DLQ alarm: site_extractor is a
# DynamoEventSource stream consumer with no async DLQ attached (dlq=None).
# --- Errors alarm: po-ingest-site-extractor ---
# Stream-consumer errors retry per the event-source config, but a
# persistent failure stalls the verified-sites pipeline. p99 / 45000 ms
# (75% of the 60s timeout) / eval 3, datapoints 2.
common.add_standard_lambda_alarms(
# persistent failure stalls the verified-sites pipeline. ALARM-only to
# site-alerts; no OK action; NOT_BREACHING when no data.
site_extractor.metric_errors(
period=Duration.minutes(5),
statistic="Sum",
).create_alarm(
self,
"SiteExtractor",
site_extractor,
"po-ingest-site-extractor",
alarm_topic,
duration_statistic="p99",
errors=True,
dlq=None,
descriptions={
"errors": "po-ingest-site-extractor invocation errors",
"throttles": "po-ingest-site-extractor invocation throttles",
"duration": "po-ingest-site-extractor p99 duration approaching the 60s timeout",
},
)
"SiteExtractorErrorsAlarm",
alarm_name="po-ingest-site-extractor-errors",
alarm_description="po-ingest-site-extractor invocation errors",
threshold=0,
evaluation_periods=1,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
# --- Throttles alarm: po-ingest-site-extractor ---
site_extractor.metric_throttles(
period=Duration.minutes(5),
statistic="Sum",
).create_alarm(
self,
"SiteExtractorThrottlesAlarm",
alarm_name="po-ingest-site-extractor-throttles",
alarm_description="po-ingest-site-extractor invocation throttles",
threshold=0,
evaluation_periods=1,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
# --- Duration alarm: po-ingest-site-extractor ---
# Net-new (no orphan exists for this function).
# p99 / 45000 ms (75% of the 60s timeout) / eval 3, datapoints 2.
site_extractor.metric_duration(
period=Duration.minutes(5),
statistic="p99",
).create_alarm(
self,
"SiteExtractorDurationAlarm",
alarm_name="po-ingest-site-extractor-duration",
alarm_description="po-ingest-site-extractor p99 duration approaching the 60s timeout",
threshold=45000,
evaluation_periods=3,
datapoints_to_alarm=2,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_OR_EQUAL_TO_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
cdk.CfnOutput(
self,
@ -551,56 +822,24 @@ class PoIngestStack(Stack):
# --- DynamoDB throttle + system-error alarms ---
# ThrottledRequests / SystemErrors emit at TableName + Operation only
# (verified against live CloudWatch: no TableName-only rollup exists, and
# metric_throttled_requests is deprecated/invalid in aws-cdk-lib 2.261.0).
# metric_throttled_requests is deprecated/invalid in aws-cdk-lib 2.259.0).
# Each table currently has zero throttle/error datapoints, so the series
# only materialise on first occurrence — NOT_BREACHING keeps them OK until
# then.
common.add_ddb_alarms(
_add_ddb_alarms(
self, "PurchaseOrdersTable", po_table, "purchase-orders", alarm_topic
)
common.add_ddb_alarms(
_add_ddb_alarms(
self,
"VerifiedSitesTable",
verified_sites_table,
"verified-sites",
alarm_topic,
)
common.add_ddb_alarms(
_add_ddb_alarms(
self,
"PendingSiteReviewTable",
pending_review_table,
"pending-site-review",
alarm_topic,
)
# --- Function ARN + consumed-table-name outputs (Phase 4, additive) ---
cdk.CfnOutput(
self,
"EmailProcessorFunctionArn",
value=email_processor.function_arn,
description="ARN of the po-email-processor Lambda",
)
cdk.CfnOutput(
self,
"WebUiFunctionArn",
value=web_ui.function_arn,
description="ARN of the po-web-ui Lambda",
)
cdk.CfnOutput(
self,
"SiteExtractorFunctionArn",
value=site_extractor.function_arn,
description="ARN of the po-ingest-site-extractor Lambda",
)
cdk.CfnOutput(
self,
"PurchaseOrdersTableName",
value=po_table.table_name,
description="purchase-orders DynamoDB table consumed by this stack",
)
cdk.CfnOutput(
self,
"PendingSiteReviewTableName",
value=pending_review_table.table_name,
description="pending-site-review DynamoDB table",
)

2
cdk/requirements.txt
View file

@ -1,2 +1,2 @@
aws-cdk-lib==2.261.0
constructs==10.6.0
constructs>=10.6.0

352
cdk/wo_stack.py
View file

@ -8,6 +8,7 @@ from aws_cdk import (
aws_cloudwatch as cloudwatch,
aws_cloudwatch_actions as cw_actions,
aws_dynamodb as dynamodb,
aws_iam as iam,
aws_lambda as lambda_,
aws_logs as logs,
aws_s3 as s3,
@ -16,10 +17,138 @@ from aws_cdk import (
aws_ses_actions as ses_actions,
aws_secretsmanager as secretsmanager,
aws_sns as sns,
aws_sqs as sqs,
)
from constructs import Construct
import common
# Operations these tables actually issue (PutItem/UpdateItem/DeleteItem writes,
# GetItem/Query/BatchGetItem reads). DynamoDB emits ThrottledRequests/SystemErrors
# keyed by TableName + Operation only, so the CDK *_for_operations helpers (which
# render a SUM MathExpression across these per-operation metrics) are the correct,
# non-deprecated way to roll a table up to a single alarmable series.
_DDB_ALARM_OPERATIONS = [
dynamodb.Operation.GET_ITEM,
dynamodb.Operation.BATCH_GET_ITEM,
dynamodb.Operation.QUERY,
dynamodb.Operation.SCAN,
dynamodb.Operation.PUT_ITEM,
dynamodb.Operation.UPDATE_ITEM,
dynamodb.Operation.DELETE_ITEM,
dynamodb.Operation.BATCH_WRITE_ITEM,
]
def _add_ddb_alarms(scope, id_prefix, table, alarm_name_prefix, alarm_topic):
"""Add throttle + system-error alarms for a DynamoDB table.
Both fire on any non-zero datapoint in a 5-min window. ALARM-only SnsAction
to site-alerts (no OK action); TreatMissingData NOT_BREACHING.
"""
table.metric_throttled_requests_for_operations(
operations=_DDB_ALARM_OPERATIONS,
period=Duration.minutes(5),
statistic="Sum",
).create_alarm(
scope,
f"{id_prefix}ThrottlesAlarm",
alarm_name=f"{alarm_name_prefix}-throttles",
alarm_description=f"{alarm_name_prefix} DynamoDB throttled requests",
threshold=0,
evaluation_periods=1,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
table.metric_system_errors_for_operations(
operations=_DDB_ALARM_OPERATIONS,
period=Duration.minutes(5),
statistic="Sum",
).create_alarm(
scope,
f"{id_prefix}SystemErrorsAlarm",
alarm_name=f"{alarm_name_prefix}-system-errors",
alarm_description=f"{alarm_name_prefix} DynamoDB server-side (5xx) errors",
threshold=0,
evaluation_periods=1,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
# CloudWatch namespace for the log-derived sender-authentication metrics.
_SENDER_AUTH_METRIC_NAMESPACE = "Seahaven/ProcurementIngest"
def _add_sender_auth_rejected_alarm(scope, id_prefix, function_name, alarm_topic):
"""Metric-filter + alarm on ``sender_auth_rejected`` warnings (INFRA-107).
A rejected inbound email is skipped without erroring the invocation, so it
is invisible to the Errors/Throttles/DLQ alarms. This turns the structured
warning log into a CloudWatch metric and pages when rejections spike --
catching a silent false-reject storm (allowlist wrong, signing-domain
drift, SES header-format change) that would otherwise discard legitimate
mail while the pipeline reports healthy. This is the safety net for the WO
allowlist domain assumption (seahaven.com) -- if the real Gmail-forward
re-signing domain differs, this alarm surfaces it instead of a silent
work-order outage.
ALARM-only SnsAction to site-alerts; no OK action. The metric filter reads
the function's own log group (imported by the deterministic
``/aws/lambda/<fn>`` name, created by the function's log_retention). A plain
substring pattern is used because Lambda prefixes each line with its own
level/timestamp/request-id, so the JSON payload is not a standalone JSON
log event a `{$.event=...}` pattern could match.
"""
metric_name = f"{function_name}-sender-auth-rejected"
logs.MetricFilter(
scope,
f"{id_prefix}SenderAuthRejectedFilter",
log_group=logs.LogGroup.from_log_group_name(
scope,
f"{id_prefix}LogGroup",
f"/aws/lambda/{function_name}",
),
filter_pattern=logs.FilterPattern.literal('"sender_auth_rejected"'),
metric_namespace=_SENDER_AUTH_METRIC_NAMESPACE,
metric_name=metric_name,
metric_value="1",
default_value=0,
)
# Fire on a *sustained* reject condition rather than a volume spike. The
# earlier Sum>=3-over-15-min threshold had a blind spot that is exactly the
# failure this alarm exists to catch: a low-traffic pipeline in total
# drift outage (allowlist wrong / signing-domain changed) may only produce
# a trickle of rejects -- one every few minutes -- that never sums to 3 in
# any window, so the outage never pages. Instead: >=1 reject per 5-min
# period, alarming when 2 of the last 6 periods breach (evaluation_periods=6
# / datapoints_to_alarm=2). Six periods (30 min) with only 2 required
# datapoints closes the sparse-outage residual: even rejections >10-15 min
# apart can still place two breaching datapoints in a single 30-min
# evaluation window. A single stray spoof probe (one lone period) is
# tolerated and self-clears, but a sustained reject condition trips even
# at very low arrival rates. default_value=0 on the metric filter keeps the
# series continuous so NOT_BREACHING only applies before the first datapoint
# ever arrives.
cloudwatch.Metric(
namespace=_SENDER_AUTH_METRIC_NAMESPACE,
metric_name=metric_name,
period=Duration.minutes(5),
statistic="Sum",
).create_alarm(
scope,
f"{id_prefix}SenderAuthRejectedAlarm",
alarm_name=f"{function_name}-sender-auth-rejected",
alarm_description=(
f"{function_name} rejected inbound mail on sender authentication "
"(possible allowlist/DKIM-domain drift silently dropping real mail)"
),
threshold=1,
evaluation_periods=6,
datapoints_to_alarm=2,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_OR_EQUAL_TO_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
class WorkorderIngestStack(Stack):
@ -38,8 +167,15 @@ class WorkorderIngestStack(Stack):
)
# --- S3 bucket for raw emails ---
email_bucket = common.make_email_bucket(
self, "EmailBucket", "workorder-ingest-emails"
email_bucket = s3.Bucket(
self,
"EmailBucket",
bucket_name=f"workorder-ingest-emails-{self.account}",
block_public_access=s3.BlockPublicAccess.BLOCK_ALL,
removal_policy=RemovalPolicy.RETAIN,
lifecycle_rules=[
s3.LifecycleRule(expiration=Duration.days(90)),
],
)
# --- DynamoDB tables ---
@ -86,7 +222,12 @@ class WorkorderIngestStack(Stack):
# parse (bad email, transient error) is silently dropped after Lambda's
# retries. CDK generates the queue name to avoid colliding with the
# interim CLI-created workorder-email-processor-dlq (removed post-deploy).
email_processor_dlq = common.make_processor_dlq(self, "EmailProcessorDlq")
email_processor_dlq = sqs.Queue(
self,
"EmailProcessorDlq",
retention_period=Duration.days(14),
enforce_ssl=True,
)
# --- Lambda function ---
email_processor = lambda_.Function(
@ -149,7 +290,20 @@ class WorkorderIngestStack(Stack):
# (us-east-1/us-east-2/us-west-2). A profile-only grant AccessDenies at
# runtime whenever the profile routes to a region whose foundation-model
# ARN is not allowed.
email_processor.add_to_role_policy(common.make_bedrock_invoke_statement(self))
email_processor.add_to_role_policy(
iam.PolicyStatement(
actions=[
"bedrock:InvokeModel",
"bedrock:InvokeModelWithResponseStream",
],
resources=[
"arn:aws:bedrock:us-east-1:328440206208:inference-profile/us.anthropic.claude-haiku-4-5-20251001-v1:0",
"arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-haiku-4-5-20251001-v1:0",
"arn:aws:bedrock:us-east-2::foundation-model/anthropic.claude-haiku-4-5-20251001-v1:0",
"arn:aws:bedrock:us-west-2::foundation-model/anthropic.claude-haiku-4-5-20251001-v1:0",
],
)
)
# NOTE: The pre-emptive grant_encrypt_decrypt on the shared DynamoDB CMK
# (alias/seahaven-dynamodb) was removed (security sweep 2026-06-17). The
@ -161,28 +315,22 @@ class WorkorderIngestStack(Stack):
# point grant_read_write_data on the (then encrypted) tables would propagate
# the needed key permissions automatically.
# --- Standard per-Lambda alarms: workorder-email-processor ---
# errors (INFRA-41 / audit H-8), throttles, DLQ-visible-messages (dropped
# emails), and a p95 duration alarm (orphan adoption of the CLI
# Lambda-Duration-workorder-email-processor under <fn>-duration naming,
# 45000 ms = 75% of the 60s timeout, eval 3 / dp 2). p95 (NOT p99) is the
# WO-specific duration statistic. All ALARM-only to site-alerts.
common.add_standard_lambda_alarms(
# --- Errors alarm (INFRA-41 / audit H-8) ---
# ALARM-only (no OK action, per the CloudWatch-alarm preference) to the
# shared site-alerts topic. Any errored invocation in a 5-min window pages.
email_processor.metric_errors(
period=Duration.minutes(5),
statistic="Sum",
).create_alarm(
self,
"EmailProcessor",
email_processor,
"workorder-email-processor",
alarm_topic,
duration_statistic="p95",
errors=True,
dlq=email_processor_dlq,
descriptions={
"errors": "workorder-email-processor async invocation errors",
"throttles": "workorder-email-processor invocation throttles",
"dlq": "workorder-email-processor DLQ has visible messages (dropped emails)",
"duration": "workorder-email-processor p95 duration approaching the 60s timeout",
},
)
"EmailProcessorErrorsAlarm",
alarm_name="workorder-email-processor-errors",
alarm_description="workorder-email-processor async invocation errors",
threshold=0,
evaluation_periods=1,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
# --- Sender-auth rejection alarm (INFRA-107) ---
# A rejected email (bad/unaligned DKIM verdict) returns normally, so it
@ -193,10 +341,68 @@ class WorkorderIngestStack(Stack):
# different domain), 100% of legitimate work-order mail is silently
# dropped. This metric filter + alarm turns those warnings into a paging
# signal so a false-reject storm surfaces instead of a silent outage.
common.add_sender_auth_rejected_alarm(
_add_sender_auth_rejected_alarm(
self, "EmailProcessor", "workorder-email-processor", alarm_topic
)
# --- Throttles alarm: workorder-email-processor ---
# Any throttled invocation (concurrency cap hit) in a 5-min window pages.
# ALARM-only to site-alerts; no OK action; NOT_BREACHING when no data.
email_processor.metric_throttles(
period=Duration.minutes(5),
statistic="Sum",
).create_alarm(
self,
"EmailProcessorThrottlesAlarm",
alarm_name="workorder-email-processor-throttles",
alarm_description="workorder-email-processor invocation throttles",
threshold=0,
evaluation_periods=1,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
# --- Duration alarm: workorder-email-processor (orphan adoption) ---
# Adopts the orphaned CLI alarm Lambda-Duration-workorder-email-processor
# under the repo's <fn>-duration naming (NEW logical name → no deploy
# collision; delete the orphan post-deploy). p95 /
# 45000 ms (75% of the 60s timeout) / eval 3 of which 2 datapoints —
# tighter than the orphan's Maximum>=48000 / 1-of-1.
email_processor.metric_duration(
period=Duration.minutes(5),
statistic="p95",
).create_alarm(
self,
"EmailProcessorDurationAlarm",
alarm_name="workorder-email-processor-duration",
alarm_description="workorder-email-processor p95 duration approaching the 60s timeout",
threshold=45000,
evaluation_periods=3,
datapoints_to_alarm=2,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_OR_EQUAL_TO_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
# --- DLQ messages-present alarm (INFRA-41 / audit H-8) ---
# The errors alarm above fires on any errored invocation, but a message
# only lands in the DLQ after Lambda exhausts its async retries and gives
# up — i.e. a genuinely dropped email. ALARM-only (no OK action) to the
# same shared site-alerts topic. MAXIMUM over a 5-min window so a single
# visible message pages even if it is later consumed/redriven.
email_processor_dlq.metric_approximate_number_of_messages_visible(
period=Duration.minutes(5),
statistic="Maximum",
).create_alarm(
self,
"EmailProcessorDlqMessagesAlarm",
alarm_name="workorder-email-processor-dlq-messages",
alarm_description="workorder-email-processor DLQ has visible messages (dropped emails)",
threshold=0,
evaluation_periods=1,
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
# --- Template fallback-rate alarm: workorder-email-processor ---
# The processor tries a deterministic template parse first and only calls
# the Bedrock AI extractor on a miss/invalid. A sustained rise in the
@ -207,28 +413,64 @@ class WorkorderIngestStack(Stack):
# ~760/day volume; FILL(0) + a >=10-sample volume floor prevent
# low-volume false pages and INSUFFICIENT_DATA. ALARM-only SnsAction to
# site-alerts, no OK action, NOT_BREACHING -- matching the stack idiom.
# rejected_included=True: the WO expression folds ai_fallback_rejected
# (rej) into BOTH numerator and denominator -- a drift outage whose AI
# output also fails the gate must still count as fallback, otherwise it
# would LOWER the observed rate while silently dropping mail. (Contrast
# PO, which excludes rej to avoid a pre-call double-count.)
common.make_fallback_rate_alarm(
fb_metric = cloudwatch.Metric(
namespace="Seahaven/WorkorderIngest",
metric_name="ParseOutcome",
dimensions_map={"ParseMethod": "ai_fallback"},
statistic="Sum",
period=Duration.minutes(15),
)
# AI-fallback parses REJECTED by the validate_ai_fallback gate emit
# ParseMethod=ai_fallback_rejected (and nothing else), so they must
# count as fallback here too -- otherwise a drift outage whose AI
# output also fails the gate would LOWER the observed fallback rate
# while silently dropping mail.
fb_rej_metric = cloudwatch.Metric(
namespace="Seahaven/WorkorderIngest",
metric_name="ParseOutcome",
dimensions_map={"ParseMethod": "ai_fallback_rejected"},
statistic="Sum",
period=Duration.minutes(15),
)
tmpl_metric = cloudwatch.Metric(
namespace="Seahaven/WorkorderIngest",
metric_name="ParseOutcome",
dimensions_map={"ParseMethod": "template"},
statistic="Sum",
period=Duration.minutes(15),
)
fallback_rate = cloudwatch.MathExpression(
expression=(
# The IF volume floor (>=10) already guarantees the denominator
# is non-zero in the true branch, so divide directly. (An earlier
# MAX([...,1]) divide-by-zero guard used array syntax CloudWatch
# rejects at deploy: "Unsupported operand type(s) for MAX".)
"IF((FILL(fb,0)+FILL(rej,0)+FILL(tmpl,0))>=10, "
"100*(FILL(fb,0)+FILL(rej,0))"
"/(FILL(fb,0)+FILL(rej,0)+FILL(tmpl,0)), 0)"
),
using_metrics={
"fb": fb_metric,
"rej": fb_rej_metric,
"tmpl": tmpl_metric,
},
period=Duration.minutes(15),
label="TemplateFallbackRatePct",
)
fallback_rate.create_alarm(
self,
"EmailProcessorTemplateFallbackRateAlarm",
namespace="Seahaven/WorkorderIngest",
alarm_topic=alarm_topic,
alarm_name="workorder-email-processor-template-fallback-rate",
alarm_description=(
"workorder-email-processor deterministic-template coverage "
"collapse: >15% of parses fell back to the Bedrock AI extractor"
),
rejected_included=True,
period=Duration.minutes(15),
threshold=15,
floor=10,
evaluation_periods=3,
datapoints_to_alarm=2,
)
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
# --- AI-fallback rejected alarm: workorder-email-processor ---
# A parse rejected by the validate_ai_fallback gate is dropped without
@ -360,43 +602,17 @@ class WorkorderIngestStack(Stack):
# --- DynamoDB throttle + system-error alarms ---
# ThrottledRequests / SystemErrors emit at TableName + Operation only
# (verified against live CloudWatch: no TableName-only rollup exists, and
# metric_throttled_requests is deprecated/invalid in aws-cdk-lib 2.261.0).
# metric_throttled_requests is deprecated/invalid in aws-cdk-lib 2.259.0).
# Each table currently has zero throttle/error datapoints, so the series
# only materialise on first occurrence — NOT_BREACHING keeps them OK until
# then.
common.add_ddb_alarms(
_add_ddb_alarms(
self, "WorkOrdersTable", work_orders_table, "WorkOrders", alarm_topic
)
common.add_ddb_alarms(
_add_ddb_alarms(
self, "WorkOrderComments", comments_table, "WorkOrderComments", alarm_topic
)
# --- Function ARN + consumed-table-name outputs (Phase 4, additive) ---
cdk.CfnOutput(
self,
"EmailProcessorFunctionArn",
value=email_processor.function_arn,
description="ARN of the workorder-email-processor Lambda",
)
cdk.CfnOutput(
self,
"WebUiFunctionArn",
value=web_ui.function_arn,
description="ARN of the workorder-web-ui Lambda",
)
cdk.CfnOutput(
self,
"WorkOrdersTableName",
value=work_orders_table.table_name,
description="WorkOrders DynamoDB table",
)
cdk.CfnOutput(
self,
"WorkOrderCommentsTableName",
value=comments_table.table_name,
description="WorkOrderComments DynamoDB table",
)
# Public Function URL removed 2026-06-08 (INFRA-74 / audit C-5): the
# unauthenticated FunctionUrlAuthType.NONE URL was deleted out-of-band
# via CLI. Removing the construct (and its auto-generated Principal:*

7
pytest.ini
View file

@ -15,10 +15,3 @@ addopts =
--cov=lambdas/shared
--cov-report=term-missing
--cov-fail-under=80
; The 80% floor is an aggregate whole-suite gate, enforced in CI by the
; reusable ci-python-sam workflow's bare `pytest` run (which inherits these
; addopts). Because the floor is inherited by every invocation, running a
; single root or file for local iteration (e.g. `pytest tests/test_web_ui_auth.py`)
; under-measures the fixed --cov set and red-exits even when the selected tests
; pass. That is expected: append `--cov-fail-under=0` (or `-p no:cov`) to a
; subset run to skip the floor while iterating. The full-suite run still gates.

3
ruff.toml
View file

@ -26,8 +26,5 @@ max-complexity = 12
"lambdas/wo/email_processor/template_parser.py" = ["C901", "PLR0911", "PLR0912"]
"lambdas/shared/ses_auth.py" = ["C901", "PLR0911"]
"lambdas/shared/emf.py" = ["PLR0913"] # 6-arg emitter IS the Phase 3 envelope contract
# Phase 4 plain helpers take (scope, id, ...) positionally by design to keep
# construct logical IDs byte-stable; the arg count IS the contract (like emf.py).
"cdk/common.py" = ["PLR0913"]
"lambdas/po/site_extractor/handler.py" = ["PLR0911"] # Phase 6 owns this file
"scripts/reprocess.py" = ["PLR0912"] # argparse validation ladder; Phase 7 file

17
tests/test_web_ui_auth.py
View file

@ -105,16 +105,11 @@ def test_header_matrix(load_auth, monkeypatch, headers, expected):
assert mod.is_authenticated(_evt(**headers)) is expected
@pytest.mark.xfail(
strict=True,
reason="web_ui_auth is frozen this phase: is_authenticated raises TypeError on "
"a non-ASCII presented token (hmac.compare_digest is ASCII-only) and surfaces "
"as a 500 instead of failing closed to False. Asserting the DESIRED behavior "
"under xfail(strict) documents the intended fix and auto-fails (xpass) the moment "
"the module is corrected, forcing the marker's removal. Tracked as a follow-up.",
raises=TypeError,
)
def test_non_ascii_token_should_fail_closed(load_auth, monkeypatch):
def test_non_ascii_token_raises_typeerror_documenting_pin(load_auth, monkeypatch):
"""PIN (current behavior, fix deferred -- module frozen this phase): a
non-ASCII presented token makes hmac.compare_digest raise TypeError (surfaces
as a 500 today). Locked so a future fix is a deliberate, reviewed change."""
mod = load_auth()
monkeypatch.setattr(mod, "_get_auth_token", lambda: "SECRET")
assert mod.is_authenticated(_evt(**{"x-auth-token": "SÉCRET"})) is False
with pytest.raises(TypeError):
mod.is_authenticated(_evt(**{"x-auth-token": "SÉCRET"}))