3 commits

Author	SHA1	Message	Date
Adam Moussa	ff368beaa4	test: consolidate test roots — one loader, shared support, enforced CI floor (phase 8) (#118) ... * test: consolidate test roots — one repo-root loader, shared support package, missing-scenario suites, enforced ruff/coverage floor (refactor phase 8) tests/conftest.py only loads for the tests/ root, not a standalone `pytest lambdas/po/email_processor/tests` run, so it could never carry session invariants like the dummy AWS env or the moto stubber registration. Add a single repo-root conftest.py (pytest.ini pins rootdir there, so it loads for every invocation) that sets the dummy AWS credentials/region, imports moto BEFORE any handler module so boto3 sessions pick up its stubber hook (carrying the explanatory comment verbatim from the old _po_parser_support.py), and exposes one load_lambda_module(pipeline, name) — the sys.modules save/restore dance stays, since template_parser is still a duplicated bare name across pipelines needing per-exec sibling binding. Add tests/support/ as the shared package both pipelines' local _*_parser_support.py modules delegate to: a superset FakeTable (PO's update_item recording + WO's put_item and keyed single-row store), FakeDynamoResource, load_email, and load_golden with parse_float=Decimal kept (load-bearing for exact money comparison at PO magnitudes — WO's prior load_golden had no parse_float and must not regress PO by losing it). Rewrite _wo_parser_support.py off the bare `import handler` / `from handler import parse_raw_email` strategy that was the source of the bare-name sys.modules collision the other two loaders defend against. Move test_po_merge.py and test_pad_zip.py into lambdas/po/email_processor/tests/ (PO-specific, belongs beside the code) via git mv so history follows; test_parse_raw_email.py and test_ses_auth.py stay at the repo root since they're genuinely cross-pipeline, parameterized over both handlers. Delete tests/test_local.py: it globs a nonexistent samples/ dir, is WO-only, and imports a handler at collection time, bypassing the loader gate entirely — the golden suites already cover its role. Its pytest.ini exclusion comment goes with it. New scenario coverage, all built on the single loader + support package: - PO+WO Bedrock transport errors (ThrottlingException, missing 'content' key, empty content list, non-JSON model text), asserting PO's pre-call ai_fallback metric survives with no partial write and the exception propagates; WO's no-datapoint-on-throttle behavior is pinned with a documenting test rather than "fixed" by reordering. - Handler-level SES-auth reject seam per pipeline: no auth monkeypatch + empty ALLOWED_DKIM_DOMAINS asserts zero Bedrock calls, zero writes, no raise — closing the hole where deleting the gate line today still passes every test. - web_ui coverage for both PO and WO (0% before this): fail-closed on unset ARN and on a Secrets Manager exception, TTL cache refresh, Bearer/X-Auth-Token/header-case-insensitivity, wrong-token 401 with no table scan, non-ASCII token, and a hostile-field-escaping regression lock. PO web_ui has no __init__.py, so these go through the loader rather than package imports. - A moto-backed mirror of test_po_merge for WO merge semantics (table 'WorkOrders'): null-status never clobbers wo_status, created_at immutable via if_not_exists, status->wo_status mapping, None fields absent from SET, record_type only-when-present. - Small pins: the PO-DC-02 64-char EMF clamp regression and per-pipeline multi-record failure-isolation (all-or-retry contract). The reprocess.py synthetic-event-shape contract test already landed in Phase 7, so it isn't duplicated here. Two WO product-code fixes ride along, since this is the phase that exercises them: (a) the invalid_status reason-code fix in template_parser.py's status check, which previously returned malformed_site_code for the same failure validate_ai_fallback already labels invalid_status, making one failure surface two codes depending on path (grepped the dashboards/metric filters for malformed_site_code first — no external references found, safe to diverge the two codes); (b) wrapping the WO Bedrock call in handler.py so a transport failure emits ai_fallback/bedrock_error in an except-and-reraise. This is deliberately not a naive reorder: the emit sits in the except block, not pre-call, so a gate-rejected email still emits only ai_fallback_rejected and wo_stack's "a rejected email emits nothing else" alarm contract doesn't double-count. A test computes the emitted series by hand to pin the no-double-count behavior. Neither change touches the handler event/return contract. _validate_new_po_values in the PO template_parser.py is split into per-rule helpers, and the V4 anchor-frame dataclass now carries summary_matches/price so V13 can consume them; extract_new_po (C901=35) is included in the split. Add ruff.toml enabling C901/PLR so the mccabe/complexity suppressions scattered through the tree stop being decorative; derived_fields.py is under the shadow-bake freeze so its violations are silenced via a per-file ignore with a justification comment instead of an in-file edit, and the handful of other pre-existing violations surfaced by turning the config on get the same per-file-ignore treatment with a reason, or a fix where the file isn't frozen. scripts/ is added to the CI lint scope. CI gains an explicit --cov module list (lambdas/po and wo email_processor + web_ui, po/site_extractor, lambdas/shared) plus --cov-fail-under=80, since web_ui and site_extractor lack __init__.py markers and a bare --cov=lambdas silently skips them for the missing package marker; .coveragerc omits the test dirs themselves from the count. The Phase 0 AST bundle-consistency test stays in the standard pytest run. .gitignore picks up the resulting .coverage data file. docs/po-template-parser.md gets a small correction: the EXTRACTION_PROMPT declares quantity/price as "number or null", not JSON strings, so parse_float=Decimal already handles a conforming Bedrock response — the doc previously implied the coercion path was the primary mechanism rather than a defensive net for non-conforming responses. * test: lock attribute-context quote escaping in web_ui hostile-field test The escaping regression lock asserted only the element-context vector (raw <script> absent, &lt;script&gt; present) while its docstring claimed quotes were covered -- the payload's " and ' were never asserted on, so a quote-escaping regression on the onclick row-link sink (attribute breakout -> event-handler injection) would have passed green. /sh-security-review finding WC-01 (confirmed medium, test-integrity). Add assertions that the onclick sink's JSON string renders its opening quote as &quot; (raw " after window.location= fails), that the payload's quote characters appear only entity-escaped, and that the raw payload never appears anywhere in the body. Mutation-verified: the test now fails when the sink's quote-escaping is dropped. * test: address Open SWE review — xfail the web_ui non-ASCII auth pin, document subset coverage-floor override - tests/test_web_ui_auth.py: replace the TypeError characterization pin with an xfail(strict, raises=TypeError) asserting the DESIRED fail-closed (False) behavior. Documents the intended fix and auto-fails (xpass) once web_ui_auth is corrected, instead of requiring a passing test to be knowingly deleted. The module stays frozen this phase; the underlying hmac.compare_digest ASCII-only defect is tracked as a follow-up. - pytest.ini: document that the aggregate 80% floor (enforced in CI via the reusable workflow's bare pytest) red-exits local subset runs by design, with the --cov-fail-under=0 override for iteration. Floor stays in addopts because the centralized ci-python-sam workflow exposes no per-run test command.	2026-07-20 16:19:15 -04:00
Adam Moussa	ca4f43a2cc	feat: decompose email-processor handlers into flat siblings + lazy boto3 clients (refactor phase 5) (#113) ... Both email-processor God-handlers split along the seams that already work in the flat-sibling pattern established by lambdas/shared/, so bare-name imports keep working under the existing bundling glob. PO (5-way split): handler.py keeps only the event loop, fail-closed auth, and email_type routing. extraction.py holds extract_with_claude and _EMAIL_TAG_RE, importing EXTRACTION_PROMPT from prompts.py and parse_raw_email from shared/email_parsing.py rather than recreating a PO-local copy. enrichment.py is a pure code move of enrich_parsed and pad_zip (PO-only; WO has no enrichment stage) with zero behavior change. telemetry.py holds the EMF ParseMethod emit wrappers. persistence.py holds _write_fields/_merge_update/save_*, collapsing the byte-identical save_new_po/save_revision bodies into one _save_merge helper that both now call through, preserving the sticky Cancelled ConditionExpression guard for both callers; save_cancellation stays separate. WO (5 concerns, no enrichment stage): the handler loop keeps validate_ai_fallback and the re.fullmatch(r"[0-9]+", work_order_id) key guard ahead of both save_work_order and save_event, since the guard protects the DynamoDB partition key and the '#'-delimited comment_id range-key segment. _header_date_iso and comment_id determinism stay colocated with persistence.py's save_event for the retry-idempotent event_id key. EXTRACTION_PROMPT (PO) moves to prompts.py with cross-reference headers to derived_fields.py's authoritative trade/site/fiscal rule tables; handler.py re-exports it (from prompts import EXTRACTION_PROMPT) since four tests dereference handler.EXTRACTION_ PROMPT directly. WO's prompt moves the same way. I/O modules (extraction.py's bedrock client, persistence.py's dynamodb resource, handler.py's s3 client) get lazy cached boto3 accessors; pure modules (enrichment.py, prompts.py, telemetry.py) import no boto3. Test monkeypatch surfaces move to the module that now owns the client (e.g. persistence.dynamodb) everywhere tests patch it, and the moto-before-handler-import ordering in _po_parser_support.py is preserved so the moto-backed suites don't hit real AWS. Behavior-preservation pins, verified with tests: PO still emits ParseMethod=ai_fallback before the Bedrock call, with ai_fallback_rejected as the additive second datapoint on rejection. WO still emits after its gate with mutually-exclusive ai_fallback / ai_fallback_rejected. Shadow DerivedFieldAgreement telemetry stays ai_fallback-only. derived_fields.py is untouched (diff against feature/phase-3-shared-extraction is empty). handler(event, context) signatures and the save_* public contract are unchanged on both pipelines; goldens unchanged. PO_EXPECTED_TOP_LEVEL_MODULES and its WO equivalent in tests/test_bundle_consistency.py are updated for the new sibling modules so the AST bundle-consistency test still fails on an unshipped or uncommented-out sibling.	2026-07-20 15:34:53 -04:00
Adam Moussa	941a0ec0fe	Add pytest suite and enable tests in CI (#95) ... The repo had no automated tests, so regressions in the email parsing pipeline could only be caught manually. The reusable ci-python-sam workflow already supports a run-tests input at the pinned SHA; enable it and add a first suite covering the pure functions pad_zip and parse_raw_email. tests/conftest.py sets AWS_DEFAULT_REGION and dummy credentials before any handler import because the handlers create boto3 clients at module import time, and loads each handler.py via importlib under a unique module name since the files share a basename. pytest.ini scopes discovery to tests/ so the manual test_local.py script at the repo root is not collected.	2026-07-15 19:06:20 -04:00