procurement-ingest/lambdas/po/email_processor/tests/test_po_healthcheck.py

179 lines
6.8 KiB
Python
Raw Normal View History

feat: deploy-pipeline guards — healthcheck, smoke gate, bundle glob + AST test (refactor phase 0) (#107) * feat: deploy-pipeline guards — healthcheck, smoke gate, bundle glob + AST test (refactor phase 0) Deploys of po-email-processor and workorder-email-processor had no verification step, so an init-time ImportError in the bundled zip could ship silently and only surface on the next real S3 event. This adds a synchronous post-deploy smoke gate wired into the deploy workflow: both Lambdas are invoked with {"healthcheck": true} and the FunctionError field is checked, since an Unhandled init error still returns HTTP 200 on RequestResponse invokes and would false-pass a plain exit-code check. The healthcheck branch is the first statement in each handler, before any boto3/S3 use or ses_auth, and only fires on a top-level direct invoke ("healthcheck" is not a key AWS ever sets on a real S3 ObjectCreated event, so mail content can't reach this path). It emits no EMF metrics and no log text that could match the sender-auth-rejected metric filter, so two deploys in one window won't trip the alarm. Separately, the PO stack's asset bundling copied a hand-maintained four-file allowlist into the zip, so every new sibling module handler.py imports had to be added by hand or the deploy shipped a Lambda that ImportErrors at cold start (bit us for template_parser in PR #105 and nearly for derived_fields in PR #2). Replaced it with a non-recursive ./*.py glob so top-level source files ship automatically while tests/ and the stale package/ dir still cannot, and added an AST-based bundle-consistency test that parses each handler's first-party imports and fails CI if the bundling command would omit any of them (a revert to an incomplete allowlist, or code moved into a subdirectory the glob doesn't cover). Includes the refactor-evaluation report that scoped this phase. * fix: review nits — unambiguous bundling-command extraction, smoke payload-parse message, dead asserts - tests/test_bundle_consistency.py: _extract_bundling_command now collects all command=[...] matches and demands exactly one per stack file, instead of silently returning whichever ast.walk visits first if a second bundled function is ever added. - scripts/post-deploy-smoke.sh: distinguish an unparseable response payload from a payload mismatch so the failure message says what actually happened (the previous "could not parse" branch was unreachable — the inline python always exited 0). - test_po_healthcheck.py: drop the substring assertions on stdout that were dead behind the stricter `captured.out == ""` assertion; keep the stderr filter-pattern check. Review follow-up on PR #107; no behavior change to any shipped code path.
2026-07-17 13:18:45 -04:00
"""Direct-invoke healthcheck early-return tests for the PO handler.
The post-deploy smoke script invokes the Lambda synchronously with
``{"healthcheck": true}`` and asserts the response is ``{"healthcheck": "ok"}``.
That branch is pinned to run as the VERY FIRST thing handler() does -- before
any S3 fetch, before ses_auth, before the Records loop -- so it neither creates
an accept path for mail nor emits any EMF metric / log line that could trip the
``sender_auth_rejected`` substring metric-filter alarm on a deploy.
These tests import ``po_handler`` from ``_po_parser_support`` (moto imported
first, PO modules loaded by file path under unique names) exactly like the rest
of the PO suite, preserving the moto-before-handler import ordering.
"""
import pytest
feat: decompose email-processor handlers into flat siblings + lazy boto3 clients (refactor phase 5) (#113) Both email-processor God-handlers split along the seams that already work in the flat-sibling pattern established by lambdas/shared/, so bare-name imports keep working under the existing bundling glob. PO (5-way split): handler.py keeps only the event loop, fail-closed auth, and email_type routing. extraction.py holds extract_with_claude and _EMAIL_TAG_RE, importing EXTRACTION_PROMPT from prompts.py and parse_raw_email from shared/email_parsing.py rather than recreating a PO-local copy. enrichment.py is a pure code move of enrich_parsed and pad_zip (PO-only; WO has no enrichment stage) with zero behavior change. telemetry.py holds the EMF ParseMethod emit wrappers. persistence.py holds _write_fields/_merge_update/save_*, collapsing the byte-identical save_new_po/save_revision bodies into one _save_merge helper that both now call through, preserving the sticky Cancelled ConditionExpression guard for both callers; save_cancellation stays separate. WO (5 concerns, no enrichment stage): the handler loop keeps validate_ai_fallback and the re.fullmatch(r"[0-9]+", work_order_id) key guard ahead of both save_work_order and save_event, since the guard protects the DynamoDB partition key and the '#'-delimited comment_id range-key segment. _header_date_iso and comment_id determinism stay colocated with persistence.py's save_event for the retry-idempotent event_id key. EXTRACTION_PROMPT (PO) moves to prompts.py with cross-reference headers to derived_fields.py's authoritative trade/site/fiscal rule tables; handler.py re-exports it (from prompts import EXTRACTION_PROMPT) since four tests dereference handler.EXTRACTION_ PROMPT directly. WO's prompt moves the same way. I/O modules (extraction.py's bedrock client, persistence.py's dynamodb resource, handler.py's s3 client) get lazy cached boto3 accessors; pure modules (enrichment.py, prompts.py, telemetry.py) import no boto3. Test monkeypatch surfaces move to the module that now owns the client (e.g. persistence.dynamodb) everywhere tests patch it, and the moto-before-handler-import ordering in _po_parser_support.py is preserved so the moto-backed suites don't hit real AWS. Behavior-preservation pins, verified with tests: PO still emits ParseMethod=ai_fallback before the Bedrock call, with ai_fallback_rejected as the additive second datapoint on rejection. WO still emits after its gate with mutually-exclusive ai_fallback / ai_fallback_rejected. Shadow DerivedFieldAgreement telemetry stays ai_fallback-only. derived_fields.py is untouched (diff against feature/phase-3-shared-extraction is empty). handler(event, context) signatures and the save_* public contract are unchanged on both pipelines; goldens unchanged. PO_EXPECTED_TOP_LEVEL_MODULES and its WO equivalent in tests/test_bundle_consistency.py are updated for the new sibling modules so the AST bundle-consistency test still fails on an unshipped or uncommented-out sibling.
2026-07-20 15:34:53 -04:00
from _po_parser_support import load_raw, po_enrichment, po_handler, po_persistence
feat: deploy-pipeline guards — healthcheck, smoke gate, bundle glob + AST test (refactor phase 0) (#107) * feat: deploy-pipeline guards — healthcheck, smoke gate, bundle glob + AST test (refactor phase 0) Deploys of po-email-processor and workorder-email-processor had no verification step, so an init-time ImportError in the bundled zip could ship silently and only surface on the next real S3 event. This adds a synchronous post-deploy smoke gate wired into the deploy workflow: both Lambdas are invoked with {"healthcheck": true} and the FunctionError field is checked, since an Unhandled init error still returns HTTP 200 on RequestResponse invokes and would false-pass a plain exit-code check. The healthcheck branch is the first statement in each handler, before any boto3/S3 use or ses_auth, and only fires on a top-level direct invoke ("healthcheck" is not a key AWS ever sets on a real S3 ObjectCreated event, so mail content can't reach this path). It emits no EMF metrics and no log text that could match the sender-auth-rejected metric filter, so two deploys in one window won't trip the alarm. Separately, the PO stack's asset bundling copied a hand-maintained four-file allowlist into the zip, so every new sibling module handler.py imports had to be added by hand or the deploy shipped a Lambda that ImportErrors at cold start (bit us for template_parser in PR #105 and nearly for derived_fields in PR #2). Replaced it with a non-recursive ./*.py glob so top-level source files ship automatically while tests/ and the stale package/ dir still cannot, and added an AST-based bundle-consistency test that parses each handler's first-party imports and fails CI if the bundling command would omit any of them (a revert to an incomplete allowlist, or code moved into a subdirectory the glob doesn't cover). Includes the refactor-evaluation report that scoped this phase. * fix: review nits — unambiguous bundling-command extraction, smoke payload-parse message, dead asserts - tests/test_bundle_consistency.py: _extract_bundling_command now collects all command=[...] matches and demands exactly one per stack file, instead of silently returning whichever ast.walk visits first if a second bundled function is ever added. - scripts/post-deploy-smoke.sh: distinguish an unparseable response payload from a payload mismatch so the failure message says what actually happened (the previous "could not parse" branch was unreachable — the inline python always exited 0). - test_po_healthcheck.py: drop the substring assertions on stdout that were dead behind the stricter `captured.out == ""` assertion; keep the stderr filter-pattern check. Review follow-up on PR #107; no behavior change to any shipped code path.
2026-07-17 13:18:45 -04:00
class _ExplodingS3:
"""Any S3 access from the healthcheck path is a contract violation."""
def get_object(self, **kwargs): # noqa: N803
raise AssertionError(f"healthcheck must not touch S3: {kwargs}")
def _exploding_auth(*args, **kwargs):
raise AssertionError("healthcheck must not call ses_auth")
def _exploding_metric(*args, **kwargs):
raise AssertionError("healthcheck must not emit any parse metric")
# --- (1) healthcheck early-return: ok payload, zero side effects ---
def test_healthcheck_returns_ok_with_zero_side_effects(
fake_dynamo, monkeypatch, capsys
):
monkeypatch.setattr(po_handler, "s3", _ExplodingS3())
monkeypatch.setattr(po_handler, "authenticate_inbound_email", _exploding_auth)
monkeypatch.setattr(po_handler, "_emit_parse_method_metric", _exploding_metric)
feat: decompose email-processor handlers into flat siblings + lazy boto3 clients (refactor phase 5) (#113) Both email-processor God-handlers split along the seams that already work in the flat-sibling pattern established by lambdas/shared/, so bare-name imports keep working under the existing bundling glob. PO (5-way split): handler.py keeps only the event loop, fail-closed auth, and email_type routing. extraction.py holds extract_with_claude and _EMAIL_TAG_RE, importing EXTRACTION_PROMPT from prompts.py and parse_raw_email from shared/email_parsing.py rather than recreating a PO-local copy. enrichment.py is a pure code move of enrich_parsed and pad_zip (PO-only; WO has no enrichment stage) with zero behavior change. telemetry.py holds the EMF ParseMethod emit wrappers. persistence.py holds _write_fields/_merge_update/save_*, collapsing the byte-identical save_new_po/save_revision bodies into one _save_merge helper that both now call through, preserving the sticky Cancelled ConditionExpression guard for both callers; save_cancellation stays separate. WO (5 concerns, no enrichment stage): the handler loop keeps validate_ai_fallback and the re.fullmatch(r"[0-9]+", work_order_id) key guard ahead of both save_work_order and save_event, since the guard protects the DynamoDB partition key and the '#'-delimited comment_id range-key segment. _header_date_iso and comment_id determinism stay colocated with persistence.py's save_event for the retry-idempotent event_id key. EXTRACTION_PROMPT (PO) moves to prompts.py with cross-reference headers to derived_fields.py's authoritative trade/site/fiscal rule tables; handler.py re-exports it (from prompts import EXTRACTION_PROMPT) since four tests dereference handler.EXTRACTION_ PROMPT directly. WO's prompt moves the same way. I/O modules (extraction.py's bedrock client, persistence.py's dynamodb resource, handler.py's s3 client) get lazy cached boto3 accessors; pure modules (enrichment.py, prompts.py, telemetry.py) import no boto3. Test monkeypatch surfaces move to the module that now owns the client (e.g. persistence.dynamodb) everywhere tests patch it, and the moto-before-handler-import ordering in _po_parser_support.py is preserved so the moto-backed suites don't hit real AWS. Behavior-preservation pins, verified with tests: PO still emits ParseMethod=ai_fallback before the Bedrock call, with ai_fallback_rejected as the additive second datapoint on rejection. WO still emits after its gate with mutually-exclusive ai_fallback / ai_fallback_rejected. Shadow DerivedFieldAgreement telemetry stays ai_fallback-only. derived_fields.py is untouched (diff against feature/phase-3-shared-extraction is empty). handler(event, context) signatures and the save_* public contract are unchanged on both pipelines; goldens unchanged. PO_EXPECTED_TOP_LEVEL_MODULES and its WO equivalent in tests/test_bundle_consistency.py are updated for the new sibling modules so the AST bundle-consistency test still fails on an unshipped or uncommented-out sibling.
2026-07-20 15:34:53 -04:00
# _emit_derived_agreement_metric is owned by telemetry.py and imported into
# enrichment.py (enrich_parsed's binding); patch it on the enrichment module,
# which is where enrich_parsed would resolve it -- it is NOT re-exported on
# handler (handler never calls it directly).
monkeypatch.setattr(
po_enrichment, "_emit_derived_agreement_metric", _exploding_metric
)
feat: deploy-pipeline guards — healthcheck, smoke gate, bundle glob + AST test (refactor phase 0) (#107) * feat: deploy-pipeline guards — healthcheck, smoke gate, bundle glob + AST test (refactor phase 0) Deploys of po-email-processor and workorder-email-processor had no verification step, so an init-time ImportError in the bundled zip could ship silently and only surface on the next real S3 event. This adds a synchronous post-deploy smoke gate wired into the deploy workflow: both Lambdas are invoked with {"healthcheck": true} and the FunctionError field is checked, since an Unhandled init error still returns HTTP 200 on RequestResponse invokes and would false-pass a plain exit-code check. The healthcheck branch is the first statement in each handler, before any boto3/S3 use or ses_auth, and only fires on a top-level direct invoke ("healthcheck" is not a key AWS ever sets on a real S3 ObjectCreated event, so mail content can't reach this path). It emits no EMF metrics and no log text that could match the sender-auth-rejected metric filter, so two deploys in one window won't trip the alarm. Separately, the PO stack's asset bundling copied a hand-maintained four-file allowlist into the zip, so every new sibling module handler.py imports had to be added by hand or the deploy shipped a Lambda that ImportErrors at cold start (bit us for template_parser in PR #105 and nearly for derived_fields in PR #2). Replaced it with a non-recursive ./*.py glob so top-level source files ship automatically while tests/ and the stale package/ dir still cannot, and added an AST-based bundle-consistency test that parses each handler's first-party imports and fails CI if the bundling command would omit any of them (a revert to an incomplete allowlist, or code moved into a subdirectory the glob doesn't cover). Includes the refactor-evaluation report that scoped this phase. * fix: review nits — unambiguous bundling-command extraction, smoke payload-parse message, dead asserts - tests/test_bundle_consistency.py: _extract_bundling_command now collects all command=[...] matches and demands exactly one per stack file, instead of silently returning whichever ast.walk visits first if a second bundled function is ever added. - scripts/post-deploy-smoke.sh: distinguish an unparseable response payload from a payload mismatch so the failure message says what actually happened (the previous "could not parse" branch was unreachable — the inline python always exited 0). - test_po_healthcheck.py: drop the substring assertions on stdout that were dead behind the stricter `captured.out == ""` assertion; keep the stderr filter-pattern check. Review follow-up on PR #107; no behavior change to any shipped code path.
2026-07-17 13:18:45 -04:00
result = po_handler.handler({"healthcheck": True}, None)
assert result == {"healthcheck": "ok"}
# ZERO DynamoDB writes: no table was ever fetched/updated.
assert fake_dynamo.tables == {}
# ZERO EMF metric emission: EMF records go to stdout via print(); the branch
# must not have printed anything the sender_auth_rejected filter could match.
# (Empty stdout subsumes any substring check on it; stderr is checked for
# the filter pattern specifically.)
captured = capsys.readouterr()
assert captured.out == ""
assert "sender_auth_rejected" not in captured.err
def test_healthcheck_branch_precedes_records_key_lookup(monkeypatch):
"""The healthcheck return fires even when a hostile payload also carries a
top-level ``Records`` key: the branch is ordered before the loop, and the
exploding S3/auth prove the loop body never runs."""
monkeypatch.setattr(po_handler, "s3", _ExplodingS3())
monkeypatch.setattr(po_handler, "authenticate_inbound_email", _exploding_auth)
event = {
"healthcheck": True,
"Records": [
{"s3": {"bucket": {"name": "b"}, "object": {"key": "k"}}},
],
}
assert po_handler.handler(event, None) == {"healthcheck": "ok"}
@pytest.mark.parametrize(
"event",
[
{"healthcheck": False},
{"healthcheck": "true"},
{"healthcheck": 1},
{"healthcheck": {"nested": True}},
{"Healthcheck": True}, # wrong case: not the contract key
{},
],
)
def test_non_healthcheck_payloads_do_not_early_return(event):
"""Only a top-level ``healthcheck`` that is exactly ``True`` takes the
branch; anything else falls through to the (empty) Records loop and returns
the normal 200 body."""
assert po_handler.handler(event, None) == {"statusCode": 200, "body": "OK"}
# --- (2) a normal S3 mail event is completely unaffected ---
class _RecordingS3:
def __init__(self, raw):
self._raw = raw
self.calls = []
def get_object(self, Bucket, Key): # noqa: N803
self.calls.append((Bucket, Key))
class _Body:
def __init__(self, data):
self._data = data
def read(self):
return self._data
return {"Body": _Body(self._raw)}
def _s3_event():
return {
"Records": [
{
"s3": {
"bucket": {"name": "po-ingest-emails-x"},
"object": {"key": "inbound/hc"},
}
}
]
}
def test_normal_mail_event_still_processed(fake_dynamo, monkeypatch):
"""Golden-path guard: a real S3 ObjectCreated mail event is unaffected by
the healthcheck branch -- it still fetches from S3 and upserts the PO."""
recording = _RecordingS3(load_raw("new-po", "new-po-01"))
monkeypatch.setattr(po_handler, "s3", recording)
monkeypatch.setattr(po_handler, "authenticate_inbound_email", lambda *a: True)
result = po_handler.handler(_s3_event(), None)
assert result == {"statusCode": 200, "body": "OK"}
# The branch was NOT taken: S3 was fetched and the PO was written through.
assert recording.calls == [("po-ingest-emails-x", "inbound/hc")]
feat: decompose email-processor handlers into flat siblings + lazy boto3 clients (refactor phase 5) (#113) Both email-processor God-handlers split along the seams that already work in the flat-sibling pattern established by lambdas/shared/, so bare-name imports keep working under the existing bundling glob. PO (5-way split): handler.py keeps only the event loop, fail-closed auth, and email_type routing. extraction.py holds extract_with_claude and _EMAIL_TAG_RE, importing EXTRACTION_PROMPT from prompts.py and parse_raw_email from shared/email_parsing.py rather than recreating a PO-local copy. enrichment.py is a pure code move of enrich_parsed and pad_zip (PO-only; WO has no enrichment stage) with zero behavior change. telemetry.py holds the EMF ParseMethod emit wrappers. persistence.py holds _write_fields/_merge_update/save_*, collapsing the byte-identical save_new_po/save_revision bodies into one _save_merge helper that both now call through, preserving the sticky Cancelled ConditionExpression guard for both callers; save_cancellation stays separate. WO (5 concerns, no enrichment stage): the handler loop keeps validate_ai_fallback and the re.fullmatch(r"[0-9]+", work_order_id) key guard ahead of both save_work_order and save_event, since the guard protects the DynamoDB partition key and the '#'-delimited comment_id range-key segment. _header_date_iso and comment_id determinism stay colocated with persistence.py's save_event for the retry-idempotent event_id key. EXTRACTION_PROMPT (PO) moves to prompts.py with cross-reference headers to derived_fields.py's authoritative trade/site/fiscal rule tables; handler.py re-exports it (from prompts import EXTRACTION_PROMPT) since four tests dereference handler.EXTRACTION_ PROMPT directly. WO's prompt moves the same way. I/O modules (extraction.py's bedrock client, persistence.py's dynamodb resource, handler.py's s3 client) get lazy cached boto3 accessors; pure modules (enrichment.py, prompts.py, telemetry.py) import no boto3. Test monkeypatch surfaces move to the module that now owns the client (e.g. persistence.dynamodb) everywhere tests patch it, and the moto-before-handler-import ordering in _po_parser_support.py is preserved so the moto-backed suites don't hit real AWS. Behavior-preservation pins, verified with tests: PO still emits ParseMethod=ai_fallback before the Bedrock call, with ai_fallback_rejected as the additive second datapoint on rejection. WO still emits after its gate with mutually-exclusive ai_fallback / ai_fallback_rejected. Shadow DerivedFieldAgreement telemetry stays ai_fallback-only. derived_fields.py is untouched (diff against feature/phase-3-shared-extraction is empty). handler(event, context) signatures and the save_* public contract are unchanged on both pipelines; goldens unchanged. PO_EXPECTED_TOP_LEVEL_MODULES and its WO equivalent in tests/test_bundle_consistency.py are updated for the new sibling modules so the AST bundle-consistency test still fails on an unshipped or uncommented-out sibling.
2026-07-20 15:34:53 -04:00
assert fake_dynamo.tables[po_persistence.PO_TABLE].updates
feat: deploy-pipeline guards — healthcheck, smoke gate, bundle glob + AST test (refactor phase 0) (#107) * feat: deploy-pipeline guards — healthcheck, smoke gate, bundle glob + AST test (refactor phase 0) Deploys of po-email-processor and workorder-email-processor had no verification step, so an init-time ImportError in the bundled zip could ship silently and only surface on the next real S3 event. This adds a synchronous post-deploy smoke gate wired into the deploy workflow: both Lambdas are invoked with {"healthcheck": true} and the FunctionError field is checked, since an Unhandled init error still returns HTTP 200 on RequestResponse invokes and would false-pass a plain exit-code check. The healthcheck branch is the first statement in each handler, before any boto3/S3 use or ses_auth, and only fires on a top-level direct invoke ("healthcheck" is not a key AWS ever sets on a real S3 ObjectCreated event, so mail content can't reach this path). It emits no EMF metrics and no log text that could match the sender-auth-rejected metric filter, so two deploys in one window won't trip the alarm. Separately, the PO stack's asset bundling copied a hand-maintained four-file allowlist into the zip, so every new sibling module handler.py imports had to be added by hand or the deploy shipped a Lambda that ImportErrors at cold start (bit us for template_parser in PR #105 and nearly for derived_fields in PR #2). Replaced it with a non-recursive ./*.py glob so top-level source files ship automatically while tests/ and the stale package/ dir still cannot, and added an AST-based bundle-consistency test that parses each handler's first-party imports and fails CI if the bundling command would omit any of them (a revert to an incomplete allowlist, or code moved into a subdirectory the glob doesn't cover). Includes the refactor-evaluation report that scoped this phase. * fix: review nits — unambiguous bundling-command extraction, smoke payload-parse message, dead asserts - tests/test_bundle_consistency.py: _extract_bundling_command now collects all command=[...] matches and demands exactly one per stack file, instead of silently returning whichever ast.walk visits first if a second bundled function is ever added. - scripts/post-deploy-smoke.sh: distinguish an unparseable response payload from a payload mismatch so the failure message says what actually happened (the previous "could not parse" branch was unreachable — the inline python always exited 0). - test_po_healthcheck.py: drop the substring assertions on stdout that were dead behind the stricter `captured.out == ""` assertion; keep the stderr filter-pattern check. Review follow-up on PR #107; no behavior change to any shipped code path.
2026-07-17 13:18:45 -04:00
def test_healthcheck_string_in_email_body_does_not_take_branch(
fake_dynamo, monkeypatch
):
"""A mail event whose EMAIL BODY contains the string 'healthcheck' must NOT
trigger the early return: the trigger is a top-level direct-invoke key that
AWS controls, and email content can never set a top-level event key. Proof:
S3 is still fetched (the branch would have skipped it)."""
raw_email = (
b"From: buyer@amazon.coupahost.com\r\n"
b"To: po@seahavenind.com\r\n"
b"Subject: FYI healthcheck notes\r\n"
b"\r\n"
b"Please run a healthcheck on this order. healthcheck healthcheck.\r\n"
)
recording = _RecordingS3(raw_email)
monkeypatch.setattr(po_handler, "s3", recording)
monkeypatch.setattr(po_handler, "authenticate_inbound_email", lambda *a: True)
# The synthetic body has no Coupa template match, so parsing would fall to
# the Bedrock extractor; stub it (returning no po_number) so the record is
# skipped cleanly. What matters here is only that S3 WAS fetched -- i.e. the
# healthcheck branch did not short-circuit on the body text.
monkeypatch.setattr(po_handler, "extract_with_claude", lambda *a: {})
result = po_handler.handler(_s3_event(), None)
# Fell through to the Records loop (no early return): S3 WAS fetched. The
# synthetic body has no PO number, so it is skipped -- return is the normal
# 200 body, never the {"healthcheck": "ok"} smoke payload.
assert result == {"statusCode": 200, "body": "OK"}
assert recording.calls == [("po-ingest-emails-x", "inbound/hc")]