2026-07-15 19:06:20 -04:00
|
|
|
"""Shared pytest configuration for the procurement-ingest test suite.
|
|
|
|
|
|
|
|
|
|
The Lambda handlers create boto3 clients at module import time, so a
|
|
|
|
|
region and dummy credentials must be present in the environment before
|
|
|
|
|
any handler module is imported. Setting them here at conftest import
|
|
|
|
|
time guarantees they exist before test collection touches a handler.
|
|
|
|
|
"""
|
|
|
|
|
|
|
|
|
|
import importlib.util
|
|
|
|
|
import os
|
|
|
|
|
import sys
|
|
|
|
|
from pathlib import Path
|
|
|
|
|
|
|
|
|
|
import pytest
|
|
|
|
|
|
|
|
|
|
os.environ.setdefault("AWS_DEFAULT_REGION", "us-east-1")
|
|
|
|
|
os.environ.setdefault("AWS_ACCESS_KEY_ID", "testing")
|
|
|
|
|
os.environ.setdefault("AWS_SECRET_ACCESS_KEY", "testing")
|
|
|
|
|
os.environ.setdefault("AWS_SESSION_TOKEN", "testing")
|
|
|
|
|
|
|
|
|
|
REPO_ROOT = Path(__file__).resolve().parents[1]
|
|
|
|
|
|
|
|
|
|
|
Add offline PO parser suite with scrubbed fixture corpus
132 tests: golden-file comparison for all 25 positive fixtures
(17 single-line new-PO + 8 cancellations, Decimal-exact via
parse_float=Decimal), every fail-closed gate reason code covered
(body-level triggers via 17 synthetic adversarial .eml mutations,
candidate-level via direct validate() unit tests), real multi-line
and comment/non-Coupa fallback fixtures, dual line-ending parse
identity, two-path enrich/save parity (site-extractor stream
guard), V10 URL-id corpus sweep, fixture hygiene (ses_auth pass +
scrub-marker leak sweep), and Bedrock dispatch/EMF assertions.
The suite loads handler/template_parser via importlib under
unique module names and binds the handler's bare sibling imports
around exec (tests/conftest.py load_handler gets the same
treatment) -- the WO suite caches bare 'handler'/'template_parser'
names in sys.modules, and bare imports here would silently bind
to the wrong pipeline. moto is imported before the handler so its
botocore stubber hook precedes boto3 session creation (the PO
conftest chain now loads at pytest session start).
Fixtures are scrubbed real S3 samples: transport/auth header
values replaced with same-shape placeholders (structure kept so
ses_auth still passes), per-file digit ciphers, amounts remapped
with sum==total re-established. The .gitignore exception is
scoped to the PO fixtures path only.
Refs: #99
2026-07-16 15:15:49 -04:00
|
|
|
# Sibling modules that exist per-pipeline and are imported by bare name from
|
|
|
|
|
# the handlers (the Lambda runtime puts each function's own directory on
|
|
|
|
|
# sys.path). Both pipelines duplicate these filenames, so the bare names MUST
|
|
|
|
|
# be bound to the right pipeline's file around each handler exec -- relying on
|
|
|
|
|
# sys.path ordering (or on whatever a previously collected suite left in
|
|
|
|
|
# sys.modules) silently binds a handler to the OTHER pipeline's sibling.
|
|
|
|
|
_SIBLING_MODULES = ("ses_auth", "template_parser")
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def _load_module(path, module_name):
|
|
|
|
|
if module_name in sys.modules:
|
|
|
|
|
return sys.modules[module_name]
|
|
|
|
|
spec = importlib.util.spec_from_file_location(module_name, path)
|
|
|
|
|
module = importlib.util.module_from_spec(spec)
|
|
|
|
|
sys.modules[module_name] = module
|
|
|
|
|
spec.loader.exec_module(module)
|
|
|
|
|
return module
|
|
|
|
|
|
|
|
|
|
|
2026-07-15 19:06:20 -04:00
|
|
|
def load_handler(relative_path, module_name):
|
|
|
|
|
"""Load a Lambda handler module by file path under a unique name.
|
|
|
|
|
|
|
|
|
|
The handler files all share the basename ``handler.py`` and are not
|
|
|
|
|
importable as packages, so a plain ``import handler`` would collide
|
Add fail-closed SES sender authentication (INFRA-107) (#98)
* Add fail-closed SES sender authentication
The From header and any raw-MIME Authentication-Results copies are
attacker-forgeable, so a forged email to apm@int.seahaven.com or
amazon_po@int.seahaven.com could create or mutate a WO/PO (INFRA-107,
CRITICAL). Both S3-triggered email processors now authenticate the
sender against the Authentication-Results header SES itself prepends
at delivery: only the topmost header is consulted, its authserv-id
must be amazonses.com, and it must carry dkim=pass for a domain in
the per-pipeline ALLOWED_DKIM_DOMAINS env var (comma-separated, set
in CDK so ops can adjust without code changes).
Allowlists come from live traffic observed 2026-07-15 on both ingest
buckets: WO mail arrives via the apm@ Google Groups forward, which
re-signs as seahaven.com (the hxgnsmartcloud.com signature does not
survive the forward); PO mail passes for amazon.coupahost.com.
amazonses.com also passes on PO mail but is deliberately excluded --
every SES customer's outbound mail passes for it.
Every failure path (env var unset, header missing or unparseable,
verdict fail, unaligned domain) rejects the email: a structured
warning with the reason and S3 key is logged and the record skipped
without erroring the invocation, so rejected mail causes no Lambda
retries or DLQ messages. Handler signatures and event sources are
unchanged.
Refs: INFRA-107
* Harden AR parser per cross-family review
Cross-family (GPT-4.1) review findings: terminate the dkim result
token at end-of-clause, whitespace, or a comment so a value like
"dkim=pass-fake" can never be read as a pass; normalize trailing
dots off allowlist entries so "seahaven.com." matches; make the
compat32 parser policy explicit. Adds tests for result-token
boundaries, comments after the result, quoted domain values, and
folding inside a dkim clause.
Refs: INFRA-107
* Harden AR parsing and alarm on sender-auth rejects
The SES-stamped Authentication-Results value echoes attacker-controlled
SMTP-session tokens (envelope-from, helo, header.from) as their own
semicolon-delimited property clauses. A naive split(";") tore an RFC 5321
quoted-local-part MAIL FROM apart and manufactured a forged dkim=pass
clause, so a fully spoofed email was accepted on the genuinely
SES-stamped topmost header. Tokenise comment- and quoted-string-aware
(RFC 8601 / RFC 5322): strip CFWS comments, split clauses only on
semicolons outside a quoted-string, and fail closed on unbalanced
quotes/comments so a ';' inside a quoted pvalue can never start a clause.
Rejected mail returns normally (no error, no retry, no DLQ message), so a
signing-domain drift or a wrong allowlist would silently discard 100% of
legitimate mail while every alarm stayed green. Add a CloudWatch Logs
metric filter + alarm on the sender_auth_rejected warning to both stacks
so a false-reject storm pages instead of vanishing. This is also the
safety net for the WO seahaven.com allowlist assumption, which must be
validated against a live SES-stamped header (a plain Gmail auto-forward
re-signs under the sending Workspace domain, not seahaven.com).
Refs: INFRA-107
* chore: retrigger CI (no run recorded for 7c74ac1)
* Fix quoted-AUID DKIM domain spoof in sender auth
Resolve three confirmed /sh-security-review findings on the fail-closed
SES sender-authentication control.
HIGH: header.i/header.d domain extraction was not quoted-string aware.
An attacker with a valid DKIM key for their own domain could set an
RFC 6376-legal AUID such as i="@seahaven.com"@attacker.com; the naive
extractor stopped at the closing quote and returned seahaven.com,
accepting forged mail. Extraction now tokenises the clause with the same
quoted-string discipline already used for clause splitting: header.d
(the plain signing domain) is authoritative when present, otherwise the
header.i domain is the part after the AUID's LAST top-level "@", so a "@"
inside a quoted local-part is treated as signer-controlled label text and
yields the true signer (attacker.com), not seahaven.com.
LOW: the topmost-header parse ran outside evaluate_sender_authentication's
try/except, so an unexpected parser exception on crafted input could
propagate into the handler and Lambda async retries/DLQ. The parse now
fails CLOSED with an authentication_results_unparseable reason.
MEDIUM: the sender_auth_rejected alarm used Sum>=3 over 15 min, blind to
a low-volume total-reject outage (a trickle that never sums to 3). Both
stacks now alarm on >=1 reject per 5-min period with evaluation_periods=3
/ datapoints_to_alarm=2, so a sustained reject condition pages even at one
reject per period while a lone stray probe self-clears.
Refs: INFRA-107
* Load Lambda function dir on sys.path in tests
Rebasing INFRA-107 onto main folded #95's pytest suite into this
branch's tests. The unified conftest loads the PO/WO handlers by file
path, and handler.py now does `from ses_auth import
authenticate_inbound_email` -- a bare sibling import that resolves in
the Lambda only because the runtime puts each function's own directory
on sys.path. The shared load_handler now adds that directory so the
handler tests import correctly alongside the sender-auth tests.
Refs: INFRA-107
* Note #97 test files in README directory tree
The rebase onto main brought in #97's tests/requirements.txt and
tests/test_po_merge.py. List both in the directory tree so it matches
the tree on disk.
Refs: INFRA-107
* Document INFRA-107 forwarder-binding risk acceptance
Record the accepted risk that WO sender auth binds to the apm@ forward's
re-signing domain (seahaven.com) rather than the Hexagon originator; the
apm@ Google Group's restricted posting policy is the load-bearing control
(escalates to HIGH if the group is opened to external posting). Also
correct the sender-auth-rejected alarm docs to match the shipped config
(>=1 per 5-min, 2-of-3 datapoints, not the superseded >=3/15min) and
note the SES-AR-01/02 parser hardening follow-ups.
Refs: INFRA-107
2026-07-15 20:58:47 -04:00
|
|
|
across Lambdas. The same loader serves the ``ses_auth.py`` modules,
|
|
|
|
|
which are likewise duplicated per pipeline and not importable as
|
|
|
|
|
packages.
|
|
|
|
|
|
Add offline PO parser suite with scrubbed fixture corpus
132 tests: golden-file comparison for all 25 positive fixtures
(17 single-line new-PO + 8 cancellations, Decimal-exact via
parse_float=Decimal), every fail-closed gate reason code covered
(body-level triggers via 17 synthetic adversarial .eml mutations,
candidate-level via direct validate() unit tests), real multi-line
and comment/non-Coupa fallback fixtures, dual line-ending parse
identity, two-path enrich/save parity (site-extractor stream
guard), V10 URL-id corpus sweep, fixture hygiene (ses_auth pass +
scrub-marker leak sweep), and Bedrock dispatch/EMF assertions.
The suite loads handler/template_parser via importlib under
unique module names and binds the handler's bare sibling imports
around exec (tests/conftest.py load_handler gets the same
treatment) -- the WO suite caches bare 'handler'/'template_parser'
names in sys.modules, and bare imports here would silently bind
to the wrong pipeline. moto is imported before the handler so its
botocore stubber hook precedes boto3 session creation (the PO
conftest chain now loads at pytest session start).
Fixtures are scrubbed real S3 samples: transport/auth header
values replaced with same-shape placeholders (structure kept so
ses_auth still passes), per-file digit ciphers, amounts remapped
with sum==total re-established. The .gitignore exception is
scoped to the PO fixtures path only.
Refs: #99
2026-07-16 15:15:49 -04:00
|
|
|
Handler modules import their siblings by bare name (e.g. ``from
|
|
|
|
|
template_parser import try_deterministic_parse``). Each sibling is loaded
|
|
|
|
|
from the handler's own directory under a unique module name and registered
|
|
|
|
|
under its bare name only for the duration of the handler exec, then the
|
|
|
|
|
previous binding is restored -- so this loader is deterministic regardless
|
|
|
|
|
of collection order and of what the per-Lambda test suites (which put
|
|
|
|
|
their own module dir on sys.path) have already cached in sys.modules.
|
2026-07-15 19:06:20 -04:00
|
|
|
"""
|
|
|
|
|
path = REPO_ROOT / relative_path
|
Add offline PO parser suite with scrubbed fixture corpus
132 tests: golden-file comparison for all 25 positive fixtures
(17 single-line new-PO + 8 cancellations, Decimal-exact via
parse_float=Decimal), every fail-closed gate reason code covered
(body-level triggers via 17 synthetic adversarial .eml mutations,
candidate-level via direct validate() unit tests), real multi-line
and comment/non-Coupa fallback fixtures, dual line-ending parse
identity, two-path enrich/save parity (site-extractor stream
guard), V10 URL-id corpus sweep, fixture hygiene (ses_auth pass +
scrub-marker leak sweep), and Bedrock dispatch/EMF assertions.
The suite loads handler/template_parser via importlib under
unique module names and binds the handler's bare sibling imports
around exec (tests/conftest.py load_handler gets the same
treatment) -- the WO suite caches bare 'handler'/'template_parser'
names in sys.modules, and bare imports here would silently bind
to the wrong pipeline. moto is imported before the handler so its
botocore stubber hook precedes boto3 session creation (the PO
conftest chain now loads at pytest session start).
Fixtures are scrubbed real S3 samples: transport/auth header
values replaced with same-shape placeholders (structure kept so
ses_auth still passes), per-file digit ciphers, amounts remapped
with sum==total re-established. The .gitignore exception is
scoped to the PO fixtures path only.
Refs: #99
2026-07-16 15:15:49 -04:00
|
|
|
if module_name in sys.modules:
|
|
|
|
|
return sys.modules[module_name]
|
|
|
|
|
# Keep the handler dir on sys.path for parity with the Lambda runtime.
|
2026-07-15 20:11:04 -04:00
|
|
|
handler_dir = str(path.parent)
|
|
|
|
|
if handler_dir not in sys.path:
|
|
|
|
|
sys.path.insert(0, handler_dir)
|
Add offline PO parser suite with scrubbed fixture corpus
132 tests: golden-file comparison for all 25 positive fixtures
(17 single-line new-PO + 8 cancellations, Decimal-exact via
parse_float=Decimal), every fail-closed gate reason code covered
(body-level triggers via 17 synthetic adversarial .eml mutations,
candidate-level via direct validate() unit tests), real multi-line
and comment/non-Coupa fallback fixtures, dual line-ending parse
identity, two-path enrich/save parity (site-extractor stream
guard), V10 URL-id corpus sweep, fixture hygiene (ses_auth pass +
scrub-marker leak sweep), and Bedrock dispatch/EMF assertions.
The suite loads handler/template_parser via importlib under
unique module names and binds the handler's bare sibling imports
around exec (tests/conftest.py load_handler gets the same
treatment) -- the WO suite caches bare 'handler'/'template_parser'
names in sys.modules, and bare imports here would silently bind
to the wrong pipeline. moto is imported before the handler so its
botocore stubber hook precedes boto3 session creation (the PO
conftest chain now loads at pytest session start).
Fixtures are scrubbed real S3 samples: transport/auth header
values replaced with same-shape placeholders (structure kept so
ses_auth still passes), per-file digit ciphers, amounts remapped
with sum==total re-established. The .gitignore exception is
scoped to the PO fixtures path only.
Refs: #99
2026-07-16 15:15:49 -04:00
|
|
|
if path.name != "handler.py":
|
|
|
|
|
# Leaf modules (e.g. ses_auth.py itself) have no sibling imports.
|
|
|
|
|
return _load_module(path, module_name)
|
|
|
|
|
saved = {}
|
|
|
|
|
for sibling in _SIBLING_MODULES:
|
|
|
|
|
sibling_path = path.parent / f"{sibling}.py"
|
|
|
|
|
if not sibling_path.exists():
|
|
|
|
|
continue
|
|
|
|
|
saved[sibling] = sys.modules.get(sibling)
|
|
|
|
|
sys.modules[sibling] = _load_module(sibling_path, f"{module_name}__{sibling}")
|
|
|
|
|
try:
|
|
|
|
|
module = _load_module(path, module_name)
|
|
|
|
|
finally:
|
|
|
|
|
for sibling, previous in saved.items():
|
|
|
|
|
if previous is not None:
|
|
|
|
|
sys.modules[sibling] = previous
|
|
|
|
|
else:
|
|
|
|
|
sys.modules.pop(sibling, None)
|
2026-07-15 19:06:20 -04:00
|
|
|
return module
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.fixture(scope="session")
|
|
|
|
|
def po_handler():
|
|
|
|
|
"""The PO email processor handler module."""
|
|
|
|
|
return load_handler(
|
|
|
|
|
"lambdas/po/email_processor/handler.py",
|
|
|
|
|
"po_email_processor_handler",
|
|
|
|
|
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.fixture(scope="session")
|
|
|
|
|
def wo_handler():
|
|
|
|
|
"""The WO email processor handler module."""
|
|
|
|
|
return load_handler(
|
|
|
|
|
"lambdas/wo/email_processor/handler.py",
|
|
|
|
|
"wo_email_processor_handler",
|
|
|
|
|
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.fixture(params=["po_handler", "wo_handler"])
|
|
|
|
|
def email_handler(request):
|
|
|
|
|
"""Parametrized fixture yielding each email processor handler module."""
|
|
|
|
|
return request.getfixturevalue(request.param)
|
Add fail-closed SES sender authentication (INFRA-107) (#98)
* Add fail-closed SES sender authentication
The From header and any raw-MIME Authentication-Results copies are
attacker-forgeable, so a forged email to apm@int.seahaven.com or
amazon_po@int.seahaven.com could create or mutate a WO/PO (INFRA-107,
CRITICAL). Both S3-triggered email processors now authenticate the
sender against the Authentication-Results header SES itself prepends
at delivery: only the topmost header is consulted, its authserv-id
must be amazonses.com, and it must carry dkim=pass for a domain in
the per-pipeline ALLOWED_DKIM_DOMAINS env var (comma-separated, set
in CDK so ops can adjust without code changes).
Allowlists come from live traffic observed 2026-07-15 on both ingest
buckets: WO mail arrives via the apm@ Google Groups forward, which
re-signs as seahaven.com (the hxgnsmartcloud.com signature does not
survive the forward); PO mail passes for amazon.coupahost.com.
amazonses.com also passes on PO mail but is deliberately excluded --
every SES customer's outbound mail passes for it.
Every failure path (env var unset, header missing or unparseable,
verdict fail, unaligned domain) rejects the email: a structured
warning with the reason and S3 key is logged and the record skipped
without erroring the invocation, so rejected mail causes no Lambda
retries or DLQ messages. Handler signatures and event sources are
unchanged.
Refs: INFRA-107
* Harden AR parser per cross-family review
Cross-family (GPT-4.1) review findings: terminate the dkim result
token at end-of-clause, whitespace, or a comment so a value like
"dkim=pass-fake" can never be read as a pass; normalize trailing
dots off allowlist entries so "seahaven.com." matches; make the
compat32 parser policy explicit. Adds tests for result-token
boundaries, comments after the result, quoted domain values, and
folding inside a dkim clause.
Refs: INFRA-107
* Harden AR parsing and alarm on sender-auth rejects
The SES-stamped Authentication-Results value echoes attacker-controlled
SMTP-session tokens (envelope-from, helo, header.from) as their own
semicolon-delimited property clauses. A naive split(";") tore an RFC 5321
quoted-local-part MAIL FROM apart and manufactured a forged dkim=pass
clause, so a fully spoofed email was accepted on the genuinely
SES-stamped topmost header. Tokenise comment- and quoted-string-aware
(RFC 8601 / RFC 5322): strip CFWS comments, split clauses only on
semicolons outside a quoted-string, and fail closed on unbalanced
quotes/comments so a ';' inside a quoted pvalue can never start a clause.
Rejected mail returns normally (no error, no retry, no DLQ message), so a
signing-domain drift or a wrong allowlist would silently discard 100% of
legitimate mail while every alarm stayed green. Add a CloudWatch Logs
metric filter + alarm on the sender_auth_rejected warning to both stacks
so a false-reject storm pages instead of vanishing. This is also the
safety net for the WO seahaven.com allowlist assumption, which must be
validated against a live SES-stamped header (a plain Gmail auto-forward
re-signs under the sending Workspace domain, not seahaven.com).
Refs: INFRA-107
* chore: retrigger CI (no run recorded for 7c74ac1)
* Fix quoted-AUID DKIM domain spoof in sender auth
Resolve three confirmed /sh-security-review findings on the fail-closed
SES sender-authentication control.
HIGH: header.i/header.d domain extraction was not quoted-string aware.
An attacker with a valid DKIM key for their own domain could set an
RFC 6376-legal AUID such as i="@seahaven.com"@attacker.com; the naive
extractor stopped at the closing quote and returned seahaven.com,
accepting forged mail. Extraction now tokenises the clause with the same
quoted-string discipline already used for clause splitting: header.d
(the plain signing domain) is authoritative when present, otherwise the
header.i domain is the part after the AUID's LAST top-level "@", so a "@"
inside a quoted local-part is treated as signer-controlled label text and
yields the true signer (attacker.com), not seahaven.com.
LOW: the topmost-header parse ran outside evaluate_sender_authentication's
try/except, so an unexpected parser exception on crafted input could
propagate into the handler and Lambda async retries/DLQ. The parse now
fails CLOSED with an authentication_results_unparseable reason.
MEDIUM: the sender_auth_rejected alarm used Sum>=3 over 15 min, blind to
a low-volume total-reject outage (a trickle that never sums to 3). Both
stacks now alarm on >=1 reject per 5-min period with evaluation_periods=3
/ datapoints_to_alarm=2, so a sustained reject condition pages even at one
reject per period while a lone stray probe self-clears.
Refs: INFRA-107
* Load Lambda function dir on sys.path in tests
Rebasing INFRA-107 onto main folded #95's pytest suite into this
branch's tests. The unified conftest loads the PO/WO handlers by file
path, and handler.py now does `from ses_auth import
authenticate_inbound_email` -- a bare sibling import that resolves in
the Lambda only because the runtime puts each function's own directory
on sys.path. The shared load_handler now adds that directory so the
handler tests import correctly alongside the sender-auth tests.
Refs: INFRA-107
* Note #97 test files in README directory tree
The rebase onto main brought in #97's tests/requirements.txt and
tests/test_po_merge.py. List both in the directory tree so it matches
the tree on disk.
Refs: INFRA-107
* Document INFRA-107 forwarder-binding risk acceptance
Record the accepted risk that WO sender auth binds to the apm@ forward's
re-signing domain (seahaven.com) rather than the Hexagon originator; the
apm@ Google Group's restricted posting policy is the load-bearing control
(escalates to HIGH if the group is opened to external posting). Also
correct the sender-auth-rejected alarm docs to match the shipped config
(>=1 per 5-min, 2-of-3 datapoints, not the superseded >=3/15min) and
note the SES-AR-01/02 parser hardening follow-ups.
Refs: INFRA-107
2026-07-15 20:58:47 -04:00
|
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.fixture(params=["wo", "po"])
|
|
|
|
|
def ses_auth(request):
|
|
|
|
|
"""The ses_auth module of each pipeline (duplicated file, kept in sync)."""
|
|
|
|
|
pipeline = request.param
|
|
|
|
|
return load_handler(
|
|
|
|
|
f"lambdas/{pipeline}/email_processor/ses_auth.py",
|
|
|
|
|
f"{pipeline}_ses_auth",
|
|
|
|
|
)
|