10 commits

Author	SHA1	Message	Date
Adam Moussa	5a3729c583	chore(infra): remove cdk tree after hcp cutover (PLAT-89) (#164) ... * chore(infra): remove cdk tree after hcp cutover Delete retired CDK sources, retarget bundle/principal contract tests to Terraform packaging, disable CDK synth in CI, and scrub deploy-adjacent docs. * fix(test): restore exact SHOC principal pin in terraform Pin shoc_consumer_role_arn's Terraform default and example to the trusted ARN, and require grant sites to consume local.shoc_consumer_role_arn only.	2026-08-07 12:20:29 -04:00
Adam Moussa	c040050373	feat(webhook): SHOC WO webhook emitter - dark-ship streams + HMAC secret/rotation (PR-2) (#137) ... * docs(webhook): revise SHOC webhook contract and plan for post-migration reality Branch re-cut on main 2026-07-23 (old base carried stale PR #99 commits). Contract Rev 2026-07-23: - Producer account corrected: seahaven-prod (011934824531); mgmt frozen - Reconciliation backstop is the new procurement read API, not SyncController - wo_status "unknown" is real; SHOC must map it (checklist item added) - write_origin forward-compat note for phase-2 write-back echo suppression - SyncVendorReplies retirement flagged (dead table, no vendor_reply event) Plan updates: - Account gate: seahaven-prod only; never enable streams on mgmt tables - Emitter ships DARK (ESMs enabled=False); activation is a deliberate flip after the SHOC receiver passes shared HMAC vectors - Post-refactor conventions: common.py helpers, bundle-consistency AST pins, pytest.ini --cov additions, consolidated test roots - Dedicated-CMK rationale, secret-ARN handooff step, consumer audit refreshed (slack-bot decommissioned), enum golden test, write_origin skip-branch test * feat(webhook): SHOC WO webhook emitter — dark-ship streams, HMAC secret + rotation Implements docs/shoc-webhook-plan.md Phases 1-5 (PR-2 of the SHOC call-and-be-called effort). Everything ships DARK: both DynamoDB event source mappings deploy enabled=False; activation is a deliberate one-line follow-up PR gated on the SHOC receiver passing the shared HMAC test vectors. - Streams: NEW_AND_OLD_IMAGES on WorkOrders + WorkOrderComments (in-place update, RETAIN + logical IDs untouched; no existing consumers — verified live, neither table had a stream). - workorder-shoc-emitter (Py3.12/ARM64): stream -> envelope -> HMAC-signed POST per docs/shoc-webhook-contract.md; strict per-shard ordering (parallelization 1, bisect off, retry until 24h age, ReportBatchItemFailures); 429/5xx/timeout block the shard in order, other 4xx park to workorder-shoc-emitter-rejected; ESM failures -> workorder-shoc-emitter-failures (metadata; replay rebuilds from DynamoDB). Echo guard skips write_origin=shoc-write-api. - Secret workorder-ingest/shoc-webhook-hmac on a dedicated CMK (alias workorder-ingest-shoc-webhook-kms); cross-account GetSecretValue/DescribeSecret + kms:Decrypt granted to exactly arn:aws:iam::396287094661:role/shoc-backend-dev. RemovalPolicy DESTROY deliberately (machine-generated material; avoids the fixed-name RETAIN-orphan deadlock). - workorder-shoc-hmac-rotator: 30-day rotation, dual-key overlap, 64-hex keys, kid = UTC %Y-%m-%dT%H. - Alarms (ALARM-only -> site-alerts): emitter errors/throttles/ duration + iterator-age (>=10 min) + failures/rejected queue depth; rotator standard trio. - scripts/replay_shoc_webhooks.py: dry-run-default operator replay (rebuilds from tables, replay:true envelopes). - Tests: 742 passing, 85.56% aggregate; golden HMAC vectors shared with SHOC in docs/shoc-webhook-test-vectors.json (emitter + replay signing pinned to identical vectors); bundle-consistency AST pins for both new bundles. - README: WO stack + webhook feed section, alarm table, runbooks; removed stale seahaven-slack-bot consumer references. * fix(webhook): kms:ViaService pins, https-only delivery, cross-account principal CI pin GPT-4.1 cross-family review of the policy surface (no BLOCK): FIX applied to the cross-account shoc-backend-dev Decrypt statement and both Lambda role KMS grants (the key is only ever used via Secrets Manager); its invariant-enforcement QUESTION answered durably with tests/test_cross_account_principal_pin.py (any new foreign IAM principal in cdk/ fails CI). Scanner mediums fixed: delivery.py and the replay script now refuse non-https URLs (urllib follows file:// and http://). SQS metadata-action and dynamodb:ListStreams NITs skipped: standard CDK grant shapes; ListStreams has no resource-level scoping. The 4 gitleaks HIGHs on docs/shoc-webhook-test-vectors.json are deliberate non-secrets (shared receiver-verification vectors) suppressed machine-level with justification. * harden(webhook): resolve /sh-security-review findings (1 confirmed medium + cheap fixes) High-recall detector fan-out (injection/authz/secrets-crypto/iac-iam/logic) + proof-or-kill verifier. Gate PASSES: 1 confirmed medium, 0 confirmed critical/high. Confirmed finding fixed; several unverified-but-cheap hardenings applied since the emitter ships dark and activation is weeks out. - CONFIRMED medium (confused deputy): the rotation Lambda's generated invoke permission for secretsmanager.amazonaws.com carried no SourceAccount/SourceArn, so any account's Secrets Manager could invoke the rotator. Patched the generated CfnPermission in place (a second permission would be additive, not restrictive) to pin account + this secret ARN. - delivery + replay: refuse to follow receiver 3xx redirects (no-redirect opener) so live X-SH-* auth headers can't be forwarded to a receiver-chosen Location and an http:// Location can't slip past the https guard. Fixed the "unfollowed 3xx" comment that was factually wrong. - delivery: classify 401/403 as retryable (invalidate key cache + retry in order) instead of parking -- transient auth failures (rotation outran the TTL cache, clock skew) are availability events, not contract bugs. - envelope: build_event now genuinely total (guarded eventID / ApproximateCreationDateTime subscripts) per its own never-raise contract. - handler: catch-all so an unexpected per-record error (e.g. SQS park failure) reports only that record instead of failing the whole batch (which would re-deliver every earlier success for 24h); per-invocation emit/skip batch summary so a systemic silent drop is queryable/alarmable. - rotator: narrow the AWSCURRENT-read except to ResourceNotFound/JSONDecode (transient SM/KMS errors re-raise so the overlap key isn't silently dropped); kid uniqueness checked against ALL retained kids with a random suffix on collision (never reissue a kid for a different secret). - contract: skeleton-upsert required on ANY unknown work_order_id (not just comment-before-create) + monotonicity guard (ignore older updated_at), so a parked created or an out-of-order replay can't corrupt receiver state. Unverified/refuted findings left as-is with rationale: the two "high" logic claims (whole-batch crash triggers, ordering violation) were refuted on reachability (real stream records carry required fields; persistence writes strings only; full-state idempotent upsert absorbs the ordering gap). Signed kid/version binding (AUTHZ-002) declined: coordinated contract change, not cheap, no exploit with one algorithm/key. * fix(webhook): drop kid from rotator test_ok log (CodeQL clear-text-logging FP) GHAS CodeQL flagged py/clear-text-logging-sensitive-data (high) at _test_secret's success log because head["kid"] is subscripted from the same parsed-secret dict that holds head["secret"] — the taint tracker can't tell the non-secret key id from the secret. The secret value is never logged. Rather than dismiss the alert (fragile; re-alerts on line moves), remove the flow: kid is already logged at stage time in _create_secret and version_id correlates the steps, so the test_ok log keeps only event + version_id. Also hardens against a future edit that swaps the logged field.	2026-07-24 22:12:20 +00:00
Adam Moussa	b89e98a879	feat(api): Redocly lint gate + SHOC-themed /docs (Redoc theming, topbar, collapsible samples) (#130) ... * feat(api): Add @redocly/cli as a dev dependency Signed-off-by: Adam Moussa <adam@seahavenind.com> * feat(api): Add Redocly configuration file with custom rules Signed-off-by: Adam Moussa <adam@seahavenind.com> * chore(api): Redocly lint config + bring openapi.json into compliance redocly.yaml from the Redocly guidelines builder, with three generated rules corrected: response-contains-property had the status codes as the required body fields (intent was the Error schema's top-level 'error'; 403 exempt since API Gateway emits AWS's {message} shape, 501 not 503); operation-4xx-problem-details-rfc7807 off (adopting RFC 7807 would be a runtime + SHOC-contract change, decided against); the two inert casing rules (parameter names, schema properties) removed because both name sets are contract-pinned (gateway resource paths, DynamoDB items). Spec changes, no runtime impact: operationIds renamed to method-prefixed kebab-case (get-work-orders, post-work-order-comment, ...); tags added to all 15 operations + root tags object (groups the Redoc sidebar); examples on all six parameters; license field; server description punctuation; two descriptions reworded to start capitalized. Real linter catches fixed: the two x-planned ops were missing their {workOrderId} path parameter and any 4xx response (403 added - true today, gateway rejects unsigned). .redocly.lint-ignore.yaml pins the six deliberate exceptions: webhook keys are the shipped SHOC contract event names (not renameable), and the x-planned ops answer only 501 (no 2xx to document). package.json: npm run lint:api. Verified: lint 0 errors, 675 pytest, headless-Chrome render of the tagged docs page. * feat(api): SHOC design-system theme for /docs (vendored fonts) Themes the Redoc page with the canonical SHOC token set: Montserrat 600 headings / DM Sans body / JetBrains Mono code, primary #1c75bc, navy #262262 sidebar text + right panel, #f9fafb background, 244px sidebar. sortRequiredPropsFirst on; 200 responses pre-expanded. Fonts ship as lambdas/api/fonts.css (latin woff2 subsets from @fontsource 5.3.0, embedded as data URIs, ~90KB) and inline via a new __FONTS_CSS__ placeholder with the same </style breakout guard -- the offline single-response invariant holds, nothing fetches Google Fonts (test-pinned). Bundling cp + bundle-consistency pin + spec-drift asset checks extended. Verified: headless-Chrome render (theme + fonts applied), ruff, 675 pytest, cdk synth + staged-asset check. * feat(api): SHOC gradient topbar on /docs 64px fixed header with the SHOC shell gradient token (#1b1f52 -> #1c4f8f -> #1c75bc), Sea Haven wordmark in Montserrat 600, page name right-aligned in DM Sans. Redoc's scrollYOffset: 64 keeps the sticky sidebar and anchor scrolling clear of the fixed bar. Verified via headless-Chrome render. * style(api): normalize /docs header and right-panel blues The right panel's #262262 is a purple-leaning navy that clashed with the cyan-leaning gradient, and the bar's brightest point sat directly over the dark panel. Right panel now uses #1b1f52 (the gradient's own dark endpoint) and the gradient runs bright-to-dark so its dark end lands flush on the panel -- no seam, one blue family. Verified via headless-Chrome render. * style(api): right-panel gradient on /docs via bundle-pinned override Redoc's theme only takes solid colors (it derives shades from rightPanel.backgroundColor), so the gradient (#1b3d79 -> #1b3068 -> #1b1f52, continuing the topbar blend) rides as a CSS override on the styled-components classes of the per-section right-panel divs (.sc-iGgWBj.sc-gsFSXq + the .sc-dExYaf stub). Those names are deterministic for the vendored 2.5.3 bundle (verified across loads) but change on any Redoc bump: re-derive via headless probe (find elements whose computed background equals the rightPanel color). If they stop matching, the panel falls back to the solid #1b1f52 theme color -- cosmetic only. Verified via headless-Chrome render. * feat(api): collapsible samples column on /docs Redoc CE has no built-in panel toggle, so the topbar gains a Hide/Show samples button that flips .samples-collapsed on <html>: the right-panel divs hide (same bundle-pinned styled-components classes as the gradient override) and each section's content half takes the full width. Choice persists in localStorage; aria-pressed tracks state. If the pinned classes stop matching after a Redoc bump the toggle goes inert -- cosmetic only. Both states verified via headless-Chrome render. * ci(api): spec-lint CI gate + npm Dependabot coverage New spec-lint job mirrors the local npm run lint:api so openapi.json cannot drift from redocly.yaml with green CI. Dependabot gains the npm ecosystem (package.json is new; nothing watched @redocly/cli). * feat(api): docs finishing touches - x-tagGroups, favicon, docs:preview x-tagGroups sections the Redoc sidebar (Read API / Meta / SHOC Feed); inline data-URI SVG favicon (SHOC blue) stops the browser's follow-up /favicon.ico request 403ing at the gateway; npm run docs:preview wraps the real-handler local render (scripts/preview_docs.py); README gains a docs-page architecture section covering the inline pattern, theme, pinned-selector caveat, and tooling. Lint 0 errors, 675 pytest, headless render verified. --------- Signed-off-by: Adam Moussa <adam@seahavenind.com>	2026-07-24 18:04:30 +00:00
Adam Moussa	24c89b43e5	feat(api): swap /docs from Swagger UI to Redoc (vendored offline) (#129) ... Redoc 2.5.3 standalone bundle (MIT) replaces the three swagger-ui-dist assets: one ~1.05MB JS file instead of ~1.8MB of JS+CSS+preset, and the layout traps (StandaloneLayout/BaseLayout) go away. Redoc is read-only by design, which matches the existing posture: try-it-out was already disabled since data routes need SigV4 (Postman for live calls). Unchanged: single token-gated response, offline vendoring (no CDN), per-request server-URL injection, script-breakout guards, cached shell with per-request spec splice. Bundle self-containment verified: the search worker is an inlined Blob, and the only new Worker(filename) path is Prism's async mode, which Redoc never invokes. Verified via headless-Chrome render of the real handler output: all routes, the OpenAPI 3.1 webhooks section, and planned-route markers render; no placeholder leakage.	2026-07-24 12:00:22 -04:00
Adam Moussa	8156b275a9	feat(api): stock Swagger UI for /docs (vendored offline) (#128) ... * feat(api): use stock Swagger UI for the /docs page Replaces the custom renderer with vendored stock Swagger UI (swagger-ui-dist 5.17.14, Apache-2.0), kept offline (no CDN) and inlined server-side into the single token-gated /docs response alongside the spec. BaseLayout (topbar hidden); try-it-out disabled since data routes need SigV4 (use Postman for live calls). Breakout guards on the inlined css/js/spec. Bundle-consistency + spec-drift + handler tests updated for the two vendored assets. cdk diff = Lambda code asset only (no IAM/API/policy change). * fix(api): render Swagger UI with the canonical StandaloneLayout recipe The BaseLayout-only init (apis preset, no standalone preset) rendered incorrectly. Switch to the canonical swagger-ui-dist recipe: vendor swagger-ui-standalone-preset.js and init with presets:[apis, SwaggerUIStandalonePreset] + layout:"StandaloneLayout" (topbar hidden, try-it-out disabled). Verified via headless Chrome against the live deployed page: all 9 endpoints + 4 webhooks + models render. Tests/bundling updated for the third vendored asset. * fix(api): declutter the Swagger UI docs page The page rendered (stock Swagger UI, StandaloneLayout) but looked cramped: a dense info.description wall of inline-code chips collided across Swagger UI's tight default line-height, and the Servers box showed a SEE-STACK-OUTPUT placeholder. - Trim info.description to a few concise lines (detail lives in README + the webhook contract doc). - Inject the live invoke URL into servers[0].url per request (from the API Gateway request context; committed literal is the fallback), so the Servers box shows the real endpoint and drops the server-variables section. - CSS: loosen description line-height + inline-code padding so chips never overlap; tidy the scheme container spacing. - Handler: cache the heavy CSS/JS/preset shell once; splice the (server- injected) spec per request. Verified via headless Chrome against the live deployed page.	2026-07-23 20:19:47 -04:00
Adam Moussa	f67d8b9907	feat(api): procurement-api read stack + OpenAPI docs (SHOC reconciliation path) (#127) ... * feat(api): add procurement-api stack - read API + OpenAPI docs page Third CDK stack: API Gateway REST API (IAM SigV4) over both pipelines' tables, replacing SHOC's retired SyncController cross-account DynamoDB scan as the reconciliation/backfill path. - lambdas/api/: handler (healthcheck + docs-token gate + router dispatch), router (single route table), pagination (opaque cursor, hostile -> 400), Decimal-safe serialization, wo_repo/po_repo reads. No VendorReplies. - OpenAPI 3.1 spec as source of truth incl. top-level webhooks section documenting the outbound SHOC feed; phase-2 write endpoints x-planned (router answers 501). Self-contained /docs page, no CDN. - Auth: AWS_IAM on data routes + resource policy scoped to exactly arn:aws:iam::396287094661:role/shoc-backend-dev on GET/*; /docs and /openapi.json carve-out is token-gated in the Lambda via shared web_ui_auth (fail-closed, INFRA-74 posture). - KMS: explicit Decrypt/DescribeKey on the DynamoDB CMK from SSM (name-imported table drops the key association - INFRA-104 class). - Alarms: errors/throttles/duration(p99>=22.5s) + gateway 5xx, ALARM-only to site-alerts. No access logging in v1 (docs ?token= shim stays out of logs); cloud_watch_role=False. - Tests: handler auth-seam + routing + Decimal round-trip; moto cursor pagination incl. hostile cursors; spec<->router drift gate; bundle AST pins for the api command; pytest.ini --cov + loader siblings. - Deploy role: third stack DescribeStacks ARN + procurement-api smoke invoke ARN (re-run create-deploy-role.sh before merge). * harden(api): apply sh-security-review findings to procurement-api Fan-out (6 detectors) + review findings resolved: Correctness / DoS: - pagination: require EXACT key-set match (was subset) so a partial/foreign composite cursor can't reach DynamoDB as an inconsistent ExclusiveStartKey -> ValidationException -> 500; comments Query now pins the cursor's work_order_id to the path entity. - handler: map botocore ValidationException to 400 (defense in depth) so a crafted cursor can't drive the zero-threshold 5xx alarm. - web_ui_auth: compare tokens as bytes; a non-ASCII presented token now fails closed (401) instead of crashing hmac.compare_digest into a 500. Resolves the pre-existing xfail(strict) follow-up test; hardens the web UIs too. Docs page: - typeStr() now escapes the one spec-derived string that reached innerHTML. - spec inlined into the docs <script> block escapes "<" -> < (</script> breakout guard); /openapi.json still served byte-faithful. - Cache-Control: no-store + Referrer-Policy: no-referrer on docs responses so the ?token= URL stays out of caches/Referer. - spec-drift test asserts the committed spec carries no "</" / "<!--". IAM / IaC: - resource policy enumerates the 7 data GET resources instead of GET/* so a future GET route can't silently inherit SHOC cross-account reach. - kms:Decrypt grant gains a kms:ViaService=dynamodb condition. - stage throttling (50 rps / 100 burst) bounds the unauthenticated /docs blast radius below the 10k account default. - corrected the PATCH/POST comment (same-account callers aren't blocked by the resource policy; 501 handler + absent write grant are the gate). - documented the RETAIN log-group first-deploy rollback trap and the resource-policy-needs-redeploy gotcha in-stack. Mandatory GPT-4.1 cross-family review of the full policy surface: no BLOCK/FIX. 675 tests pass, ruff clean, cdk synth green.	2026-07-23 19:32:20 -04:00
Adam Moussa	ca4f43a2cc	feat: decompose email-processor handlers into flat siblings + lazy boto3 clients (refactor phase 5) (#113) ... Both email-processor God-handlers split along the seams that already work in the flat-sibling pattern established by lambdas/shared/, so bare-name imports keep working under the existing bundling glob. PO (5-way split): handler.py keeps only the event loop, fail-closed auth, and email_type routing. extraction.py holds extract_with_claude and _EMAIL_TAG_RE, importing EXTRACTION_PROMPT from prompts.py and parse_raw_email from shared/email_parsing.py rather than recreating a PO-local copy. enrichment.py is a pure code move of enrich_parsed and pad_zip (PO-only; WO has no enrichment stage) with zero behavior change. telemetry.py holds the EMF ParseMethod emit wrappers. persistence.py holds _write_fields/_merge_update/save_*, collapsing the byte-identical save_new_po/save_revision bodies into one _save_merge helper that both now call through, preserving the sticky Cancelled ConditionExpression guard for both callers; save_cancellation stays separate. WO (5 concerns, no enrichment stage): the handler loop keeps validate_ai_fallback and the re.fullmatch(r"[0-9]+", work_order_id) key guard ahead of both save_work_order and save_event, since the guard protects the DynamoDB partition key and the '#'-delimited comment_id range-key segment. _header_date_iso and comment_id determinism stay colocated with persistence.py's save_event for the retry-idempotent event_id key. EXTRACTION_PROMPT (PO) moves to prompts.py with cross-reference headers to derived_fields.py's authoritative trade/site/fiscal rule tables; handler.py re-exports it (from prompts import EXTRACTION_PROMPT) since four tests dereference handler.EXTRACTION_ PROMPT directly. WO's prompt moves the same way. I/O modules (extraction.py's bedrock client, persistence.py's dynamodb resource, handler.py's s3 client) get lazy cached boto3 accessors; pure modules (enrichment.py, prompts.py, telemetry.py) import no boto3. Test monkeypatch surfaces move to the module that now owns the client (e.g. persistence.dynamodb) everywhere tests patch it, and the moto-before-handler-import ordering in _po_parser_support.py is preserved so the moto-backed suites don't hit real AWS. Behavior-preservation pins, verified with tests: PO still emits ParseMethod=ai_fallback before the Bedrock call, with ai_fallback_rejected as the additive second datapoint on rejection. WO still emits after its gate with mutually-exclusive ai_fallback / ai_fallback_rejected. Shadow DerivedFieldAgreement telemetry stays ai_fallback-only. derived_fields.py is untouched (diff against feature/phase-3-shared-extraction is empty). handler(event, context) signatures and the save_* public contract are unchanged on both pipelines; goldens unchanged. PO_EXPECTED_TOP_LEVEL_MODULES and its WO equivalent in tests/test_bundle_consistency.py are updated for the new sibling modules so the AST bundle-consistency test still fails on an unshipped or uncommented-out sibling.	2026-07-20 15:34:53 -04:00
Adam Moussa	30112cc680	feat: extract lambdas/shared/ — single-source ses_auth, web_ui auth, email parsing, EMF emitter (refactor phase 3) (#111) ... Four modules move into the handbook-mandated lambdas/shared/ location, collapsing duplicated logic that had to be kept in sync by hand across the PO and WO pipelines: - ses_auth.py: the PO and WO copies were verified sha256-identical against the feature/phase-7-ops-recovery baseline before the move (no drift since the last audit). shared/ses_auth.py is the exact bytes of that one copy; both originals are git rm'd (the PO copy via rename, the WO copy as a straight delete). Bundling lands the module flat in /asset-output for both email processors, so the handlers keep `from ses_auth import authenticate_inbound_email` unchanged — zero handler diff for this move, which is what keeps fail-closed auth byte-identical through the change. - web_ui_auth.py: extracts the byte-identical _get_auth_token / _header / is_authenticated block plus the four token-cache globals out of both web_ui handlers. The per-stack INFRA-74 comments stay in each handler as-is (deliberately drifted wording, stack-specific) rather than being unified into the shared module. Fail-closed semantics (unset ARN or Secrets Manager exception -> deny) are unchanged. - email_parsing.py: parse_raw_email ships as the superset version that returns cc unconditionally. WO's output is bit-identical to before; PO simply ignores the cc field rather than being "cleaned up" to consume it. No second variant is kept. - emf.py: a generic emitter parameterized by namespace, dimension sets, and properties. Every call site's emitted EMF envelope is unchanged, including the load-bearing [["ParseMethod"],["ParseMethod","TemplateId"]] dimension-set shape the alarms and metric filters depend on. Emission ordering is untouched: PO still emits ai_fallback before the Bedrock call, WO still emits its mutually-exclusive ai_fallback/ai_fallback_rejected after its gate. The deliberate-double-count comments survive. _emit_derived_agreement_metric was found living inside derived_fields.py, so per the DERIVED-FIELDS exception it is left as a third, unconverted copy (derived_fields.py and the shadow DerivedFieldAgreement telemetry stay untouchable while that bake runs) — a comment there points at shared/emf.py for the eventual follow-up. Bundling: both email-processor cdk bundling commands gain a trailing `cp shared/*.py /asset-output/` (they were already cp-only post-Phase 7, so no pip step or manylinux pin is reintroduced). Both web_ui functions gain the same widened-root staging so web_ui_auth.py ships beside their handler; site_extractor's from_asset is untouched. Tests: PO_EXPECTED_TOP_LEVEL_MODULES gains the shared modules that now ship, the AST sibling-import check resolves imports whose source now lives under shared/, and the new shared cp line has its own revert/mutation detection. _SIBLING_MODULES resolution and _po_parser_support.py now load ses_auth/email_parsing/emf from shared/; the two-copy ses_auth byte-identity fixture-hygiene test is retired as obsolete now that there is one copy, and the ses_auth fixture parameterization over two identical copies is dropped. The sys.modules save/restore dance for template_parser (still duplicated per-pipeline) is left in place.	2026-07-20 13:38:23 -04:00
Adam Moussa	df03f3497f	feat: widen email-processor asset roots to lambdas/ with scoped globs + excludes (refactor phase 2) (#109) ... Both email-processor Code.from_asset calls now bundle from lambdas/ instead of their per-function subdirectory, so Phase 3's shared/ module is reachable from the asset root once it lands. The bundling commands were rewritten for the new cwd (pip install -r <po|wo>/ email_processor/requirements.txt -t /asset-output && cp <po|wo>/ email_processor/*.py /asset-output/), preserving the ARM64 --platform manylinux2014_aarch64 --only-binary=:all: pin exactly — its removal shipped x86 wheels into the ARM64 function and caused a 100% outage (PR #34). All five from_asset calls (both email processors, po web_ui, po site_extractor, wo web_ui) now exclude **/__pycache__/**; the two widened ones also exclude **/tests/** and **/package/**. Without the package/ exclude, the stale untracked 44 MB lambdas/po/email_processor/package/ dir (local-only, never present in CI) would diverge local vs CI asset hashes and force spurious redeploys — from_asset doesn't honor .gitignore. That dir is left in place; deleting it is Adam's call. WO's prod zip shrinks as deliberate cleanup, not a byte-identical match to PO: the old `cp -r .` shipped tests/ (real scrubbed .eml fixtures), __pycache__/, and requirements.txt into production. The acceptance bar for WO is runtime-imported module set unchanged + smoke, not a byte-identical zip; PO keeps the byte-identical first-party file set guarantee. tests/test_bundle_consistency.py is updated in the same change to recognize the scoped `cp po/email_processor/*.py` (resp. wo) glob as the new unconditionally-safe shape, without loosening the allowlist-revert detection, the detection-logic mutation test, or the PO_EXPECTED_TOP_LEVEL_MODULES exact-set pin. No code moved under lambdas/ in this change (git diff main...HEAD -- lambdas/ is empty); only CDK asset wiring and its tests changed.	2026-07-17 15:47:01 -04:00
Adam Moussa	cb5539bd68	feat: deploy-pipeline guards — healthcheck, smoke gate, bundle glob + AST test (refactor phase 0) (#107) ... * feat: deploy-pipeline guards — healthcheck, smoke gate, bundle glob + AST test (refactor phase 0) Deploys of po-email-processor and workorder-email-processor had no verification step, so an init-time ImportError in the bundled zip could ship silently and only surface on the next real S3 event. This adds a synchronous post-deploy smoke gate wired into the deploy workflow: both Lambdas are invoked with {"healthcheck": true} and the FunctionError field is checked, since an Unhandled init error still returns HTTP 200 on RequestResponse invokes and would false-pass a plain exit-code check. The healthcheck branch is the first statement in each handler, before any boto3/S3 use or ses_auth, and only fires on a top-level direct invoke ("healthcheck" is not a key AWS ever sets on a real S3 ObjectCreated event, so mail content can't reach this path). It emits no EMF metrics and no log text that could match the sender-auth-rejected metric filter, so two deploys in one window won't trip the alarm. Separately, the PO stack's asset bundling copied a hand-maintained four-file allowlist into the zip, so every new sibling module handler.py imports had to be added by hand or the deploy shipped a Lambda that ImportErrors at cold start (bit us for template_parser in PR #105 and nearly for derived_fields in PR #2). Replaced it with a non-recursive ./*.py glob so top-level source files ship automatically while tests/ and the stale package/ dir still cannot, and added an AST-based bundle-consistency test that parses each handler's first-party imports and fails CI if the bundling command would omit any of them (a revert to an incomplete allowlist, or code moved into a subdirectory the glob doesn't cover). Includes the refactor-evaluation report that scoped this phase. * fix: review nits — unambiguous bundling-command extraction, smoke payload-parse message, dead asserts - tests/test_bundle_consistency.py: _extract_bundling_command now collects all command=[...] matches and demands exactly one per stack file, instead of silently returning whichever ast.walk visits first if a second bundled function is ever added. - scripts/post-deploy-smoke.sh: distinguish an unparseable response payload from a payload mismatch so the failure message says what actually happened (the previous "could not parse" branch was unreachable — the inline python always exited 0). - test_po_healthcheck.py: drop the substring assertions on stdout that were dead behind the stricter `captured.out == ""` assertion; keep the stderr filter-pattern check. Review follow-up on PR #107; no behavior change to any shipped code path.	2026-07-17 13:18:45 -04:00