procurement-ingest/scripts/post-deploy-smoke.sh

96 lines
2.9 KiB
Bash
Raw Permalink Normal View History

feat: deploy-pipeline guards — healthcheck, smoke gate, bundle glob + AST test (refactor phase 0) (#107) * feat: deploy-pipeline guards — healthcheck, smoke gate, bundle glob + AST test (refactor phase 0) Deploys of po-email-processor and workorder-email-processor had no verification step, so an init-time ImportError in the bundled zip could ship silently and only surface on the next real S3 event. This adds a synchronous post-deploy smoke gate wired into the deploy workflow: both Lambdas are invoked with {"healthcheck": true} and the FunctionError field is checked, since an Unhandled init error still returns HTTP 200 on RequestResponse invokes and would false-pass a plain exit-code check. The healthcheck branch is the first statement in each handler, before any boto3/S3 use or ses_auth, and only fires on a top-level direct invoke ("healthcheck" is not a key AWS ever sets on a real S3 ObjectCreated event, so mail content can't reach this path). It emits no EMF metrics and no log text that could match the sender-auth-rejected metric filter, so two deploys in one window won't trip the alarm. Separately, the PO stack's asset bundling copied a hand-maintained four-file allowlist into the zip, so every new sibling module handler.py imports had to be added by hand or the deploy shipped a Lambda that ImportErrors at cold start (bit us for template_parser in PR #105 and nearly for derived_fields in PR #2). Replaced it with a non-recursive ./*.py glob so top-level source files ship automatically while tests/ and the stale package/ dir still cannot, and added an AST-based bundle-consistency test that parses each handler's first-party imports and fails CI if the bundling command would omit any of them (a revert to an incomplete allowlist, or code moved into a subdirectory the glob doesn't cover). Includes the refactor-evaluation report that scoped this phase. * fix: review nits — unambiguous bundling-command extraction, smoke payload-parse message, dead asserts - tests/test_bundle_consistency.py: _extract_bundling_command now collects all command=[...] matches and demands exactly one per stack file, instead of silently returning whichever ast.walk visits first if a second bundled function is ever added. - scripts/post-deploy-smoke.sh: distinguish an unparseable response payload from a payload mismatch so the failure message says what actually happened (the previous "could not parse" branch was unreachable — the inline python always exited 0). - test_po_healthcheck.py: drop the substring assertions on stdout that were dead behind the stricter `captured.out == ""` assertion; keep the stderr filter-pattern check. Review follow-up on PR #107; no behavior change to any shipped code path.
2026-07-17 13:18:45 -04:00
#!/usr/bin/env bash
# Synchronous post-deploy smoke test for the PO and WO email-processor Lambdas.
#
# Invokes both functions with a {"healthcheck": true} payload using
# RequestResponse (synchronous) invocation and asserts:
# 1. The invoke response has no FunctionError field. A Lambda init failure
# (e.g. an ImportError from a broken bundle) still returns HTTP 200 from
# the Invoke API with FunctionError=Unhandled, so an exit-code-only check
# would false-pass -- this script inspects the response JSON explicitly.
# 2. The returned payload is exactly {"healthcheck": "ok"}.
#
# Exits non-zero on any failure (bad invoke, FunctionError set, wrong payload).
set -euo pipefail
REGION="us-east-1"
feat(api): procurement-api read stack + OpenAPI docs (SHOC reconciliation path) (#127) * feat(api): add procurement-api stack - read API + OpenAPI docs page Third CDK stack: API Gateway REST API (IAM SigV4) over both pipelines' tables, replacing SHOC's retired SyncController cross-account DynamoDB scan as the reconciliation/backfill path. - lambdas/api/: handler (healthcheck + docs-token gate + router dispatch), router (single route table), pagination (opaque cursor, hostile -> 400), Decimal-safe serialization, wo_repo/po_repo reads. No VendorReplies. - OpenAPI 3.1 spec as source of truth incl. top-level webhooks section documenting the outbound SHOC feed; phase-2 write endpoints x-planned (router answers 501). Self-contained /docs page, no CDN. - Auth: AWS_IAM on data routes + resource policy scoped to exactly arn:aws:iam::396287094661:role/shoc-backend-dev on GET/*; /docs and /openapi.json carve-out is token-gated in the Lambda via shared web_ui_auth (fail-closed, INFRA-74 posture). - KMS: explicit Decrypt/DescribeKey on the DynamoDB CMK from SSM (name-imported table drops the key association - INFRA-104 class). - Alarms: errors/throttles/duration(p99>=22.5s) + gateway 5xx, ALARM-only to site-alerts. No access logging in v1 (docs ?token= shim stays out of logs); cloud_watch_role=False. - Tests: handler auth-seam + routing + Decimal round-trip; moto cursor pagination incl. hostile cursors; spec<->router drift gate; bundle AST pins for the api command; pytest.ini --cov + loader siblings. - Deploy role: third stack DescribeStacks ARN + procurement-api smoke invoke ARN (re-run create-deploy-role.sh before merge). * harden(api): apply sh-security-review findings to procurement-api Fan-out (6 detectors) + review findings resolved: Correctness / DoS: - pagination: require EXACT key-set match (was subset) so a partial/foreign composite cursor can't reach DynamoDB as an inconsistent ExclusiveStartKey -> ValidationException -> 500; comments Query now pins the cursor's work_order_id to the path entity. - handler: map botocore ValidationException to 400 (defense in depth) so a crafted cursor can't drive the zero-threshold 5xx alarm. - web_ui_auth: compare tokens as bytes; a non-ASCII presented token now fails closed (401) instead of crashing hmac.compare_digest into a 500. Resolves the pre-existing xfail(strict) follow-up test; hardens the web UIs too. Docs page: - typeStr() now escapes the one spec-derived string that reached innerHTML. - spec inlined into the docs <script> block escapes "<" -> < (</script> breakout guard); /openapi.json still served byte-faithful. - Cache-Control: no-store + Referrer-Policy: no-referrer on docs responses so the ?token= URL stays out of caches/Referer. - spec-drift test asserts the committed spec carries no "</" / "<!--". IAM / IaC: - resource policy enumerates the 7 data GET resources instead of GET/* so a future GET route can't silently inherit SHOC cross-account reach. - kms:Decrypt grant gains a kms:ViaService=dynamodb condition. - stage throttling (50 rps / 100 burst) bounds the unauthenticated /docs blast radius below the 10k account default. - corrected the PATCH/POST comment (same-account callers aren't blocked by the resource policy; 501 handler + absent write grant are the gate). - documented the RETAIN log-group first-deploy rollback trap and the resource-policy-needs-redeploy gotcha in-stack. Mandatory GPT-4.1 cross-family review of the full policy surface: no BLOCK/FIX. 675 tests pass, ruff clean, cdk synth green.
2026-07-23 19:32:20 -04:00
FUNCTION_NAMES=("po-email-processor" "workorder-email-processor" "procurement-api")
feat: deploy-pipeline guards — healthcheck, smoke gate, bundle glob + AST test (refactor phase 0) (#107) * feat: deploy-pipeline guards — healthcheck, smoke gate, bundle glob + AST test (refactor phase 0) Deploys of po-email-processor and workorder-email-processor had no verification step, so an init-time ImportError in the bundled zip could ship silently and only surface on the next real S3 event. This adds a synchronous post-deploy smoke gate wired into the deploy workflow: both Lambdas are invoked with {"healthcheck": true} and the FunctionError field is checked, since an Unhandled init error still returns HTTP 200 on RequestResponse invokes and would false-pass a plain exit-code check. The healthcheck branch is the first statement in each handler, before any boto3/S3 use or ses_auth, and only fires on a top-level direct invoke ("healthcheck" is not a key AWS ever sets on a real S3 ObjectCreated event, so mail content can't reach this path). It emits no EMF metrics and no log text that could match the sender-auth-rejected metric filter, so two deploys in one window won't trip the alarm. Separately, the PO stack's asset bundling copied a hand-maintained four-file allowlist into the zip, so every new sibling module handler.py imports had to be added by hand or the deploy shipped a Lambda that ImportErrors at cold start (bit us for template_parser in PR #105 and nearly for derived_fields in PR #2). Replaced it with a non-recursive ./*.py glob so top-level source files ship automatically while tests/ and the stale package/ dir still cannot, and added an AST-based bundle-consistency test that parses each handler's first-party imports and fails CI if the bundling command would omit any of them (a revert to an incomplete allowlist, or code moved into a subdirectory the glob doesn't cover). Includes the refactor-evaluation report that scoped this phase. * fix: review nits — unambiguous bundling-command extraction, smoke payload-parse message, dead asserts - tests/test_bundle_consistency.py: _extract_bundling_command now collects all command=[...] matches and demands exactly one per stack file, instead of silently returning whichever ast.walk visits first if a second bundled function is ever added. - scripts/post-deploy-smoke.sh: distinguish an unparseable response payload from a payload mismatch so the failure message says what actually happened (the previous "could not parse" branch was unreachable — the inline python always exited 0). - test_po_healthcheck.py: drop the substring assertions on stdout that were dead behind the stricter `captured.out == ""` assertion; keep the stderr filter-pattern check. Review follow-up on PR #107; no behavior change to any shipped code path.
2026-07-17 13:18:45 -04:00
HEALTHCHECK_PAYLOAD='{"healthcheck": true}'
work_dir="$(mktemp -d)"
trap 'rm -rf "${work_dir}"' EXIT
overall_status=0
for function_name in "${FUNCTION_NAMES[@]}"; do
echo "==> Smoke-testing ${function_name} (region ${REGION})"
response_meta_file="${work_dir}/${function_name}.meta.json"
response_payload_file="${work_dir}/${function_name}.payload.json"
if ! aws lambda invoke \
--function-name "${function_name}" \
--invocation-type RequestResponse \
--payload "${HEALTHCHECK_PAYLOAD}" \
--cli-binary-format raw-in-base64-out \
--region "${REGION}" \
"${response_payload_file}" \
>"${response_meta_file}"; then
echo "FAIL: ${function_name}: aws lambda invoke command failed" >&2
overall_status=1
continue
fi
function_error="$(python3 -c '
import json, sys
with open(sys.argv[1]) as f:
meta = json.load(f)
print(meta.get("FunctionError", ""))
' "${response_meta_file}")"
if [[ -n "${function_error}" ]]; then
echo "FAIL: ${function_name}: invoke returned FunctionError=${function_error}" >&2
echo "----- response payload -----" >&2
cat "${response_payload_file}" >&2
echo >&2
overall_status=1
continue
fi
payload_check="$(python3 -c '
import json, sys
try:
with open(sys.argv[1]) as f:
payload = json.load(f)
except (json.JSONDecodeError, UnicodeDecodeError):
print("unparseable")
sys.exit(0)
print("ok" if payload == {"healthcheck": "ok"} else "mismatch")
' "${response_payload_file}")"
if [[ "${payload_check}" == "unparseable" ]]; then
echo "FAIL: ${function_name}: response payload is not valid JSON:" >&2
cat "${response_payload_file}" >&2
echo >&2
overall_status=1
continue
fi
if [[ "${payload_check}" != "ok" ]]; then
echo "FAIL: ${function_name}: unexpected healthcheck payload:" >&2
cat "${response_payload_file}" >&2
echo >&2
overall_status=1
continue
fi
echo "OK: ${function_name} healthcheck passed"
done
if [[ "${overall_status}" -ne 0 ]]; then
echo "post-deploy smoke: one or more functions failed the healthcheck" >&2
exit 1
fi
echo "post-deploy smoke: all functions healthy"