mirror of
https://github.com/Sea-Haven-Industries/procurement-ingest.git
synced 2026-09-30 18:53:14 +00:00
cb5539bd68
Some checks are pending
Deploy / deploy (push) Waiting to run
* feat: deploy-pipeline guards — healthcheck, smoke gate, bundle glob + AST test (refactor phase 0)
Deploys of po-email-processor and workorder-email-processor had no
verification step, so an init-time ImportError in the bundled zip
could ship silently and only surface on the next real S3 event. This
adds a synchronous post-deploy smoke gate wired into the deploy
workflow: both Lambdas are invoked with {"healthcheck": true} and the
FunctionError field is checked, since an Unhandled init error still
returns HTTP 200 on RequestResponse invokes and would false-pass a
plain exit-code check.
The healthcheck branch is the first statement in each handler, before
any boto3/S3 use or ses_auth, and only fires on a top-level direct
invoke ("healthcheck" is not a key AWS ever sets on a real S3
ObjectCreated event, so mail content can't reach this path). It emits
no EMF metrics and no log text that could match the
sender-auth-rejected metric filter, so two deploys in one window
won't trip the alarm.
Separately, the PO stack's asset bundling copied a hand-maintained
four-file allowlist into the zip, so every new sibling module
handler.py imports had to be added by hand or the deploy shipped a
Lambda that ImportErrors at cold start (bit us for template_parser in
PR #105 and nearly for derived_fields in PR #2). Replaced it with a
non-recursive ./*.py glob so top-level source files ship
automatically while tests/ and the stale package/ dir still cannot,
and added an AST-based bundle-consistency test that parses each
handler's first-party imports and fails CI if the bundling command
would omit any of them (a revert to an incomplete allowlist, or code
moved into a subdirectory the glob doesn't cover).
Includes the refactor-evaluation report that scoped this phase.
* fix: review nits — unambiguous bundling-command extraction, smoke payload-parse message, dead asserts
- tests/test_bundle_consistency.py: _extract_bundling_command now collects
all command=[...] matches and demands exactly one per stack file, instead
of silently returning whichever ast.walk visits first if a second bundled
function is ever added.
- scripts/post-deploy-smoke.sh: distinguish an unparseable response payload
from a payload mismatch so the failure message says what actually happened
(the previous "could not parse" branch was unreachable — the inline python
always exited 0).
- test_po_healthcheck.py: drop the substring assertions on stdout that were
dead behind the stricter `captured.out == ""` assertion; keep the stderr
filter-pattern check.
Review follow-up on PR #107; no behavior change to any shipped code path.
95 lines
2.9 KiB
Bash
Executable file
95 lines
2.9 KiB
Bash
Executable file
#!/usr/bin/env bash
|
|
# Synchronous post-deploy smoke test for the PO and WO email-processor Lambdas.
|
|
#
|
|
# Invokes both functions with a {"healthcheck": true} payload using
|
|
# RequestResponse (synchronous) invocation and asserts:
|
|
# 1. The invoke response has no FunctionError field. A Lambda init failure
|
|
# (e.g. an ImportError from a broken bundle) still returns HTTP 200 from
|
|
# the Invoke API with FunctionError=Unhandled, so an exit-code-only check
|
|
# would false-pass -- this script inspects the response JSON explicitly.
|
|
# 2. The returned payload is exactly {"healthcheck": "ok"}.
|
|
#
|
|
# Exits non-zero on any failure (bad invoke, FunctionError set, wrong payload).
|
|
|
|
set -euo pipefail
|
|
|
|
REGION="us-east-1"
|
|
FUNCTION_NAMES=("po-email-processor" "workorder-email-processor")
|
|
HEALTHCHECK_PAYLOAD='{"healthcheck": true}'
|
|
|
|
work_dir="$(mktemp -d)"
|
|
trap 'rm -rf "${work_dir}"' EXIT
|
|
|
|
overall_status=0
|
|
|
|
for function_name in "${FUNCTION_NAMES[@]}"; do
|
|
echo "==> Smoke-testing ${function_name} (region ${REGION})"
|
|
|
|
response_meta_file="${work_dir}/${function_name}.meta.json"
|
|
response_payload_file="${work_dir}/${function_name}.payload.json"
|
|
|
|
if ! aws lambda invoke \
|
|
--function-name "${function_name}" \
|
|
--invocation-type RequestResponse \
|
|
--payload "${HEALTHCHECK_PAYLOAD}" \
|
|
--cli-binary-format raw-in-base64-out \
|
|
--region "${REGION}" \
|
|
"${response_payload_file}" \
|
|
>"${response_meta_file}"; then
|
|
echo "FAIL: ${function_name}: aws lambda invoke command failed" >&2
|
|
overall_status=1
|
|
continue
|
|
fi
|
|
|
|
function_error="$(python3 -c '
|
|
import json, sys
|
|
with open(sys.argv[1]) as f:
|
|
meta = json.load(f)
|
|
print(meta.get("FunctionError", ""))
|
|
' "${response_meta_file}")"
|
|
|
|
if [[ -n "${function_error}" ]]; then
|
|
echo "FAIL: ${function_name}: invoke returned FunctionError=${function_error}" >&2
|
|
echo "----- response payload -----" >&2
|
|
cat "${response_payload_file}" >&2
|
|
echo >&2
|
|
overall_status=1
|
|
continue
|
|
fi
|
|
|
|
payload_check="$(python3 -c '
|
|
import json, sys
|
|
try:
|
|
with open(sys.argv[1]) as f:
|
|
payload = json.load(f)
|
|
except (json.JSONDecodeError, UnicodeDecodeError):
|
|
print("unparseable")
|
|
sys.exit(0)
|
|
print("ok" if payload == {"healthcheck": "ok"} else "mismatch")
|
|
' "${response_payload_file}")"
|
|
|
|
if [[ "${payload_check}" == "unparseable" ]]; then
|
|
echo "FAIL: ${function_name}: response payload is not valid JSON:" >&2
|
|
cat "${response_payload_file}" >&2
|
|
echo >&2
|
|
overall_status=1
|
|
continue
|
|
fi
|
|
|
|
if [[ "${payload_check}" != "ok" ]]; then
|
|
echo "FAIL: ${function_name}: unexpected healthcheck payload:" >&2
|
|
cat "${response_payload_file}" >&2
|
|
echo >&2
|
|
overall_status=1
|
|
continue
|
|
fi
|
|
|
|
echo "OK: ${function_name} healthcheck passed"
|
|
done
|
|
|
|
if [[ "${overall_status}" -ne 0 ]]; then
|
|
echo "post-deploy smoke: one or more functions failed the healthcheck" >&2
|
|
exit 1
|
|
fi
|
|
|
|
echo "post-deploy smoke: all functions healthy"
|