procurement-ingest/tests/test_api_spec_drift.py

135 lines
5.1 KiB
Python
Raw Normal View History

feat(api): procurement-api read stack + OpenAPI docs (SHOC reconciliation path) (#127) * feat(api): add procurement-api stack - read API + OpenAPI docs page Third CDK stack: API Gateway REST API (IAM SigV4) over both pipelines' tables, replacing SHOC's retired SyncController cross-account DynamoDB scan as the reconciliation/backfill path. - lambdas/api/: handler (healthcheck + docs-token gate + router dispatch), router (single route table), pagination (opaque cursor, hostile -> 400), Decimal-safe serialization, wo_repo/po_repo reads. No VendorReplies. - OpenAPI 3.1 spec as source of truth incl. top-level webhooks section documenting the outbound SHOC feed; phase-2 write endpoints x-planned (router answers 501). Self-contained /docs page, no CDN. - Auth: AWS_IAM on data routes + resource policy scoped to exactly arn:aws:iam::396287094661:role/shoc-backend-dev on GET/*; /docs and /openapi.json carve-out is token-gated in the Lambda via shared web_ui_auth (fail-closed, INFRA-74 posture). - KMS: explicit Decrypt/DescribeKey on the DynamoDB CMK from SSM (name-imported table drops the key association - INFRA-104 class). - Alarms: errors/throttles/duration(p99>=22.5s) + gateway 5xx, ALARM-only to site-alerts. No access logging in v1 (docs ?token= shim stays out of logs); cloud_watch_role=False. - Tests: handler auth-seam + routing + Decimal round-trip; moto cursor pagination incl. hostile cursors; spec<->router drift gate; bundle AST pins for the api command; pytest.ini --cov + loader siblings. - Deploy role: third stack DescribeStacks ARN + procurement-api smoke invoke ARN (re-run create-deploy-role.sh before merge). * harden(api): apply sh-security-review findings to procurement-api Fan-out (6 detectors) + review findings resolved: Correctness / DoS: - pagination: require EXACT key-set match (was subset) so a partial/foreign composite cursor can't reach DynamoDB as an inconsistent ExclusiveStartKey -> ValidationException -> 500; comments Query now pins the cursor's work_order_id to the path entity. - handler: map botocore ValidationException to 400 (defense in depth) so a crafted cursor can't drive the zero-threshold 5xx alarm. - web_ui_auth: compare tokens as bytes; a non-ASCII presented token now fails closed (401) instead of crashing hmac.compare_digest into a 500. Resolves the pre-existing xfail(strict) follow-up test; hardens the web UIs too. Docs page: - typeStr() now escapes the one spec-derived string that reached innerHTML. - spec inlined into the docs <script> block escapes "<" -> < (</script> breakout guard); /openapi.json still served byte-faithful. - Cache-Control: no-store + Referrer-Policy: no-referrer on docs responses so the ?token= URL stays out of caches/Referer. - spec-drift test asserts the committed spec carries no "</" / "<!--". IAM / IaC: - resource policy enumerates the 7 data GET resources instead of GET/* so a future GET route can't silently inherit SHOC cross-account reach. - kms:Decrypt grant gains a kms:ViaService=dynamodb condition. - stage throttling (50 rps / 100 burst) bounds the unauthenticated /docs blast radius below the 10k account default. - corrected the PATCH/POST comment (same-account callers aren't blocked by the resource policy; 501 handler + absent write grant are the gate). - documented the RETAIN log-group first-deploy rollback trap and the resource-policy-needs-redeploy gotcha in-stack. Mandatory GPT-4.1 cross-family review of the full policy surface: no BLOCK/FIX. 675 tests pass, ruff clean, cdk synth green.
2026-07-23 19:32:20 -04:00
"""Spec <-> implementation drift gate.
lambdas/api/openapi.json is the published contract; lambdas/api/router.py is
what the Lambda actually serves. This test makes them the SAME set: an
endpoint added/removed/renamed on one side without the other fails CI here,
so the docs page can never silently lie. Also pins the phase-2 x-planned
markers, the four outbound webhook events, and the enum values the spec
promises against the WO extraction contract (prompts.py).
"""
import json
from pathlib import Path
from tests.support import REPO_ROOT, load_lambda_module
_SPEC_PATH = Path(REPO_ROOT) / "lambdas" / "api" / "openapi.json"
_HTTP_METHODS = {"get", "put", "post", "patch", "delete", "head", "options"}
def _spec():
return json.loads(_SPEC_PATH.read_text(encoding="utf-8"))
def _spec_routes(spec):
implemented, planned = set(), set()
for path, ops in spec["paths"].items():
for method, op in ops.items():
if method not in _HTTP_METHODS:
continue
key = (method.upper(), path)
if op.get("x-planned"):
planned.add(key)
else:
implemented.add(key)
return implemented, planned
def test_spec_is_openapi_31():
assert _spec()["openapi"] == "3.1.0"
def test_implemented_routes_match_router_exactly():
router = load_lambda_module("api", "router")
implemented, planned = _spec_routes(_spec())
assert implemented == set(router.DATA_ROUTES) | set(router.DOCS_ROUTES)
assert planned == set(router.PLANNED_ROUTES)
def test_every_data_route_has_a_repo_function():
router = load_lambda_module("api", "router")
handler = load_lambda_module("api", "handler")
assert set(router.DATA_ROUTES.values()) == set(handler._REPO_FUNCS)
def test_webhooks_section_documents_all_four_events():
spec = _spec()
assert set(spec["webhooks"]) == {
"work_order.created",
"work_order.updated",
"work_order.cancelled",
"work_order.comment_added",
}
for ops in spec["webhooks"].values():
assert "post" in ops
def test_spec_enums_match_wo_extraction_contract():
# The WO pipeline's prompts.py is the enum source of truth (the webhook
# contract pins it too). If the pipeline ever widens wo_status or
# record_type, the published spec must move in the same PR.
prompts = load_lambda_module("wo", "email_processor/prompts")
prompt_text = prompts.EXTRACTION_PROMPT
spec = _spec()
schemas = spec["components"]["schemas"]
wo_status_enum = {
value
for value in schemas["WorkOrder"]["properties"]["wo_status"]["enum"]
if value is not None
}
record_type_enum = {
value
for value in schemas["WorkOrder"]["properties"]["record_type"]["enum"]
if value is not None
}
for value in wo_status_enum - {"unknown"}:
assert value in prompt_text, f"wo_status {value!r} not in extraction prompt"
for value in record_type_enum:
assert value in prompt_text, f"record_type {value!r} not in extraction prompt"
event_data = schemas["WorkOrderEventData"]["properties"]
assert set(event_data["wo_status"]["enum"]) == set(
schemas["WorkOrder"]["properties"]["wo_status"]["enum"]
)
def test_spec_has_no_script_breakout_sequence():
# The spec is inlined into a <script type="application/json"> block on the
# docs page. The handler escapes "<" -> < defensively, but keep the
# committed spec itself clean so the raw /openapi.json is also breakout-safe
# and a reviewer sees the invariant here.
raw = _SPEC_PATH.read_text(encoding="utf-8")
assert "</" not in raw and "<!--" not in raw
def test_docs_and_spec_files_ship_with_the_handler():
feat(api): stock Swagger UI for /docs (vendored offline) (#128) * feat(api): use stock Swagger UI for the /docs page Replaces the custom renderer with vendored stock Swagger UI (swagger-ui-dist 5.17.14, Apache-2.0), kept offline (no CDN) and inlined server-side into the single token-gated /docs response alongside the spec. BaseLayout (topbar hidden); try-it-out disabled since data routes need SigV4 (use Postman for live calls). Breakout guards on the inlined css/js/spec. Bundle-consistency + spec-drift + handler tests updated for the two vendored assets. cdk diff = Lambda code asset only (no IAM/API/policy change). * fix(api): render Swagger UI with the canonical StandaloneLayout recipe The BaseLayout-only init (apis preset, no standalone preset) rendered incorrectly. Switch to the canonical swagger-ui-dist recipe: vendor swagger-ui-standalone-preset.js and init with presets:[apis, SwaggerUIStandalonePreset] + layout:"StandaloneLayout" (topbar hidden, try-it-out disabled). Verified via headless Chrome against the live deployed page: all 9 endpoints + 4 webhooks + models render. Tests/bundling updated for the third vendored asset. * fix(api): declutter the Swagger UI docs page The page rendered (stock Swagger UI, StandaloneLayout) but looked cramped: a dense info.description wall of inline-code chips collided across Swagger UI's tight default line-height, and the Servers box showed a SEE-STACK-OUTPUT placeholder. - Trim info.description to a few concise lines (detail lives in README + the webhook contract doc). - Inject the live invoke URL into servers[0].url per request (from the API Gateway request context; committed literal is the fallback), so the Servers box shows the real endpoint and drops the server-variables section. - CSS: loosen description line-height + inline-code padding so chips never overlap; tidy the scheme container spacing. - Handler: cache the heavy CSS/JS/preset shell once; splice the (server- injected) spec per request. Verified via headless Chrome against the live deployed page.
2026-07-23 20:19:47 -04:00
# The handler serves these from its own package dir; if any file is missing
# from lambdas/api/ the bundle would 500 at runtime. Swagger UI is vendored
# offline (no CDN) and inlined into the single token-gated response.
feat(api): procurement-api read stack + OpenAPI docs (SHOC reconciliation path) (#127) * feat(api): add procurement-api stack - read API + OpenAPI docs page Third CDK stack: API Gateway REST API (IAM SigV4) over both pipelines' tables, replacing SHOC's retired SyncController cross-account DynamoDB scan as the reconciliation/backfill path. - lambdas/api/: handler (healthcheck + docs-token gate + router dispatch), router (single route table), pagination (opaque cursor, hostile -> 400), Decimal-safe serialization, wo_repo/po_repo reads. No VendorReplies. - OpenAPI 3.1 spec as source of truth incl. top-level webhooks section documenting the outbound SHOC feed; phase-2 write endpoints x-planned (router answers 501). Self-contained /docs page, no CDN. - Auth: AWS_IAM on data routes + resource policy scoped to exactly arn:aws:iam::396287094661:role/shoc-backend-dev on GET/*; /docs and /openapi.json carve-out is token-gated in the Lambda via shared web_ui_auth (fail-closed, INFRA-74 posture). - KMS: explicit Decrypt/DescribeKey on the DynamoDB CMK from SSM (name-imported table drops the key association - INFRA-104 class). - Alarms: errors/throttles/duration(p99>=22.5s) + gateway 5xx, ALARM-only to site-alerts. No access logging in v1 (docs ?token= shim stays out of logs); cloud_watch_role=False. - Tests: handler auth-seam + routing + Decimal round-trip; moto cursor pagination incl. hostile cursors; spec<->router drift gate; bundle AST pins for the api command; pytest.ini --cov + loader siblings. - Deploy role: third stack DescribeStacks ARN + procurement-api smoke invoke ARN (re-run create-deploy-role.sh before merge). * harden(api): apply sh-security-review findings to procurement-api Fan-out (6 detectors) + review findings resolved: Correctness / DoS: - pagination: require EXACT key-set match (was subset) so a partial/foreign composite cursor can't reach DynamoDB as an inconsistent ExclusiveStartKey -> ValidationException -> 500; comments Query now pins the cursor's work_order_id to the path entity. - handler: map botocore ValidationException to 400 (defense in depth) so a crafted cursor can't drive the zero-threshold 5xx alarm. - web_ui_auth: compare tokens as bytes; a non-ASCII presented token now fails closed (401) instead of crashing hmac.compare_digest into a 500. Resolves the pre-existing xfail(strict) follow-up test; hardens the web UIs too. Docs page: - typeStr() now escapes the one spec-derived string that reached innerHTML. - spec inlined into the docs <script> block escapes "<" -> < (</script> breakout guard); /openapi.json still served byte-faithful. - Cache-Control: no-store + Referrer-Policy: no-referrer on docs responses so the ?token= URL stays out of caches/Referer. - spec-drift test asserts the committed spec carries no "</" / "<!--". IAM / IaC: - resource policy enumerates the 7 data GET resources instead of GET/* so a future GET route can't silently inherit SHOC cross-account reach. - kms:Decrypt grant gains a kms:ViaService=dynamodb condition. - stage throttling (50 rps / 100 burst) bounds the unauthenticated /docs blast radius below the 10k account default. - corrected the PATCH/POST comment (same-account callers aren't blocked by the resource policy; 501 handler + absent write grant are the gate). - documented the RETAIN log-group first-deploy rollback trap and the resource-policy-needs-redeploy gotcha in-stack. Mandatory GPT-4.1 cross-family review of the full policy surface: no BLOCK/FIX. 675 tests pass, ruff clean, cdk synth green.
2026-07-23 19:32:20 -04:00
api_dir = Path(REPO_ROOT) / "lambdas" / "api"
assert (api_dir / "openapi.json").is_file()
assert (api_dir / "docs.html").is_file()
feat(api): stock Swagger UI for /docs (vendored offline) (#128) * feat(api): use stock Swagger UI for the /docs page Replaces the custom renderer with vendored stock Swagger UI (swagger-ui-dist 5.17.14, Apache-2.0), kept offline (no CDN) and inlined server-side into the single token-gated /docs response alongside the spec. BaseLayout (topbar hidden); try-it-out disabled since data routes need SigV4 (use Postman for live calls). Breakout guards on the inlined css/js/spec. Bundle-consistency + spec-drift + handler tests updated for the two vendored assets. cdk diff = Lambda code asset only (no IAM/API/policy change). * fix(api): render Swagger UI with the canonical StandaloneLayout recipe The BaseLayout-only init (apis preset, no standalone preset) rendered incorrectly. Switch to the canonical swagger-ui-dist recipe: vendor swagger-ui-standalone-preset.js and init with presets:[apis, SwaggerUIStandalonePreset] + layout:"StandaloneLayout" (topbar hidden, try-it-out disabled). Verified via headless Chrome against the live deployed page: all 9 endpoints + 4 webhooks + models render. Tests/bundling updated for the third vendored asset. * fix(api): declutter the Swagger UI docs page The page rendered (stock Swagger UI, StandaloneLayout) but looked cramped: a dense info.description wall of inline-code chips collided across Swagger UI's tight default line-height, and the Servers box showed a SEE-STACK-OUTPUT placeholder. - Trim info.description to a few concise lines (detail lives in README + the webhook contract doc). - Inject the live invoke URL into servers[0].url per request (from the API Gateway request context; committed literal is the fallback), so the Servers box shows the real endpoint and drops the server-variables section. - CSS: loosen description line-height + inline-code padding so chips never overlap; tidy the scheme container spacing. - Handler: cache the heavy CSS/JS/preset shell once; splice the (server- injected) spec per request. Verified via headless Chrome against the live deployed page.
2026-07-23 20:19:47 -04:00
assert (api_dir / "swagger-ui-bundle.js").is_file()
assert (api_dir / "swagger-ui-standalone-preset.js").is_file()
assert (api_dir / "swagger-ui.css").is_file()
feat(api): procurement-api read stack + OpenAPI docs (SHOC reconciliation path) (#127) * feat(api): add procurement-api stack - read API + OpenAPI docs page Third CDK stack: API Gateway REST API (IAM SigV4) over both pipelines' tables, replacing SHOC's retired SyncController cross-account DynamoDB scan as the reconciliation/backfill path. - lambdas/api/: handler (healthcheck + docs-token gate + router dispatch), router (single route table), pagination (opaque cursor, hostile -> 400), Decimal-safe serialization, wo_repo/po_repo reads. No VendorReplies. - OpenAPI 3.1 spec as source of truth incl. top-level webhooks section documenting the outbound SHOC feed; phase-2 write endpoints x-planned (router answers 501). Self-contained /docs page, no CDN. - Auth: AWS_IAM on data routes + resource policy scoped to exactly arn:aws:iam::396287094661:role/shoc-backend-dev on GET/*; /docs and /openapi.json carve-out is token-gated in the Lambda via shared web_ui_auth (fail-closed, INFRA-74 posture). - KMS: explicit Decrypt/DescribeKey on the DynamoDB CMK from SSM (name-imported table drops the key association - INFRA-104 class). - Alarms: errors/throttles/duration(p99>=22.5s) + gateway 5xx, ALARM-only to site-alerts. No access logging in v1 (docs ?token= shim stays out of logs); cloud_watch_role=False. - Tests: handler auth-seam + routing + Decimal round-trip; moto cursor pagination incl. hostile cursors; spec<->router drift gate; bundle AST pins for the api command; pytest.ini --cov + loader siblings. - Deploy role: third stack DescribeStacks ARN + procurement-api smoke invoke ARN (re-run create-deploy-role.sh before merge). * harden(api): apply sh-security-review findings to procurement-api Fan-out (6 detectors) + review findings resolved: Correctness / DoS: - pagination: require EXACT key-set match (was subset) so a partial/foreign composite cursor can't reach DynamoDB as an inconsistent ExclusiveStartKey -> ValidationException -> 500; comments Query now pins the cursor's work_order_id to the path entity. - handler: map botocore ValidationException to 400 (defense in depth) so a crafted cursor can't drive the zero-threshold 5xx alarm. - web_ui_auth: compare tokens as bytes; a non-ASCII presented token now fails closed (401) instead of crashing hmac.compare_digest into a 500. Resolves the pre-existing xfail(strict) follow-up test; hardens the web UIs too. Docs page: - typeStr() now escapes the one spec-derived string that reached innerHTML. - spec inlined into the docs <script> block escapes "<" -> < (</script> breakout guard); /openapi.json still served byte-faithful. - Cache-Control: no-store + Referrer-Policy: no-referrer on docs responses so the ?token= URL stays out of caches/Referer. - spec-drift test asserts the committed spec carries no "</" / "<!--". IAM / IaC: - resource policy enumerates the 7 data GET resources instead of GET/* so a future GET route can't silently inherit SHOC cross-account reach. - kms:Decrypt grant gains a kms:ViaService=dynamodb condition. - stage throttling (50 rps / 100 burst) bounds the unauthenticated /docs blast radius below the 10k account default. - corrected the PATCH/POST comment (same-account callers aren't blocked by the resource policy; 501 handler + absent write grant are the gate). - documented the RETAIN log-group first-deploy rollback trap and the resource-policy-needs-redeploy gotcha in-stack. Mandatory GPT-4.1 cross-family review of the full policy surface: no BLOCK/FIX. 675 tests pass, ruff clean, cdk synth green.
2026-07-23 19:32:20 -04:00
docs = (api_dir / "docs.html").read_text(encoding="utf-8")
assert "__OPENAPI_SPEC_JSON__" in docs
feat(api): stock Swagger UI for /docs (vendored offline) (#128) * feat(api): use stock Swagger UI for the /docs page Replaces the custom renderer with vendored stock Swagger UI (swagger-ui-dist 5.17.14, Apache-2.0), kept offline (no CDN) and inlined server-side into the single token-gated /docs response alongside the spec. BaseLayout (topbar hidden); try-it-out disabled since data routes need SigV4 (use Postman for live calls). Breakout guards on the inlined css/js/spec. Bundle-consistency + spec-drift + handler tests updated for the two vendored assets. cdk diff = Lambda code asset only (no IAM/API/policy change). * fix(api): render Swagger UI with the canonical StandaloneLayout recipe The BaseLayout-only init (apis preset, no standalone preset) rendered incorrectly. Switch to the canonical swagger-ui-dist recipe: vendor swagger-ui-standalone-preset.js and init with presets:[apis, SwaggerUIStandalonePreset] + layout:"StandaloneLayout" (topbar hidden, try-it-out disabled). Verified via headless Chrome against the live deployed page: all 9 endpoints + 4 webhooks + models render. Tests/bundling updated for the third vendored asset. * fix(api): declutter the Swagger UI docs page The page rendered (stock Swagger UI, StandaloneLayout) but looked cramped: a dense info.description wall of inline-code chips collided across Swagger UI's tight default line-height, and the Servers box showed a SEE-STACK-OUTPUT placeholder. - Trim info.description to a few concise lines (detail lives in README + the webhook contract doc). - Inject the live invoke URL into servers[0].url per request (from the API Gateway request context; committed literal is the fallback), so the Servers box shows the real endpoint and drops the server-variables section. - CSS: loosen description line-height + inline-code padding so chips never overlap; tidy the scheme container spacing. - Handler: cache the heavy CSS/JS/preset shell once; splice the (server- injected) spec per request. Verified via headless Chrome against the live deployed page.
2026-07-23 20:19:47 -04:00
assert "__SWAGGER_UI_CSS__" in docs
assert "__SWAGGER_UI_JS__" in docs
assert "__SWAGGER_UI_STANDALONE_PRESET_JS__" in docs
assert "SwaggerUIStandalonePreset" in docs
assert "StandaloneLayout" in docs
# The vendored Swagger UI blob must carry no raw </script>: it is inlined
# into a <script> block, and a literal </script would break it out. (The
# handler also escapes it defensively, but keeping the pinned asset clean
# is the load-bearing guarantee and catches a bad version bump here.)
assert "</script" not in (api_dir / "swagger-ui-bundle.js").read_text(
encoding="utf-8"
)
assert "</script" not in (api_dir / "swagger-ui-standalone-preset.js").read_text(
encoding="utf-8"
)
assert "</style" not in (api_dir / "swagger-ui.css").read_text(encoding="utf-8")