procurement-ingest/lambdas/wo/shoc_emitter/envelope.py
Adam Moussa c040050373
feat(webhook): SHOC WO webhook emitter - dark-ship streams + HMAC secret/rotation (PR-2) (#137)
* docs(webhook): revise SHOC webhook contract and plan for post-migration reality

Branch re-cut on main 2026-07-23 (old base carried stale PR #99 commits).

Contract Rev 2026-07-23:
- Producer account corrected: seahaven-prod (011934824531); mgmt frozen
- Reconciliation backstop is the new procurement read API, not SyncController
- wo_status "unknown" is real; SHOC must map it (checklist item added)
- write_origin forward-compat note for phase-2 write-back echo suppression
- SyncVendorReplies retirement flagged (dead table, no vendor_reply event)

Plan updates:
- Account gate: seahaven-prod only; never enable streams on mgmt tables
- Emitter ships DARK (ESMs enabled=False); activation is a deliberate flip
  after the SHOC receiver passes shared HMAC vectors
- Post-refactor conventions: common.py helpers, bundle-consistency AST pins,
  pytest.ini --cov additions, consolidated test roots
- Dedicated-CMK rationale, secret-ARN handooff step, consumer audit refreshed
  (slack-bot decommissioned), enum golden test, write_origin skip-branch test

* feat(webhook): SHOC WO webhook emitter — dark-ship streams, HMAC secret + rotation

Implements docs/shoc-webhook-plan.md Phases 1-5 (PR-2 of the SHOC
call-and-be-called effort). Everything ships DARK: both DynamoDB event
source mappings deploy enabled=False; activation is a deliberate
one-line follow-up PR gated on the SHOC receiver passing the shared
HMAC test vectors.

- Streams: NEW_AND_OLD_IMAGES on WorkOrders + WorkOrderComments
  (in-place update, RETAIN + logical IDs untouched; no existing
  consumers — verified live, neither table had a stream).
- workorder-shoc-emitter (Py3.12/ARM64): stream -> envelope ->
  HMAC-signed POST per docs/shoc-webhook-contract.md; strict per-shard
  ordering (parallelization 1, bisect off, retry until 24h age,
  ReportBatchItemFailures); 429/5xx/timeout block the shard in order,
  other 4xx park to workorder-shoc-emitter-rejected; ESM failures ->
  workorder-shoc-emitter-failures (metadata; replay rebuilds from
  DynamoDB). Echo guard skips write_origin=shoc-write-api.
- Secret workorder-ingest/shoc-webhook-hmac on a dedicated CMK
  (alias workorder-ingest-shoc-webhook-kms); cross-account
  GetSecretValue/DescribeSecret + kms:Decrypt granted to exactly
  arn:aws:iam::396287094661:role/shoc-backend-dev. RemovalPolicy
  DESTROY deliberately (machine-generated material; avoids the
  fixed-name RETAIN-orphan deadlock).
- workorder-shoc-hmac-rotator: 30-day rotation, dual-key overlap,
  64-hex keys, kid = UTC %Y-%m-%dT%H.
- Alarms (ALARM-only -> site-alerts): emitter errors/throttles/
  duration + iterator-age (>=10 min) + failures/rejected queue
  depth; rotator standard trio.
- scripts/replay_shoc_webhooks.py: dry-run-default operator replay
  (rebuilds from tables, replay:true envelopes).
- Tests: 742 passing, 85.56% aggregate; golden HMAC vectors shared
  with SHOC in docs/shoc-webhook-test-vectors.json (emitter + replay
  signing pinned to identical vectors); bundle-consistency AST pins
  for both new bundles.
- README: WO stack + webhook feed section, alarm table, runbooks;
  removed stale seahaven-slack-bot consumer references.

* fix(webhook): kms:ViaService pins, https-only delivery, cross-account principal CI pin

GPT-4.1 cross-family review of the policy surface (no BLOCK): FIX applied
to the cross-account shoc-backend-dev Decrypt statement and both Lambda
role KMS grants (the key is only ever used via Secrets Manager); its
invariant-enforcement QUESTION answered durably with
tests/test_cross_account_principal_pin.py (any new foreign IAM principal
in cdk/ fails CI). Scanner mediums fixed: delivery.py and the replay
script now refuse non-https URLs (urllib follows file:// and http://).
SQS metadata-action and dynamodb:ListStreams NITs skipped: standard CDK
grant shapes; ListStreams has no resource-level scoping. The 4 gitleaks
HIGHs on docs/shoc-webhook-test-vectors.json are deliberate non-secrets
(shared receiver-verification vectors) suppressed machine-level with
justification.

* harden(webhook): resolve /sh-security-review findings (1 confirmed medium + cheap fixes)

High-recall detector fan-out (injection/authz/secrets-crypto/iac-iam/logic)
+ proof-or-kill verifier. Gate PASSES: 1 confirmed medium, 0 confirmed
critical/high. Confirmed finding fixed; several unverified-but-cheap
hardenings applied since the emitter ships dark and activation is weeks out.

- CONFIRMED medium (confused deputy): the rotation Lambda's generated
  invoke permission for secretsmanager.amazonaws.com carried no
  SourceAccount/SourceArn, so any account's Secrets Manager could invoke
  the rotator. Patched the generated CfnPermission in place (a second
  permission would be additive, not restrictive) to pin account + this
  secret ARN.
- delivery + replay: refuse to follow receiver 3xx redirects (no-redirect
  opener) so live X-SH-* auth headers can't be forwarded to a
  receiver-chosen Location and an http:// Location can't slip past the
  https guard. Fixed the "unfollowed 3xx" comment that was factually wrong.
- delivery: classify 401/403 as retryable (invalidate key cache + retry in
  order) instead of parking -- transient auth failures (rotation outran the
  TTL cache, clock skew) are availability events, not contract bugs.
- envelope: build_event now genuinely total (guarded eventID /
  ApproximateCreationDateTime subscripts) per its own never-raise contract.
- handler: catch-all so an unexpected per-record error (e.g. SQS park
  failure) reports only that record instead of failing the whole batch
  (which would re-deliver every earlier success for 24h); per-invocation
  emit/skip batch summary so a systemic silent drop is queryable/alarmable.
- rotator: narrow the AWSCURRENT-read except to ResourceNotFound/JSONDecode
  (transient SM/KMS errors re-raise so the overlap key isn't silently
  dropped); kid uniqueness checked against ALL retained kids with a random
  suffix on collision (never reissue a kid for a different secret).
- contract: skeleton-upsert required on ANY unknown work_order_id (not just
  comment-before-create) + monotonicity guard (ignore older updated_at), so
  a parked created or an out-of-order replay can't corrupt receiver state.

Unverified/refuted findings left as-is with rationale: the two "high" logic
claims (whole-batch crash triggers, ordering violation) were refuted on
reachability (real stream records carry required fields; persistence writes
strings only; full-state idempotent upsert absorbs the ordering gap). Signed
kid/version binding (AUTHZ-002) declined: coordinated contract change, not
cheap, no exploit with one algorithm/key.

* fix(webhook): drop kid from rotator test_ok log (CodeQL clear-text-logging FP)

GHAS CodeQL flagged py/clear-text-logging-sensitive-data (high) at
_test_secret's success log because head["kid"] is subscripted from the
same parsed-secret dict that holds head["secret"] — the taint tracker
can't tell the non-secret key id from the secret. The secret value is
never logged. Rather than dismiss the alert (fragile; re-alerts on line
moves), remove the flow: kid is already logged at stage time in
_create_secret and version_id correlates the steps, so the test_ok log
keeps only event + version_id. Also hardens against a future edit that
swaps the logged field.
2026-07-24 22:12:20 +00:00

163 lines
6.1 KiB
Python

"""Stream-record -> SHOC webhook envelope mapping (contract sections 3-4).
Pure and total: no boto3 clients, no environment reads, no network. Every
function either returns a value or returns None (skip) -- a malformed stream
record must never raise here; classification gaps fall through to None so the
shard is never blocked by an unmappable record. ``build_event`` is the single
entry point the handler calls per record.
Classification (docs/shoc-webhook-contract.md section 4):
WorkOrders INSERT -> work_order.created
WorkOrders MODIFY -> work_order.cancelled iff OLD wo_status was not
"cancelled" AND NEW wo_status is "cancelled",
else work_order.updated
WorkOrderComments INSERT -> work_order.comment_added
REMOVE (either table) -> skip (no deletes in this pipeline)
NEW write_origin == "shoc-write-api" -> skip (echo guard; a no-op branch
today -- nothing writes that attribute yet -- kept so the phase-2
write-back API never echoes SHOC's own writes back at it)
Anything else (unknown table, comment MODIFYs) -> skip.
"""
from datetime import datetime, timezone
from decimal import Decimal
from boto3.dynamodb.types import TypeDeserializer
SCHEMA_VERSION = 1
SOURCE = "procurement-ingest/workorder-shoc-emitter"
CANCELLED_STATUS = "cancelled"
SHOC_WRITE_ORIGIN = "shoc-write-api"
WORK_ORDERS_TABLE = "WorkOrders"
COMMENTS_TABLE = "WorkOrderComments"
# data payload field lists (contract sections 4.1 / 4.2). Absent attributes are
# null-filled; source_email_s3_key is deliberately EXCLUDED (internal key, not
# part of the contract).
WO_DATA_FIELDS = (
"work_order_id",
"wo_status",
"description",
"customer",
"site_code",
"building",
"address",
"severity",
"priority",
"assigned_to",
"date_reported",
"scheduled_start",
"due_date",
"record_type",
"created_at",
"updated_at",
)
COMMENT_DATA_FIELDS = (
"work_order_id",
"comment_id",
"record_type",
"commenter",
"text",
"created_at",
"ingested_at",
)
_deserializer = TypeDeserializer()
# arn:aws:dynamodb:...:table/NAME[/stream/LABEL] -- slash-split needs at least
# the ":table" head plus the name segment for the parse to be meaningful.
_MIN_ARN_SLASH_SEGMENTS = 2
def _plain(value):
"""Convert TypeDeserializer output to json.dumps-safe plain Python.
Mirrors lambdas/api/serialization.py: integral Decimals become JSON
integers (exact), non-integral Decimals become floats, sets become sorted
lists. Local copy on purpose -- the emitter bundles flat and must not grow
a cross-package import for twelve lines of logic.
"""
if isinstance(value, Decimal):
if value == value.to_integral_value():
return int(value)
return float(value)
if isinstance(value, dict):
return {key: _plain(item) for key, item in value.items()}
if isinstance(value, list):
return [_plain(item) for item in value]
if isinstance(value, set):
return sorted(_plain(item) for item in value)
return value
def _deserialize_image(image: dict) -> dict:
"""DynamoDB stream image (attribute-value encoded) -> plain dict."""
return {key: _plain(_deserializer.deserialize(av)) for key, av in image.items()}
def _table_name(event_source_arn: str) -> str | None:
"""Extract the exact table-name segment from a stream eventSourceARN.
ARN format: arn:aws:dynamodb:region:acct:table/NAME/stream/LABEL. Parsing
the segment exactly (not a loose substring) matters: "WorkOrders" is a
prefix of nothing, but a substring test for it WOULD match
"WorkOrderComments"-adjacent names -- segment equality can't.
"""
parts = event_source_arn.split("/")
if len(parts) >= _MIN_ARN_SLASH_SEGMENTS and parts[0].endswith(":table"):
return parts[1]
return None
def _classify(table, event_name, new_image, old_image) -> str | None:
if table == WORK_ORDERS_TABLE:
if event_name == "INSERT":
return "work_order.created"
if event_name == "MODIFY":
was_cancelled = old_image.get("wo_status") == CANCELLED_STATUS
if not was_cancelled and new_image.get("wo_status") == CANCELLED_STATUS:
return "work_order.cancelled"
return "work_order.updated"
return None
if table == COMMENTS_TABLE and event_name == "INSERT":
return "work_order.comment_added"
return None
def build_event(record: dict) -> dict | None:
"""Map one DynamoDB stream record to a webhook envelope, or None to skip."""
event_name = record.get("eventName")
if event_name == "REMOVE":
return None
stream = record.get("dynamodb") or {}
new_image = _deserialize_image(stream.get("NewImage") or {})
if new_image.get("write_origin") == SHOC_WRITE_ORIGIN:
return None # echo guard (forward-compat, contract section 3)
table = _table_name(record.get("eventSourceARN") or "")
old_image = _deserialize_image(stream.get("OldImage") or {})
event_type = _classify(table, event_name, new_image, old_image)
if event_type is None:
return None
if event_type == "work_order.comment_added":
fields = COMMENT_DATA_FIELDS
else:
fields = WO_DATA_FIELDS
# eventID and ApproximateCreationDateTime are present on every real
# DynamoDB stream record; guarding them keeps this function total (the
# "never raise" contract above) rather than trusting a hard subscript.
event_id = record.get("eventID")
approx_creation = stream.get("ApproximateCreationDateTime")
if event_id is None or approx_creation is None:
return None
# ApproximateCreationDateTime arrives as epoch seconds (float/Decimal).
occurred_at = datetime.fromtimestamp(
float(approx_creation), tz=timezone.utc
).isoformat()
return {
"schema_version": SCHEMA_VERSION,
"delivery_id": event_id,
"event_type": event_type,
"occurred_at": occurred_at,
"source": SOURCE,
"replay": False,
"data": {field: new_image.get(field) for field in fields},
}