procurement-ingest/lambdas/wo/shoc_hmac_rotator/handler.py
Adam Moussa c040050373
feat(webhook): SHOC WO webhook emitter - dark-ship streams + HMAC secret/rotation (PR-2) (#137)
* docs(webhook): revise SHOC webhook contract and plan for post-migration reality

Branch re-cut on main 2026-07-23 (old base carried stale PR #99 commits).

Contract Rev 2026-07-23:
- Producer account corrected: seahaven-prod (011934824531); mgmt frozen
- Reconciliation backstop is the new procurement read API, not SyncController
- wo_status "unknown" is real; SHOC must map it (checklist item added)
- write_origin forward-compat note for phase-2 write-back echo suppression
- SyncVendorReplies retirement flagged (dead table, no vendor_reply event)

Plan updates:
- Account gate: seahaven-prod only; never enable streams on mgmt tables
- Emitter ships DARK (ESMs enabled=False); activation is a deliberate flip
  after the SHOC receiver passes shared HMAC vectors
- Post-refactor conventions: common.py helpers, bundle-consistency AST pins,
  pytest.ini --cov additions, consolidated test roots
- Dedicated-CMK rationale, secret-ARN handooff step, consumer audit refreshed
  (slack-bot decommissioned), enum golden test, write_origin skip-branch test

* feat(webhook): SHOC WO webhook emitter — dark-ship streams, HMAC secret + rotation

Implements docs/shoc-webhook-plan.md Phases 1-5 (PR-2 of the SHOC
call-and-be-called effort). Everything ships DARK: both DynamoDB event
source mappings deploy enabled=False; activation is a deliberate
one-line follow-up PR gated on the SHOC receiver passing the shared
HMAC test vectors.

- Streams: NEW_AND_OLD_IMAGES on WorkOrders + WorkOrderComments
  (in-place update, RETAIN + logical IDs untouched; no existing
  consumers — verified live, neither table had a stream).
- workorder-shoc-emitter (Py3.12/ARM64): stream -> envelope ->
  HMAC-signed POST per docs/shoc-webhook-contract.md; strict per-shard
  ordering (parallelization 1, bisect off, retry until 24h age,
  ReportBatchItemFailures); 429/5xx/timeout block the shard in order,
  other 4xx park to workorder-shoc-emitter-rejected; ESM failures ->
  workorder-shoc-emitter-failures (metadata; replay rebuilds from
  DynamoDB). Echo guard skips write_origin=shoc-write-api.
- Secret workorder-ingest/shoc-webhook-hmac on a dedicated CMK
  (alias workorder-ingest-shoc-webhook-kms); cross-account
  GetSecretValue/DescribeSecret + kms:Decrypt granted to exactly
  arn:aws:iam::396287094661:role/shoc-backend-dev. RemovalPolicy
  DESTROY deliberately (machine-generated material; avoids the
  fixed-name RETAIN-orphan deadlock).
- workorder-shoc-hmac-rotator: 30-day rotation, dual-key overlap,
  64-hex keys, kid = UTC %Y-%m-%dT%H.
- Alarms (ALARM-only -> site-alerts): emitter errors/throttles/
  duration + iterator-age (>=10 min) + failures/rejected queue
  depth; rotator standard trio.
- scripts/replay_shoc_webhooks.py: dry-run-default operator replay
  (rebuilds from tables, replay:true envelopes).
- Tests: 742 passing, 85.56% aggregate; golden HMAC vectors shared
  with SHOC in docs/shoc-webhook-test-vectors.json (emitter + replay
  signing pinned to identical vectors); bundle-consistency AST pins
  for both new bundles.
- README: WO stack + webhook feed section, alarm table, runbooks;
  removed stale seahaven-slack-bot consumer references.

* fix(webhook): kms:ViaService pins, https-only delivery, cross-account principal CI pin

GPT-4.1 cross-family review of the policy surface (no BLOCK): FIX applied
to the cross-account shoc-backend-dev Decrypt statement and both Lambda
role KMS grants (the key is only ever used via Secrets Manager); its
invariant-enforcement QUESTION answered durably with
tests/test_cross_account_principal_pin.py (any new foreign IAM principal
in cdk/ fails CI). Scanner mediums fixed: delivery.py and the replay
script now refuse non-https URLs (urllib follows file:// and http://).
SQS metadata-action and dynamodb:ListStreams NITs skipped: standard CDK
grant shapes; ListStreams has no resource-level scoping. The 4 gitleaks
HIGHs on docs/shoc-webhook-test-vectors.json are deliberate non-secrets
(shared receiver-verification vectors) suppressed machine-level with
justification.

* harden(webhook): resolve /sh-security-review findings (1 confirmed medium + cheap fixes)

High-recall detector fan-out (injection/authz/secrets-crypto/iac-iam/logic)
+ proof-or-kill verifier. Gate PASSES: 1 confirmed medium, 0 confirmed
critical/high. Confirmed finding fixed; several unverified-but-cheap
hardenings applied since the emitter ships dark and activation is weeks out.

- CONFIRMED medium (confused deputy): the rotation Lambda's generated
  invoke permission for secretsmanager.amazonaws.com carried no
  SourceAccount/SourceArn, so any account's Secrets Manager could invoke
  the rotator. Patched the generated CfnPermission in place (a second
  permission would be additive, not restrictive) to pin account + this
  secret ARN.
- delivery + replay: refuse to follow receiver 3xx redirects (no-redirect
  opener) so live X-SH-* auth headers can't be forwarded to a
  receiver-chosen Location and an http:// Location can't slip past the
  https guard. Fixed the "unfollowed 3xx" comment that was factually wrong.
- delivery: classify 401/403 as retryable (invalidate key cache + retry in
  order) instead of parking -- transient auth failures (rotation outran the
  TTL cache, clock skew) are availability events, not contract bugs.
- envelope: build_event now genuinely total (guarded eventID /
  ApproximateCreationDateTime subscripts) per its own never-raise contract.
- handler: catch-all so an unexpected per-record error (e.g. SQS park
  failure) reports only that record instead of failing the whole batch
  (which would re-deliver every earlier success for 24h); per-invocation
  emit/skip batch summary so a systemic silent drop is queryable/alarmable.
- rotator: narrow the AWSCURRENT-read except to ResourceNotFound/JSONDecode
  (transient SM/KMS errors re-raise so the overlap key isn't silently
  dropped); kid uniqueness checked against ALL retained kids with a random
  suffix on collision (never reissue a kid for a different secret).
- contract: skeleton-upsert required on ANY unknown work_order_id (not just
  comment-before-create) + monotonicity guard (ignore older updated_at), so
  a parked created or an out-of-order replay can't corrupt receiver state.

Unverified/refuted findings left as-is with rationale: the two "high" logic
claims (whole-batch crash triggers, ordering violation) were refuted on
reachability (real stream records carry required fields; persistence writes
strings only; full-state idempotent upsert absorbs the ordering gap). Signed
kid/version binding (AUTHZ-002) declined: coordinated contract change, not
cheap, no exploit with one algorithm/key.

* fix(webhook): drop kid from rotator test_ok log (CodeQL clear-text-logging FP)

GHAS CodeQL flagged py/clear-text-logging-sensitive-data (high) at
_test_secret's success log because head["kid"] is subscripted from the
same parsed-secret dict that holds head["secret"] — the taint tracker
can't tell the non-secret key id from the secret. The secret value is
never logged. Rather than dismiss the alert (fragile; re-alerts on line
moves), remove the flow: kid is already logged at stage time in
_create_secret and version_id correlates the steps, so the test_ok log
keeps only event + version_id. Also hardens against a future edit that
swaps the logged field.
2026-07-24 22:12:20 +00:00

234 lines
9 KiB
Python

"""Rotation Lambda for the SHOC webhook HMAC secret.
Secrets Manager invokes this on the 30-day rotation schedule for
``workorder-ingest/shoc-webhook-hmac`` (docs/shoc-webhook-contract.md section
6.1). Secret value shape::
{"keys": [{"kid": "<YYYY-MM-DDTHH>", "secret": "<64 hex chars>"}, ...]}
newest first, truncated to 2 entries (one overlap cycle). The emitter always
signs with ``keys[0]``; the SHOC receiver accepts any listed ``kid``.
This is *single-user* rotation: no external system holds the value -- both
sides re-fetch from Secrets Manager on their own <=5-minute cache TTLs -- so
the standard 4-step protocol collapses to createSecret/finishSecret, with
setSecret a logged no-op and testSecret validating the pending value's shape.
The secret material is machine-generated HMAC key bytes and is NEVER logged;
structured logs carry only step names, version ids, and ``kid`` values.
"""
import json
import logging
import re
import secrets
from datetime import datetime, timezone
import boto3
logger = logging.getLogger()
logger.setLevel(logging.INFO)
# One overlap cycle: the new key plus the previous key (contract section 6.1).
MAX_KEYS = 2
# 32 random bytes -> secrets.token_hex emits 64 lowercase hex chars.
SECRET_NUM_BYTES = 32
SECRET_HEX_PATTERN = re.compile(r"[0-9a-f]{64}")
KID_FORMAT = "%Y-%m-%dT%H"
# Forced re-rotation within the same hour needs a distinct kid.
KID_FORMAT_INTRA_HOUR = "%Y-%m-%dT%H%M"
# Lazy cached Secrets Manager client. Keeps the public attribute name
# ``secretsmanager_client`` so the test monkeypatch target is stable.
secretsmanager_client = None
def _get_client():
global secretsmanager_client
if secretsmanager_client is None:
secretsmanager_client = boto3.client("secretsmanager")
return secretsmanager_client
def _current_keys(client, secret_id: str) -> list:
"""Read the AWSCURRENT ``keys`` list, tolerating malformed prior values.
The CDK-seeded bootstrap value is ``{"keys": [], "bootstrap_entropy": ...}``;
any unparseable or wrong-shape prior value likewise falls back to ``[]``
(rotation then simply starts a fresh key list -- the material is fully
regenerable, receivers refresh within their TTL).
"""
try:
raw = client.get_secret_value(SecretId=secret_id, VersionStage="AWSCURRENT")
parsed = json.loads(raw["SecretString"])
except (client.exceptions.ResourceNotFoundException, json.JSONDecodeError):
# Genuinely-absent or corrupt current value -> start a fresh list.
# NB: this does NOT catch transient failures (throttling, KMS/IAM
# blips): those re-raise so Secrets Manager marks the rotation failed
# and retries with the prior AWSCURRENT intact, rather than silently
# dropping the overlap key and stranding in-flight deliveries.
logger.warning(
json.dumps(
{"event": "hmac_rotation_current_unreadable", "fallback": "empty"}
)
)
return []
keys = parsed.get("keys") if isinstance(parsed, dict) else None
if not isinstance(keys, list) or not all(
isinstance(key, dict)
and isinstance(key.get("kid"), str)
and isinstance(key.get("secret"), str)
for key in keys
):
logger.warning(
json.dumps(
{"event": "hmac_rotation_current_malformed", "fallback": "empty"}
)
)
return []
return keys
def _create_secret(client, secret_id: str, token: str) -> None:
"""Stage a new AWSPENDING value: fresh key prepended, list truncated to 2."""
versions = client.describe_secret(SecretId=secret_id).get("VersionIdsToStages", {})
if "AWSCURRENT" in versions.get(token, []):
logger.info(
json.dumps(
{"event": "hmac_rotation_create_already_current", "version_id": token}
)
)
return
try:
client.get_secret_value(
SecretId=secret_id, VersionId=token, VersionStage="AWSPENDING"
)
# This token's version already holds a value: retried createSecret
# invocation, nothing to do.
logger.info(
json.dumps(
{"event": "hmac_rotation_create_idempotent", "version_id": token}
)
)
return
except client.exceptions.ResourceNotFoundException:
pass
current_keys = _current_keys(client, secret_id)
now = datetime.now(timezone.utc)
existing_kids = {key["kid"] for key in current_keys}
new_kid = now.strftime(KID_FORMAT)
if new_kid in existing_kids:
# Collision against ANY retained kid (not just keys[0]): repeated
# same-hour forced rotations must never reissue a kid for a different
# secret, or a receiver's cached kid->secret map desyncs. Append a
# random suffix guaranteed distinct from the retained set.
new_kid = f"{now.strftime(KID_FORMAT_INTRA_HOUR)}{secrets.token_hex(2)}"
new_key = {"kid": new_kid, "secret": secrets.token_hex(SECRET_NUM_BYTES)}
client.put_secret_value(
SecretId=secret_id,
ClientRequestToken=token,
SecretString=json.dumps({"keys": [new_key] + current_keys[: MAX_KEYS - 1]}),
VersionStages=["AWSPENDING"],
)
logger.info(
json.dumps(
{
"event": "hmac_rotation_pending_staged",
"version_id": token,
"kid": new_kid,
"key_count": 1 + len(current_keys[: MAX_KEYS - 1]),
}
)
)
def _set_secret(client, secret_id: str, token: str) -> None:
"""No-op: no external system holds the value (single-user rotation)."""
logger.info(
json.dumps({"event": "hmac_rotation_set_secret_noop", "version_id": token})
)
def _test_secret(client, secret_id: str, token: str) -> None:
"""Validate the AWSPENDING value's shape; raise ValueError to fail rotation."""
pending = client.get_secret_value(
SecretId=secret_id, VersionId=token, VersionStage="AWSPENDING"
)
# json.JSONDecodeError is a ValueError subclass: bad JSON fails the rotation.
parsed = json.loads(pending["SecretString"])
keys = parsed.get("keys") if isinstance(parsed, dict) else None
if not isinstance(keys, list) or not keys:
raise ValueError("AWSPENDING value has no keys list")
head = keys[0]
if not isinstance(head, dict) or not isinstance(head.get("kid"), str):
raise ValueError("AWSPENDING keys[0] has no string kid")
secret_value = head.get("secret")
if not isinstance(secret_value, str) or not SECRET_HEX_PATTERN.fullmatch(
secret_value
):
raise ValueError("AWSPENDING keys[0].secret is not 64 lowercase hex chars")
# Log only the rotation token, never a value pulled from the parsed
# secret dict. kid is non-sensitive (it rides X-SH-Key-Id in the clear)
# and is already logged at stage time in _create_secret, but subscripting
# the secret-bearing dict here trips CodeQL's clear-text-logging taint
# (py/clear-text-logging-sensitive-data) and is fragile if a later edit
# swaps the field -- version_id already correlates this step to the stage.
logger.info(json.dumps({"event": "hmac_rotation_test_ok", "version_id": token}))
def _finish_secret(client, secret_id: str, token: str) -> None:
"""Move AWSCURRENT to the pending version (idempotent on retry)."""
versions = client.describe_secret(SecretId=secret_id).get("VersionIdsToStages", {})
current_version_id = None
for version_id, stages in versions.items():
if "AWSCURRENT" in stages:
current_version_id = version_id
break
if current_version_id == token:
logger.info(
json.dumps(
{"event": "hmac_rotation_finish_idempotent", "version_id": token}
)
)
return
stage_move = {
"SecretId": secret_id,
"VersionStage": "AWSCURRENT",
"MoveToVersionId": token,
}
if current_version_id is not None:
stage_move["RemoveFromVersionId"] = current_version_id
client.update_secret_version_stage(**stage_move)
logger.info(
json.dumps(
{
"event": "hmac_rotation_finished",
"moved_to": token,
"removed_from": current_version_id,
}
)
)
def handler(event, context):
"""Secrets Manager rotation entry point.
``event`` = ``{"SecretId", "ClientRequestToken", "Step"}``; dispatches the
standard rotation steps. Unknown steps raise (rotation marked failed).
"""
secret_id = event["SecretId"]
token = event["ClientRequestToken"]
step = event["Step"]
logger.info(
json.dumps({"event": "hmac_rotation_step", "step": step, "version_id": token})
)
client = _get_client()
steps = {
"createSecret": _create_secret,
"setSecret": _set_secret,
"testSecret": _test_secret,
"finishSecret": _finish_secret,
}
if step not in steps:
raise ValueError(f"Unknown rotation step: {step}")
steps[step](client, secret_id, token)