mirror of
https://github.com/Sea-Haven-Industries/procurement-ingest.git
synced 2026-09-30 21:13:15 +00:00
* docs(webhook): revise SHOC webhook contract and plan for post-migration reality Branch re-cut on main 2026-07-23 (old base carried stale PR #99 commits). Contract Rev 2026-07-23: - Producer account corrected: seahaven-prod (011934824531); mgmt frozen - Reconciliation backstop is the new procurement read API, not SyncController - wo_status "unknown" is real; SHOC must map it (checklist item added) - write_origin forward-compat note for phase-2 write-back echo suppression - SyncVendorReplies retirement flagged (dead table, no vendor_reply event) Plan updates: - Account gate: seahaven-prod only; never enable streams on mgmt tables - Emitter ships DARK (ESMs enabled=False); activation is a deliberate flip after the SHOC receiver passes shared HMAC vectors - Post-refactor conventions: common.py helpers, bundle-consistency AST pins, pytest.ini --cov additions, consolidated test roots - Dedicated-CMK rationale, secret-ARN handooff step, consumer audit refreshed (slack-bot decommissioned), enum golden test, write_origin skip-branch test * feat(webhook): SHOC WO webhook emitter — dark-ship streams, HMAC secret + rotation Implements docs/shoc-webhook-plan.md Phases 1-5 (PR-2 of the SHOC call-and-be-called effort). Everything ships DARK: both DynamoDB event source mappings deploy enabled=False; activation is a deliberate one-line follow-up PR gated on the SHOC receiver passing the shared HMAC test vectors. - Streams: NEW_AND_OLD_IMAGES on WorkOrders + WorkOrderComments (in-place update, RETAIN + logical IDs untouched; no existing consumers — verified live, neither table had a stream). - workorder-shoc-emitter (Py3.12/ARM64): stream -> envelope -> HMAC-signed POST per docs/shoc-webhook-contract.md; strict per-shard ordering (parallelization 1, bisect off, retry until 24h age, ReportBatchItemFailures); 429/5xx/timeout block the shard in order, other 4xx park to workorder-shoc-emitter-rejected; ESM failures -> workorder-shoc-emitter-failures (metadata; replay rebuilds from DynamoDB). Echo guard skips write_origin=shoc-write-api. - Secret workorder-ingest/shoc-webhook-hmac on a dedicated CMK (alias workorder-ingest-shoc-webhook-kms); cross-account GetSecretValue/DescribeSecret + kms:Decrypt granted to exactly arn:aws:iam::396287094661:role/shoc-backend-dev. RemovalPolicy DESTROY deliberately (machine-generated material; avoids the fixed-name RETAIN-orphan deadlock). - workorder-shoc-hmac-rotator: 30-day rotation, dual-key overlap, 64-hex keys, kid = UTC %Y-%m-%dT%H. - Alarms (ALARM-only -> site-alerts): emitter errors/throttles/ duration + iterator-age (>=10 min) + failures/rejected queue depth; rotator standard trio. - scripts/replay_shoc_webhooks.py: dry-run-default operator replay (rebuilds from tables, replay:true envelopes). - Tests: 742 passing, 85.56% aggregate; golden HMAC vectors shared with SHOC in docs/shoc-webhook-test-vectors.json (emitter + replay signing pinned to identical vectors); bundle-consistency AST pins for both new bundles. - README: WO stack + webhook feed section, alarm table, runbooks; removed stale seahaven-slack-bot consumer references. * fix(webhook): kms:ViaService pins, https-only delivery, cross-account principal CI pin GPT-4.1 cross-family review of the policy surface (no BLOCK): FIX applied to the cross-account shoc-backend-dev Decrypt statement and both Lambda role KMS grants (the key is only ever used via Secrets Manager); its invariant-enforcement QUESTION answered durably with tests/test_cross_account_principal_pin.py (any new foreign IAM principal in cdk/ fails CI). Scanner mediums fixed: delivery.py and the replay script now refuse non-https URLs (urllib follows file:// and http://). SQS metadata-action and dynamodb:ListStreams NITs skipped: standard CDK grant shapes; ListStreams has no resource-level scoping. The 4 gitleaks HIGHs on docs/shoc-webhook-test-vectors.json are deliberate non-secrets (shared receiver-verification vectors) suppressed machine-level with justification. * harden(webhook): resolve /sh-security-review findings (1 confirmed medium + cheap fixes) High-recall detector fan-out (injection/authz/secrets-crypto/iac-iam/logic) + proof-or-kill verifier. Gate PASSES: 1 confirmed medium, 0 confirmed critical/high. Confirmed finding fixed; several unverified-but-cheap hardenings applied since the emitter ships dark and activation is weeks out. - CONFIRMED medium (confused deputy): the rotation Lambda's generated invoke permission for secretsmanager.amazonaws.com carried no SourceAccount/SourceArn, so any account's Secrets Manager could invoke the rotator. Patched the generated CfnPermission in place (a second permission would be additive, not restrictive) to pin account + this secret ARN. - delivery + replay: refuse to follow receiver 3xx redirects (no-redirect opener) so live X-SH-* auth headers can't be forwarded to a receiver-chosen Location and an http:// Location can't slip past the https guard. Fixed the "unfollowed 3xx" comment that was factually wrong. - delivery: classify 401/403 as retryable (invalidate key cache + retry in order) instead of parking -- transient auth failures (rotation outran the TTL cache, clock skew) are availability events, not contract bugs. - envelope: build_event now genuinely total (guarded eventID / ApproximateCreationDateTime subscripts) per its own never-raise contract. - handler: catch-all so an unexpected per-record error (e.g. SQS park failure) reports only that record instead of failing the whole batch (which would re-deliver every earlier success for 24h); per-invocation emit/skip batch summary so a systemic silent drop is queryable/alarmable. - rotator: narrow the AWSCURRENT-read except to ResourceNotFound/JSONDecode (transient SM/KMS errors re-raise so the overlap key isn't silently dropped); kid uniqueness checked against ALL retained kids with a random suffix on collision (never reissue a kid for a different secret). - contract: skeleton-upsert required on ANY unknown work_order_id (not just comment-before-create) + monotonicity guard (ignore older updated_at), so a parked created or an out-of-order replay can't corrupt receiver state. Unverified/refuted findings left as-is with rationale: the two "high" logic claims (whole-batch crash triggers, ordering violation) were refuted on reachability (real stream records carry required fields; persistence writes strings only; full-state idempotent upsert absorbs the ordering gap). Signed kid/version binding (AUTHZ-002) declined: coordinated contract change, not cheap, no exploit with one algorithm/key. * fix(webhook): drop kid from rotator test_ok log (CodeQL clear-text-logging FP) GHAS CodeQL flagged py/clear-text-logging-sensitive-data (high) at _test_secret's success log because head["kid"] is subscripted from the same parsed-secret dict that holds head["secret"] — the taint tracker can't tell the non-secret key id from the secret. The secret value is never logged. Rather than dismiss the alert (fragile; re-alerts on line moves), remove the flow: kid is already logged at stage time in _create_secret and version_id correlates the steps, so the test_ok log keeps only event + version_id. Also hardens against a future edit that swaps the logged field.
351 lines
12 KiB
Python
351 lines
12 KiB
Python
"""Rotation-step tests for workorder-shoc-hmac-rotator (plan Phase 5).
|
|
|
|
Exercises the Secrets Manager rotation protocol against a dict-backed fake
|
|
client (no moto, no AWS): createSecret from the CDK bootstrap value and from
|
|
a populated list (newest-first, truncated to 2), createSecret idempotency on
|
|
retried tokens, the intra-hour kid-collision fallback, testSecret's
|
|
shape/hex validation, finishSecret's stage move + idempotency, the setSecret
|
|
no-op, and the unknown-step failure.
|
|
|
|
datetime is monkeypatched to a fixed instant so kid assertions are exact and
|
|
the collision case is deterministic (no wall-clock hour-boundary flake).
|
|
"""
|
|
|
|
import json
|
|
import re
|
|
from datetime import datetime, timezone
|
|
|
|
import pytest
|
|
|
|
from tests.support import load_lambda_module
|
|
|
|
SECRET_ID = "workorder-ingest/shoc-webhook-hmac"
|
|
HEX_64_RE = re.compile(r"^[0-9a-f]{64}$")
|
|
|
|
# Fixed rotation instant: kid "2026-07-24T15" (intra-hour fallback
|
|
# "2026-07-24T1507").
|
|
FIXED_NOW = datetime(2026, 7, 24, 15, 7, 42, tzinfo=timezone.utc)
|
|
|
|
KEY_A = {"kid": "2026-06-20T00", "secret": "a" * 64}
|
|
KEY_B = {"kid": "2026-05-21T00", "secret": "b" * 64}
|
|
|
|
|
|
class _ResourceNotFound(Exception):
|
|
pass
|
|
|
|
|
|
class _Exceptions:
|
|
ResourceNotFoundException = _ResourceNotFound
|
|
|
|
|
|
class FakeSecretsManager:
|
|
"""Dict-backed Secrets Manager stub recording every mutating call.
|
|
|
|
``versions`` maps version_id -> {"stages": set[str], "value": str}.
|
|
"""
|
|
|
|
exceptions = _Exceptions
|
|
|
|
def __init__(self, versions=None):
|
|
self.versions = {
|
|
vid: {"stages": set(v["stages"]), "value": v["value"]}
|
|
for vid, v in (versions or {}).items()
|
|
}
|
|
self.put_calls = []
|
|
self.stage_calls = []
|
|
|
|
def describe_secret(self, SecretId): # noqa: N803 (boto3 kwarg names)
|
|
return {
|
|
"VersionIdsToStages": {
|
|
vid: sorted(v["stages"]) for vid, v in self.versions.items()
|
|
}
|
|
}
|
|
|
|
def get_secret_value( # noqa: N803 (boto3 kwarg names)
|
|
self, SecretId, VersionId=None, VersionStage=None
|
|
):
|
|
if VersionId is not None:
|
|
version = self.versions.get(VersionId)
|
|
if version is None or (
|
|
VersionStage is not None and VersionStage not in version["stages"]
|
|
):
|
|
raise _ResourceNotFound(f"{VersionId} / {VersionStage}")
|
|
return {"SecretString": version["value"]}
|
|
stage = VersionStage or "AWSCURRENT"
|
|
for version in self.versions.values():
|
|
if stage in version["stages"]:
|
|
return {"SecretString": version["value"]}
|
|
raise _ResourceNotFound(stage)
|
|
|
|
def put_secret_value( # noqa: N803 (boto3 kwarg names)
|
|
self, SecretId, ClientRequestToken, SecretString, VersionStages
|
|
):
|
|
self.put_calls.append(
|
|
{
|
|
"SecretId": SecretId,
|
|
"ClientRequestToken": ClientRequestToken,
|
|
"SecretString": SecretString,
|
|
"VersionStages": list(VersionStages),
|
|
}
|
|
)
|
|
self.versions[ClientRequestToken] = {
|
|
"stages": set(VersionStages),
|
|
"value": SecretString,
|
|
}
|
|
|
|
def update_secret_version_stage(self, **kwargs):
|
|
self.stage_calls.append(kwargs)
|
|
stage = kwargs["VersionStage"]
|
|
removed = kwargs.get("RemoveFromVersionId")
|
|
if removed is not None:
|
|
self.versions[removed]["stages"].discard(stage)
|
|
self.versions[kwargs["MoveToVersionId"]]["stages"].add(stage)
|
|
|
|
|
|
class _FixedDatetime:
|
|
@classmethod
|
|
def now(cls, tz=None):
|
|
return FIXED_NOW
|
|
|
|
|
|
@pytest.fixture(scope="module")
|
|
def rotator():
|
|
return load_lambda_module("wo", "shoc_hmac_rotator/handler")
|
|
|
|
|
|
@pytest.fixture
|
|
def fixed_now(monkeypatch, rotator):
|
|
monkeypatch.setattr(rotator, "datetime", _FixedDatetime)
|
|
|
|
|
|
def _run(monkeypatch, rotator, fake, step, token="tok-1"):
|
|
monkeypatch.setattr(rotator, "secretsmanager_client", fake)
|
|
rotator.handler(
|
|
{"SecretId": SECRET_ID, "ClientRequestToken": token, "Step": step}, None
|
|
)
|
|
|
|
|
|
def _bootstrap_fake(value=None):
|
|
current = value if value is not None else {"keys": [], "bootstrap_entropy": "x"}
|
|
return FakeSecretsManager(
|
|
{"cur-1": {"stages": {"AWSCURRENT"}, "value": json.dumps(current)}}
|
|
)
|
|
|
|
|
|
# --- createSecret ------------------------------------------------------------
|
|
|
|
|
|
def test_create_from_bootstrap_stages_one_fresh_key(monkeypatch, rotator, fixed_now):
|
|
fake = _bootstrap_fake()
|
|
_run(monkeypatch, rotator, fake, "createSecret")
|
|
|
|
assert len(fake.put_calls) == 1
|
|
put = fake.put_calls[0]
|
|
assert put["SecretId"] == SECRET_ID
|
|
assert put["ClientRequestToken"] == "tok-1"
|
|
assert put["VersionStages"] == ["AWSPENDING"]
|
|
staged = json.loads(put["SecretString"])
|
|
assert list(staged) == ["keys"]
|
|
assert len(staged["keys"]) == 1
|
|
assert staged["keys"][0]["kid"] == "2026-07-24T15"
|
|
assert HEX_64_RE.fullmatch(staged["keys"][0]["secret"])
|
|
|
|
|
|
def test_create_prepends_and_truncates_to_two_newest_first(
|
|
monkeypatch, rotator, fixed_now
|
|
):
|
|
fake = _bootstrap_fake({"keys": [KEY_A, KEY_B]})
|
|
_run(monkeypatch, rotator, fake, "createSecret")
|
|
|
|
staged = json.loads(fake.put_calls[0]["SecretString"])
|
|
assert len(staged["keys"]) == 2
|
|
# Newest first: the fresh key leads, the previous head is retained for
|
|
# one overlap cycle, the oldest key falls off.
|
|
assert staged["keys"][0]["kid"] == "2026-07-24T15"
|
|
assert staged["keys"][1] == KEY_A
|
|
assert KEY_B["secret"] not in fake.put_calls[0]["SecretString"]
|
|
|
|
|
|
def test_create_is_idempotent_when_pending_already_staged(
|
|
monkeypatch, rotator, fixed_now
|
|
):
|
|
fake = _bootstrap_fake({"keys": [KEY_A]})
|
|
fake.versions["tok-1"] = {
|
|
"stages": {"AWSPENDING"},
|
|
"value": json.dumps({"keys": [KEY_A]}),
|
|
}
|
|
_run(monkeypatch, rotator, fake, "createSecret")
|
|
assert fake.put_calls == []
|
|
|
|
|
|
def test_create_is_noop_when_token_already_current(monkeypatch, rotator, fixed_now):
|
|
fake = FakeSecretsManager(
|
|
{
|
|
"tok-1": {
|
|
"stages": {"AWSCURRENT"},
|
|
"value": json.dumps({"keys": [KEY_A]}),
|
|
}
|
|
}
|
|
)
|
|
_run(monkeypatch, rotator, fake, "createSecret")
|
|
assert fake.put_calls == []
|
|
|
|
|
|
def test_create_kid_collision_same_hour_gets_distinct_suffix(
|
|
monkeypatch, rotator, fixed_now
|
|
):
|
|
# Forced re-rotation within the same hour: the hour-format kid already
|
|
# exists, so the new kid extends to minutes + a random suffix, guaranteed
|
|
# distinct from every retained kid (never reissue a kid for a new secret).
|
|
fake = _bootstrap_fake({"keys": [{"kid": "2026-07-24T15", "secret": "c" * 64}]})
|
|
_run(monkeypatch, rotator, fake, "createSecret")
|
|
|
|
staged = json.loads(fake.put_calls[0]["SecretString"])
|
|
new_kid = staged["keys"][0]["kid"]
|
|
assert new_kid.startswith("2026-07-24T1507")
|
|
assert new_kid != "2026-07-24T15"
|
|
assert staged["keys"][1]["kid"] == "2026-07-24T15"
|
|
|
|
|
|
def test_create_kid_collision_against_second_key_also_avoided(
|
|
monkeypatch, rotator, fixed_now
|
|
):
|
|
# Uniqueness is checked against ALL retained kids, not just keys[0]: if the
|
|
# hour-format kid matches keys[1] (a third same-hour rotation), it still
|
|
# gets a distinct suffix rather than being reissued.
|
|
fake = _bootstrap_fake(
|
|
{
|
|
"keys": [
|
|
{"kid": "2026-07-24T1500ab", "secret": "a" * 64},
|
|
{"kid": "2026-07-24T15", "secret": "b" * 64},
|
|
]
|
|
}
|
|
)
|
|
_run(monkeypatch, rotator, fake, "createSecret")
|
|
new_kid = json.loads(fake.put_calls[0]["SecretString"])["keys"][0]["kid"]
|
|
assert new_kid not in {"2026-07-24T15", "2026-07-24T1500ab"}
|
|
|
|
|
|
def test_create_malformed_current_value_starts_fresh_list(
|
|
monkeypatch, rotator, fixed_now
|
|
):
|
|
fake = FakeSecretsManager(
|
|
{"cur-1": {"stages": {"AWSCURRENT"}, "value": "not json {"}}
|
|
)
|
|
_run(monkeypatch, rotator, fake, "createSecret")
|
|
staged = json.loads(fake.put_calls[0]["SecretString"])
|
|
assert len(staged["keys"]) == 1
|
|
assert HEX_64_RE.fullmatch(staged["keys"][0]["secret"])
|
|
|
|
|
|
def test_create_transient_current_read_error_propagates_not_swallowed(
|
|
monkeypatch, rotator, fixed_now
|
|
):
|
|
# A transient AWSCURRENT read failure (throttle/KMS blip) must NOT be
|
|
# swallowed into an empty key list -- that would drop the overlap key and
|
|
# strand in-flight deliveries. It re-raises so Secrets Manager fails and
|
|
# retries the rotation with the prior AWSCURRENT intact.
|
|
class _ThrottlingFake(FakeSecretsManager):
|
|
def get_secret_value(self, SecretId, VersionId=None, VersionStage=None): # noqa: N803
|
|
if VersionStage == "AWSCURRENT":
|
|
raise RuntimeError("ThrottlingException")
|
|
return super().get_secret_value(
|
|
SecretId, VersionId=VersionId, VersionStage=VersionStage
|
|
)
|
|
|
|
fake = _ThrottlingFake(
|
|
{"cur-1": {"stages": {"AWSCURRENT"}, "value": json.dumps({"keys": []})}}
|
|
)
|
|
with pytest.raises(RuntimeError, match="ThrottlingException"):
|
|
_run(monkeypatch, rotator, fake, "createSecret")
|
|
assert fake.put_calls == []
|
|
|
|
|
|
# --- testSecret --------------------------------------------------------------
|
|
|
|
|
|
def _pending_fake(value):
|
|
return FakeSecretsManager(
|
|
{"tok-1": {"stages": {"AWSPENDING"}, "value": json.dumps(value)}}
|
|
)
|
|
|
|
|
|
def test_test_secret_accepts_valid_pending_value(monkeypatch, rotator):
|
|
fake = _pending_fake({"keys": [{"kid": "2026-07-24T15", "secret": "d" * 64}]})
|
|
_run(monkeypatch, rotator, fake, "testSecret") # no raise
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"pending_value",
|
|
[
|
|
{"keys": []},
|
|
{"keys": [{"kid": "2026-07-24T15", "secret": "d" * 63}]},
|
|
{"keys": [{"kid": "2026-07-24T15", "secret": "z" * 64}]},
|
|
{"keys": [{"kid": "2026-07-24T15", "secret": "D" * 64}]},
|
|
{"keys": [{"secret": "d" * 64}]},
|
|
],
|
|
ids=["empty-keys", "short-secret", "non-hex", "uppercase-hex", "missing-kid"],
|
|
)
|
|
def test_test_secret_rejects_bad_pending_values(monkeypatch, rotator, pending_value):
|
|
fake = _pending_fake(pending_value)
|
|
with pytest.raises(ValueError):
|
|
_run(monkeypatch, rotator, fake, "testSecret")
|
|
|
|
|
|
# --- finishSecret ------------------------------------------------------------
|
|
|
|
|
|
def test_finish_moves_current_stage_from_old_version(monkeypatch, rotator):
|
|
fake = FakeSecretsManager(
|
|
{
|
|
"old-1": {
|
|
"stages": {"AWSCURRENT"},
|
|
"value": json.dumps({"keys": [KEY_A]}),
|
|
},
|
|
"tok-1": {
|
|
"stages": {"AWSPENDING"},
|
|
"value": json.dumps({"keys": [KEY_B, KEY_A]}),
|
|
},
|
|
}
|
|
)
|
|
_run(monkeypatch, rotator, fake, "finishSecret")
|
|
|
|
assert fake.stage_calls == [
|
|
{
|
|
"SecretId": SECRET_ID,
|
|
"VersionStage": "AWSCURRENT",
|
|
"MoveToVersionId": "tok-1",
|
|
"RemoveFromVersionId": "old-1",
|
|
}
|
|
]
|
|
assert "AWSCURRENT" in fake.versions["tok-1"]["stages"]
|
|
assert "AWSCURRENT" not in fake.versions["old-1"]["stages"]
|
|
|
|
|
|
def test_finish_is_idempotent_when_token_already_current(monkeypatch, rotator):
|
|
fake = FakeSecretsManager(
|
|
{
|
|
"tok-1": {
|
|
"stages": {"AWSCURRENT", "AWSPENDING"},
|
|
"value": json.dumps({"keys": [KEY_A]}),
|
|
}
|
|
}
|
|
)
|
|
_run(monkeypatch, rotator, fake, "finishSecret")
|
|
assert fake.stage_calls == []
|
|
|
|
|
|
# --- Dispatch ----------------------------------------------------------------
|
|
|
|
|
|
def test_set_secret_is_a_noop(monkeypatch, rotator):
|
|
fake = _bootstrap_fake()
|
|
_run(monkeypatch, rotator, fake, "setSecret")
|
|
assert fake.put_calls == []
|
|
assert fake.stage_calls == []
|
|
|
|
|
|
def test_unknown_step_raises(monkeypatch, rotator):
|
|
fake = _bootstrap_fake()
|
|
with pytest.raises(ValueError, match="Unknown rotation step"):
|
|
_run(monkeypatch, rotator, fake, "rotateHarder")
|