mirror of
https://github.com/Sea-Haven-Industries/procurement-ingest.git
synced 2026-10-07 16:08:54 +00:00
Compare commits
1 commit
4da79a1e47
...
6d08c05729
| Author | SHA1 | Date | |
|---|---|---|---|
|
6d08c05729 |
36 changed files with 18 additions and 4333 deletions
4
.github/workflows/ci.yaml
vendored
4
.github/workflows/ci.yaml
vendored
|
|
@ -19,8 +19,8 @@ jobs:
|
|||
spec-lint:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
|
||||
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
|
||||
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
|
||||
with:
|
||||
node-version: 22
|
||||
cache: npm
|
||||
|
|
|
|||
54
README.md
54
README.md
|
|
@ -22,7 +22,9 @@ Coupa PO emails are received at `amazon_po@int.seahaven.com`, parsed **determini
|
|||
- `new_po` — merge insert. Creates the PO, or backfills data into a pre-existing `Cancelled` skeleton left by an out-of-order cancellation (preserving the `Cancelled` status). No longer silently dropped when a record already exists.
|
||||
- `revision` — field-level merge (`update_item` SETs only the fields present in the revision). A revision that omits `line_items`/`supplier` no longer deletes them. Will not un-cancel a `Cancelled` PO.
|
||||
- `cancellation` — marks the row `Cancelled` (creating a minimal skeleton if the cancellation arrives before the `new_po`).
|
||||
7. DynamoDB Streams feeds the **site extractor** (`po-ingest-site-extractor`) — real-time site address extraction into the `verified-sites` table. (LedgerFlow / `seahaven-slack-bot/po-sync`, the former second stream consumer, was decommissioned 2026-07-23.)
|
||||
7. DynamoDB Streams feeds downstream consumers:
|
||||
- **LedgerFlow** (`seahaven-slack-bot/po-sync`) — daily KB sync
|
||||
- **Site extractor** (`po-ingest-site-extractor`) — real-time site address extraction into `verified-sites` table
|
||||
|
||||
**Deterministic template parser.** `template_parser.try_deterministic_parse()` classifies by exact subject regex, extracts the nested contract (`supplier{}`, `ship_to{}` — 8 keys, `line_items[]` — 10 keys per item), and returns a parsed result **only if** it passes a fail-closed validation gate: recursive exact key-set at every nesting level; `email_type` emitted **only** on the exact new-PO subject *and* a confirmed-safe `Status` (never defaulted — a cancellation misrouted as `new_po` would defeat the sticky-`Cancelled` guard); `po_number` shape + byte-equality with the subject, the body `PO ID`, the `Amazon Purchase Order #` heading, and the `orders/<id>` URL; duplicate-label anchor integrity (`Supplier`/`Shipping`/`Total` each appear twice, the first `Shipping` must be the literal `None` placeholder); money fidelity (every `Decimal` re-serializes byte-identically to its source token with a digit/comma border check — the thousands-separator-truncation kill switch — plus `sum(line amounts) == total`, proven against **both** `Total` blocks); bullet-metadata label discipline (closed label set, assigned by leading label, never ordinal — immune to the optional `Part Number` segment); USPS address shape on the raw pre-enrichment value; and rejection of any unparseable sentinel or residual `\r`/`\xa0` artifact. Multi-line-item (0.18%) and non-USD (0 observed) new-POs, comment emails, and anything else falls back to the AI extractor. The AI path is gated too: the untrusted email reaches Bedrock inside a neutralized `<email>` data block (forged tag lookalikes in the body are defanged) with `temperature=0`, and the raw model output must pass the fail-closed `validate_ai_fallback()` gate — the PO-specific nested contract (exact key-set at every level, with missing keys normalized rather than rejected), a `po_number` shape check hardened against fullwidth-digit and trailing-artifact injection (the same regex family protecting the DynamoDB partition key the handler builds from it), an `email_type` allow-list enforced *before* dispatch so a miss can never fall into the `new_po` default branch, and `Decimal`/`int`/`None` money typing (PO decodes with `parse_float=Decimal`) — before any DynamoDB write. **The template path is gate-enforced end-to-end; the AI-fallback path is validated and fail-closed — output that fails the gate is skipped, never written (see `ai_fallback_rejected` below), so a malformed or injected email raises the fallback rate rather than corrupting a record.** Every record emits one CloudWatch EMF metric (see below).
|
||||
|
||||
|
|
@ -55,7 +57,7 @@ Or aggregate overall agreement per field: `| filter ispresent(DerivedFieldAgreem
|
|||
| `po-web-ui` | Manual invoke (authenticated — see Setup §6) | HTML dashboard (public Function URL removed 2026-06-08, INFRA-74) |
|
||||
|
||||
**Tables:**
|
||||
- `purchase-orders` (PK: `po_number`, Streams: NEW_IMAGE) — read by `procurement-api` (see Shared Resources; the former `seahaven-slack-bot` reader was decommissioned 2026-07-23)
|
||||
- `purchase-orders` (PK: `po_number`, Streams: NEW_IMAGE) — shared with seahaven-slack-bot (read-only; see Shared Resources)
|
||||
- `verified-sites` (PK: `siteCode`) — ~1,100 unique Amazon facility sites (`by-state` GSI removed 2026-06-03, audit M-20)
|
||||
- `pending-site-review` (PK: `po_number`) — unresolvable POs for manual Payee Central verification
|
||||
|
||||
|
|
@ -79,23 +81,10 @@ Amazon APM work order emails (from Hexagon EAM / HxGN SmartCloud) are received a
|
|||
|---|---|---|
|
||||
| `workorder-email-processor` | S3 ObjectCreated | Claude extraction + DynamoDB write |
|
||||
| `workorder-web-ui` | Manual invoke (authenticated — see Setup §6) | HTML dashboard (public Function URL removed 2026-06-08, INFRA-74) |
|
||||
| `workorder-shoc-emitter` | DynamoDB Streams, both WO tables (**ESMs ship disabled** — see [SHOC webhook feed](#shoc-webhook-feed-workorder-shoc-emitter)) | HMAC-signed webhook push of every WO mutation to the SHOC backend |
|
||||
| `workorder-shoc-hmac-rotator` | Secrets Manager rotation schedule (30 days) | Rotates the webhook HMAC signing keys (dual-key overlap) |
|
||||
|
||||
**Tables:**
|
||||
- `WorkOrders` (PK: `work_order_id`, Streams: NEW_AND_OLD_IMAGES) — `site-code-index` and `status-index` GSIs removed 2026-06-03 (audit M-20)
|
||||
- `WorkOrderComments` (PK: `work_order_id`, SK: `comment_id`, Streams: NEW_AND_OLD_IMAGES) — see the `comment_id` format note below
|
||||
|
||||
### SHOC webhook feed (`workorder-shoc-emitter`)
|
||||
|
||||
**Flow:** `WorkOrders` / `WorkOrderComments` DynamoDB Streams (`NEW_AND_OLD_IMAGES` — the OLD image is what lets the emitter detect the `wo_status → cancelled` transition) → `workorder-shoc-emitter` → HMAC-signed HTTPS POST → SHOC backend. Every WO mutation becomes one webhook event (`work_order.created` / `.updated` / `.cancelled` / `.comment_added`) within seconds of the DynamoDB commit; DynamoDB stays the source of truth.
|
||||
|
||||
- **Contract:** `docs/shoc-webhook-contract.md` (Rev 2026-07-23) is the producer/consumer contract SHOC builds its receiver against, and `docs/shoc-webhook-test-vectors.json` is the shared receiver-verification vector set — both sides pin their HMAC implementation against the same vectors (producer-side via the golden-vector tests over `delivery.sign_body`).
|
||||
- **Ships DARK.** Both event-source mappings deploy with `enabled=False`: the full stack (Lambda, queues, alarms, secret, rotation) deploys and is testable with zero deliveries while SHOC has no receiver, so this repo's merge cadence never depends on SHOC's. Activation is a deliberate one-line `enabled=True` PR, gated on the SHOC receiver passing the shared test vectors. The ESMs start at `LATEST` — no historical flood; SHOC loads history through the `procurement-api` read API at activation instead.
|
||||
- **Ordering/retry semantics.** `parallelization_factor=1`, `bisect_batch_on_error=False`, `retry_attempts=-1`, `maximum_record_age=24h`: a retryable failure (429/5xx/timeout/connection error) blocks the shard and retries from the failed record — per-work-order commit order is the guarantee, and blocking is the intended behavior when SHOC is down. `report_batch_item_failures` keeps earlier in-batch successes from being re-delivered. Records that exhaust the 24h age are parked as ESM **failure metadata** (not full records) on `workorder-shoc-emitter-failures` (`on_failure` destination); a non-retryable 4xx (a contract bug, never worth blocking the shard for 24h) parks the **full `{envelope, response_status}` payload** on `workorder-shoc-emitter-rejected` and the loop continues. Both queues: 14-day retention, SSL-enforced, alarmed (see [CloudWatch alarms](#cloudwatch-alarms)); recovery is `scripts/replay_shoc_webhooks.py` (see [Scripts](#scripts)).
|
||||
- **Secret + KMS.** The HMAC signing keys live in Secrets Manager secret `workorder-ingest/shoc-webhook-hmac` (value `{"keys": [{"kid", "secret"}, ...]}`, newest first, max 2), encrypted with the dedicated CMK `workorder-ingest-shoc-webhook-kms` — deliberately **not** `alias/seahaven-dynamodb`, so the SHOC cross-account grant's decrypt reach covers exactly this one secret and nothing else. The secret's removal policy is **`DESTROY`, deliberately not `RETAIN`**: the value is machine-generated HMAC material, fully regenerable by a single rotation, so `RETAIN` buys nothing and would expose the fixed-name RETAIN-orphan deadlock (a failed create orphans an empty shell holding the global name; every later create fails `AlreadyExists`). **Accidental-deletion recovery runbook:** redeploy to recreate the secret, force a rotation (`aws secretsmanager rotate-secret --secret-id workorder-ingest/shoc-webhook-hmac`), notify the SHOC team — receivers re-fetch within their ≤5-minute cache TTL, so no coordination window is needed — then watch the `-failures` queue and replay the gap with the replay script.
|
||||
- **Rotation.** `workorder-shoc-hmac-rotator` runs on a 30-day schedule: it prepends a fresh 64-hex-char key as `keys[0]` and truncates the list to 2 entries (one overlap cycle). `kid` format is `YYYY-MM-DDTHH`. The emitter always signs with `keys[0]` behind a 5-minute TTL cache; the receiver accepts any listed `kid` and re-fetches on an unknown one — there is no delivery window in which signatures can't verify.
|
||||
- **Cross-account grants (exact ARN only):** `arn:aws:iam::396287094661:role/shoc-backend-dev` is granted `secretsmanager:GetSecretValue` on the secret's resource policy **and** `kms:Decrypt` on the CMK's key policy — both halves are required; either one alone fails silently at the receiver. Future staging/prod receiver roles are each a deliberate, individually-reviewed policy addition — no wildcard/prefix trust.
|
||||
- `WorkOrders` (PK: `work_order_id`) — `site-code-index` and `status-index` GSIs removed 2026-06-03 (audit M-20)
|
||||
- `WorkOrderComments` (PK: `work_order_id`, SK: `comment_id`) — see the `comment_id` format note below
|
||||
|
||||
### Procurement API (`procurement-api` stack)
|
||||
|
||||
|
|
@ -111,7 +100,6 @@ A read-only REST API (API Gateway + the `procurement-api` Lambda, `lambdas/api/`
|
|||
|
||||
- **Spec is source of truth:** `lambdas/api/openapi.json` (OpenAPI 3.1). Its top-level `webhooks` section documents the outbound SHOC work-order feed, so one page describes both directions (call + be-called). `tests/test_api_spec_drift.py` pins the spec's paths to the router table, so spec and implementation cannot drift.
|
||||
- **Auth:** data routes use API Gateway `AWS_IAM` (SigV4) plus a resource policy allowing exactly `arn:aws:iam::396287094661:role/shoc-backend-dev` on `GET/*`; same-account admin callers authorize via identity policy (Postman signs SigV4 natively). Docs routes are auth `NONE` at the gateway (resource-policy carve-out for exactly those two GETs) but the handler fails closed on the shared token (`lambdas/shared/web_ui_auth.py`, secret `procurement-ingest/web-ui-auth-token`) — not an unauthenticated data path (INFRA-74 posture).
|
||||
- **Custom domain:** `https://procurement-api.seahaven.com` (REGIONAL API Gateway domain, TLS 1.2, empty base-path mapping to the `prod` stage, so callers hit `/work-orders` with no `/prod` segment). The stable SHOC-facing endpoint; the raw `*.execute-api.us-east-1.amazonaws.com/prod` URL still works. **Cross-account DNS:** the `seahaven.com` public zone is in the mgmt account (`328440206208`), so the ACM cert's validation record and the A-alias are added there out of band — `scripts/setup_procurement_api_domain.sh cert` issues the cert (DNS-validated against the mgmt zone) and writes its ARN to prod SSM `/procurement-api/custom-domain/certificate-arn`, which the stack reads (`value_for_string_parameter`, CFN-resolved at deploy); after `cdk deploy procurement-api`, `scripts/setup_procurement_api_domain.sh alias` adds the A-alias from the stack's `ProcurementApiAliasTarget`/`ProcurementApiAliasHostedZoneId` outputs. SigV4 is unaffected (same underlying API id + resource policy); the docs page injects whichever host served the request into `servers[0].url`.
|
||||
- **KMS:** `purchase-orders` is CMK-encrypted; the imported-by-name table doesn't carry the key association, so the stack grants `kms:Decrypt`/`DescribeKey` on the CMK from SSM `/seahaven/dynamodb/cmk-arn` explicitly (the INFRA-104 failure class).
|
||||
- **No access logging in v1** (keeps the `?token=` shim out of any log and avoids the account-level API Gateway CloudWatch role); rotate the docs token before ever enabling it. No CORS (server-to-server + Postman callers).
|
||||
- **Alarms:** `procurement-api-errors`/`-throttles`/`-duration` (p99 ≥ 22.5 s) + gateway `procurement-api-5xx`, all ALARM-only → `site-alerts`. No 4XX alarm (401/403/404 are expected traffic).
|
||||
|
|
@ -172,9 +160,9 @@ Every alarm is **ALARM-only** (no OK action), sends to the shared `site-alerts`
|
|||
|
||||
| Alarm | Functions | Metric / config |
|
||||
|---|---|---|
|
||||
| `<fn>-errors` | `po-email-processor`, `po-ingest-site-extractor`, `workorder-email-processor`, `workorder-shoc-emitter`, `workorder-shoc-hmac-rotator` | `Errors` Sum, 5 min, `> 0`, eval 1 |
|
||||
| `<fn>-throttles` | `po-email-processor`, `po-ingest-site-extractor`, `po-web-ui`, `workorder-email-processor`, `workorder-shoc-emitter`, `workorder-shoc-hmac-rotator` | `Throttles` Sum, 5 min, `> 0`, eval 1 |
|
||||
| `<fn>-duration` | `po-email-processor`, `po-ingest-site-extractor`, `po-web-ui`, `workorder-shoc-emitter`, `workorder-shoc-hmac-rotator` (p99); `workorder-email-processor` (p95) | `Duration` percentile, 5 min, `>= 45000` ms (75% of the 60s timeout), eval 3 / datapoints 2 |
|
||||
| `<fn>-errors` | `po-email-processor`, `po-ingest-site-extractor`, `workorder-email-processor` | `Errors` Sum, 5 min, `> 0`, eval 1 |
|
||||
| `<fn>-throttles` | `po-email-processor`, `po-ingest-site-extractor`, `po-web-ui`, `workorder-email-processor` | `Throttles` Sum, 5 min, `> 0`, eval 1 |
|
||||
| `<fn>-duration` | `po-email-processor`, `po-ingest-site-extractor`, `po-web-ui` (p99); `workorder-email-processor` (p95) | `Duration` percentile, 5 min, `>= 45000` ms (75% of the 60s timeout), eval 3 / datapoints 2 |
|
||||
| `<fn>-sender-auth-rejected` | `po-email-processor`, `workorder-email-processor` | Log-metric-filter count (namespace `Seahaven/ProcurementIngest`, `default_value=0`) on `sender_auth_rejected` warnings, `Sum` 5 min, `>= 1`, eval 3 / datapoints 2 |
|
||||
|
||||
The `<fn>-sender-auth-rejected` alarm closes the silent-drop gap in INFRA-107: a rejected email returns normally (no error, no retry, no DLQ message), so without a log-metric filter a signing-domain drift or a wrong allowlist would discard 100% of legitimate mail while every other alarm stayed green. It counts `sender_auth_rejected` warnings per 5-minute period (`default_value=0` keeps the series continuous) and pages when 2 of the last 3 periods each see at least one rejection — a lone stray spoof probe to the internal ingest address self-clears, but a sustained false-reject storm pages within ~10–15 minutes even at low mail volume; the config is easy to tune in the CDK helper. (A residual gap remains for a *very* sparse total-reject outage — see the SES-AR-01/02 hardening issue.)
|
||||
|
|
@ -183,8 +171,6 @@ The `<fn>-duration` and `<fn>-throttles` alarms for `po-email-processor` and `wo
|
|||
|
||||
**DLQ alarms** (`AWS/SQS`): `po-email-processor-dlq-messages` and `workorder-email-processor-dlq-messages` fire when any message is visible on an email-processor DLQ (`ApproximateNumberOfMessagesVisible` Maximum, 5 min, `> 0`, eval 1) — a message there means an email was dropped after Lambda exhausted its async retries. Recovery from a DLQ message (no console redrive) is documented in the [DLQ recovery runbook](docs/runbook-dlq-recovery.md).
|
||||
|
||||
**SHOC emitter alarms** (bespoke — these don't fit the standard-Lambda-alarm helper's shape): `workorder-shoc-emitter-iterator-age` (`IteratorAge` Maximum, 5 min, `>= 600000` ms, eval 3 / datapoints 2) fires when the stream lags ≥ 10 minutes — SHOC is likely down and the shard is blocking on retries, which is exactly the ordered-backpressure design working, but an operator should know. `workorder-shoc-emitter-failures-messages` and `workorder-shoc-emitter-rejected-messages` (`ApproximateNumberOfMessagesVisible` Maximum, 5 min, `> 0`, eval 1) page the replay runbook: the `-failures` queue receives ESM failure **metadata** for retry-exhausted records, the `-rejected` queue receives the **full parked payloads** of non-retryable 4xx deliveries (see the [SHOC webhook feed](#shoc-webhook-feed-workorder-shoc-emitter) section and `scripts/replay_shoc_webhooks.py`). All three: ALARM-only → `site-alerts`, `NOT_BREACHING`, per the house rules above.
|
||||
|
||||
**Parse-outcome metric + fallback-rate alarm (workorder-ingest):** the WO processor writes one CloudWatch **EMF** line per email to namespace `Seahaven/WorkorderIngest`, metric `ParseOutcome` (Unit Count, value 1), dimensioned by `ParseMethod` (`template` | `ai_fallback` | `ai_fallback_rejected`) and `TemplateId` (`update_plaintext` | `assign_html` | `unknown`). `ai_fallback_rejected` counts AI-fallback output that failed the fail-closed `validate_ai_fallback()` gate (schema/enum/date contract on raw Bedrock output — prompt-injection defence) and was dropped without a DynamoDB write. Non-dimension EMF properties `ReasonCode` and `work_order_id` are queryable in Logs Insights but not promoted to metrics (kept low-cardinality). EMF is used instead of `PutMetricData` so there is no extra sync call / latency / IAM grant on the async hot path (the role already has `logs:PutLogEvents`). The alarm `workorder-email-processor-template-fallback-rate` fires when the AI-fallback share of parses — rejected fallback parses included, so a drift outage whose AI output also fails the gate cannot lower the observed rate while dropping mail — exceeds **15%** sustained (a `MathExpression` with `FILL(...,0)` and a ≥10-sample volume floor over 15-minute periods, eval 3 / datapoints 2) — catching Hexagon template-drift coverage collapse while the volume floor + `FILL` prevent low-volume false pages / `INSUFFICIENT_DATA`. ALARM-only `SnsAction` to `site-alerts`, no OK action, `NOT_BREACHING`. The 15-minute period is a deliberate deviation from the 5-minute house style to accumulate a stable denominator at the low ~760/day volume. A second alarm, `workorder-email-processor-ai-fallback-rejected`, pages on the rejected series itself (≥1 rejection per 5-min period, 2 of the last 6 periods — the sender-auth-rejected sparse-arrival idiom) because a gate rejection drops mail without error/retry/DLQ and would otherwise be silent.
|
||||
|
||||
**WO Bedrock transport-error metric (Phase 8).** A Bedrock-side transport error (throttling, malformed response, non-JSON model text) during the AI-fallback attempt previously emitted **zero** `ParseOutcome` datapoints — the only emit sites were post-gate. `handler.py` now wraps the `extract_with_bedrock` call in a try/except that emits exactly one `ParseMethod=ai_fallback` / `ReasonCode=bedrock_error` datapoint and then re-raises (the exception still propagates into the errors alarm / DLQ path unchanged). This is an except-and-reraise, not a reorder: a gate-rejected email (Bedrock *returns* successfully, `validate_ai_fallback()` then rejects it) still emits only the single `ai_fallback_rejected` datapoint and nothing else — the except branch never fires because Bedrock did not raise — so the `workorder-email-processor-ai-fallback-rejected` "a rejected email emits nothing else" alarm contract holds with no double-count.
|
||||
|
|
@ -317,7 +303,7 @@ Both tables are **owned by this repo's `WorkorderIngestStack`** (`cdk/wo_stack.p
|
|||
|
||||
Both currently use default DynamoDB encryption — they are **not** yet on the shared customer-managed CMK (`alias/seahaven-dynamodb`, INFRA-95 / M-3); that migration is tracked in INFRA-6. The `workorder-email-processor` role no longer holds a pre-emptive encrypt/decrypt grant on that CMK (removed in the 2026-06-17 security sweep — it was unused while the tables are unencrypted and extended the role's decrypt reach to the CMK protecting `purchase-orders`). Re-add the grant as part of the INFRA-6 migration, at which point `grant_read_write_data` on the then-encrypted tables propagates the needed key permissions automatically.
|
||||
|
||||
**Consumers (read-only) — data contract:** `seahaven-slack-bot` (the former external reader) was decommissioned 2026-07-23; its grants are gone. Current consumers: the `procurement-api` Lambda (this repo, `Table.from_table_name` + `grant_read_data`, serving `GET /work-orders*`), and the `workorder-shoc-emitter` stream consumer (this repo) — both tables now stream `NEW_AND_OLD_IMAGES`, and the emitter is a live consumer of those streams, dark (ESMs disabled) until the SHOC activation PR. External readers (SHOC) consume through the REST contract (`lambdas/api/openapi.json`) and the webhook contract (`docs/shoc-webhook-contract.md`); both documents' field lists mirror `lambdas/wo/email_processor/persistence.py`. Any change to table name, key schema, attribute names, or encryption configuration (e.g. the INFRA-6 CMK migration) must update those two contracts in the same PR — the tables are imported by name, so there is no compile-time link and breakage surfaces at runtime.
|
||||
**Consumers (read-only) — data contract:** `seahaven-slack-bot` (the former external reader) was decommissioned 2026-07-23; its grants are gone. Current consumers: the `procurement-api` Lambda (this repo, `Table.from_table_name` + `grant_read_data`, serving `GET /work-orders*`), and — once the SHOC webhook ships — the `workorder-shoc-emitter` stream consumer. External readers (SHOC) consume through the REST contract (`lambdas/api/openapi.json`) and the webhook contract (`docs/shoc-webhook-contract.md`); both documents' field lists mirror `lambdas/wo/email_processor/persistence.py`. Any change to table name, key schema, attribute names, or encryption configuration (e.g. the INFRA-6 CMK migration) must update those two contracts in the same PR — the tables are imported by name, so there is no compile-time link and breakage surfaces at runtime.
|
||||
|
||||
**Stream field contract (`site_code`).** Three definitions of "is this a valid site code" have existed in this repo at once. The canonical shape is `derived_fields._STRICT_CODE_RE` (`[A-Z][A-Z0-9]{2,4}`, `fullmatch`) plus its skip-list semantics (a code-shaped token is only a real site code if it is *not* skip-listed, e.g. `LLC`/`INC`/`CORP`/`LTD`/`ATTN`), used by `derive_site_code()` (the `enrich_parsed()` classifier, see the **Derived fields** note in the Purchase Orders flow above). Honestly noted, not papered over: `lambdas/po/site_extractor/handler.py`'s own `SITE_CODE_PATTERN` (`[A-Z]{2,4}\d{1,2}`, prefix-anchored `.match`, digit-requiring) still diverges from the canonical shape as of this phase — it rejects valid all-letter codes like `KLAL` (which surface as permanent `pending-site-review` rows) and accepts overlong junk like `DLI6X`/`SNY55` that the canonical `fullmatch` would not. Reconciling `po-ingest-site-extractor`'s direct-field validation onto the canonical `derived_fields` shape is Phase 6 scope, tracked separately — this paragraph will be updated when it lands.
|
||||
|
||||
|
|
@ -419,13 +405,6 @@ python scripts/reprocess.py --pipeline wo --all --execute
|
|||
python scripts/backfill_sites.py
|
||||
```
|
||||
|
||||
**Replay SHOC webhooks** (rebuild work-order webhook events from DynamoDB and re-POST them, marked `"replay": true`). `replay_shoc_webhooks.py` is the operator runbook for the `workorder-shoc-emitter-failures` / `-rejected` alarms: when deliveries were parked (SHOC down past the 24h retry window, or a contract-bug 4xx), replay re-sends the affected work orders from **current table state** — receivers dedupe on the deterministic `delivery_id`, so overlapping or repeated runs are harmless. Selection is exactly one of `--work-order-id` (repeatable, targeted) or `--since` (a full table Scan — a count banner prints per table); `--events` narrows to state rows, comments, or both. `--url` is required with no default — replay must be a deliberate act against a known receiver. Every mode is dry-run unless `--execute`.
|
||||
```bash
|
||||
python scripts/replay_shoc_webhooks.py --url https://... --work-order-id 11144580730 # targeted, dry-run first
|
||||
python scripts/replay_shoc_webhooks.py --url https://... --work-order-id 11144580730 --execute # then re-POST
|
||||
python scripts/replay_shoc_webhooks.py --url https://... --since 2026-07-24T02:00:00Z --execute # everything touched since (full Scan)
|
||||
```
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
|
|
@ -435,8 +414,7 @@ cdk/
|
|||
# email bucket, processor DLQ, fallback-rate alarm) -- called with
|
||||
# each stack's own scope + literal construct ids, logical-ID-safe
|
||||
po_stack.py # Purchase order pipeline resources
|
||||
wo_stack.py # Work order pipeline resources + the SHOC webhook feed (secret/CMK,
|
||||
# rotator, emitter, disabled ESMs, queues, alarms)
|
||||
wo_stack.py # Work order pipeline resources
|
||||
procurement_api_stack.py # REST API (IAM SigV4 + resource policy) over both pipelines' tables + token-gated /docs
|
||||
lambdas/ # Phase 2: shared Code.from_asset("../lambdas") bundling root for
|
||||
# BOTH po-email-processor and workorder-email-processor (and, since
|
||||
|
|
@ -508,19 +486,9 @@ lambdas/ # Phase 2: shared Code.from_asset("../lambdas") bundling
|
|||
ses-stamped/
|
||||
auth-pass-01.eml # Phase 8: the one new fixture allowed this phase -- a synthesized Authentication-Results header block (from WO_SES_HEADER in test_ses_auth.py), not scraped mail
|
||||
web_ui/ # handler.py imports `from web_ui_auth import is_authenticated` (shared)
|
||||
shoc_emitter/ # SHOC webhook emitter (ships dark -- ESMs disabled; see the SHOC
|
||||
# webhook feed section). Flat siblings, bare-name imports
|
||||
handler.py # thin per-record event loop + partial-batch failure report + rejected-queue parking
|
||||
envelope.py # pure stream-record -> envelope mapping + event classification (incl. echo guard)
|
||||
delivery.py # TTL-cached secret fetch + HMAC signing (sign_body) + POST + response classification
|
||||
shoc_hmac_rotator/
|
||||
handler.py # 30-day Secrets Manager rotation (single-user: createSecret/finishSecret;
|
||||
# dual-key overlap, kid YYYY-MM-DDTHH)
|
||||
scripts/
|
||||
reprocess.py
|
||||
backfill_sites.py
|
||||
replay_shoc_webhooks.py # rebuild + re-POST webhook events ("replay": true) -- the
|
||||
# -failures/-rejected alarm runbook; dry-run unless --execute
|
||||
post-deploy-smoke.sh # CD gate: synchronous healthcheck invoke of the three smoke-gated functions, checks FunctionError
|
||||
conftest.py # Phase 8: THE repo-root session-invariant conftest. rootdir is pinned by
|
||||
# pytest.ini at repo root, so this loads for every pytest invocation shape --
|
||||
|
|
|
|||
|
|
@ -18,7 +18,6 @@ from aws_cdk import (
|
|||
Duration,
|
||||
Stack,
|
||||
aws_apigateway as apigateway,
|
||||
aws_certificatemanager as acm,
|
||||
aws_cloudwatch as cloudwatch,
|
||||
aws_cloudwatch_actions as cw_actions,
|
||||
aws_dynamodb as dynamodb,
|
||||
|
|
@ -40,12 +39,6 @@ SHOC_BACKEND_DEV_ROLE_ARN = "arn:aws:iam::396287094661:role/shoc-backend-dev"
|
|||
|
||||
STAGE_NAME = "prod"
|
||||
|
||||
# Custom domain for the SHOC-facing read API. The seahaven.com zone is in the
|
||||
# mgmt account, so the cert is issued out of band and its ARN handed in via
|
||||
# this SSM parameter (see the custom-domain block below).
|
||||
CUSTOM_DOMAIN_NAME = "procurement-api.seahaven.com"
|
||||
CUSTOM_DOMAIN_CERT_ARN_SSM_PARAM = "/procurement-api/custom-domain/certificate-arn"
|
||||
|
||||
|
||||
class ProcurementApiStack(Stack):
|
||||
def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
|
||||
|
|
@ -287,41 +280,6 @@ class ProcurementApiStack(Stack):
|
|||
authorization_type=apigateway.AuthorizationType.NONE,
|
||||
)
|
||||
|
||||
# --- Custom domain: procurement-api.seahaven.com ---
|
||||
# Stable, brandable endpoint for the SHOC backend to sign SigV4 against
|
||||
# (replaces the opaque execute-api URL). REGIONAL to match the API, so
|
||||
# the ACM cert lives in this account+region (us-east-1).
|
||||
#
|
||||
# CROSS-ACCOUNT DNS: the seahaven.com public zone
|
||||
# (Z06652411XKH89KTZD3XA) is in the mgmt account (328440206208), not
|
||||
# here. So the cert's DNS-validation CNAME and the final A-alias record
|
||||
# are added to that zone OUT OF BAND (see the README runbook /
|
||||
# scripts/setup_procurement_api_domain.sh), and the issued cert ARN is
|
||||
# handed to this stack via SSM. Reading it with value_for_string_
|
||||
# parameter keeps the reference CFN-resolved at deploy -- no synth-time
|
||||
# AWS creds, unlike a cross-account HostedZone.from_lookup. The
|
||||
# A-alias record itself is created out of band too (the zone is not in
|
||||
# this account, so CDK cannot own it); the outputs below give the exact
|
||||
# alias target.
|
||||
cert_arn = ssm.StringParameter.value_for_string_parameter(
|
||||
self, CUSTOM_DOMAIN_CERT_ARN_SSM_PARAM
|
||||
)
|
||||
certificate = acm.Certificate.from_certificate_arn(
|
||||
self, "ProcurementApiCertificate", cert_arn
|
||||
)
|
||||
custom_domain = apigateway.DomainName(
|
||||
self,
|
||||
"ProcurementApiDomain",
|
||||
domain_name=CUSTOM_DOMAIN_NAME,
|
||||
certificate=certificate,
|
||||
endpoint_type=apigateway.EndpointType.REGIONAL,
|
||||
security_policy=apigateway.SecurityPolicy.TLS_1_2,
|
||||
)
|
||||
# Empty base path: the custom domain root maps to the prod stage, so
|
||||
# callers hit https://procurement-api.seahaven.com/work-orders (no
|
||||
# /prod segment -- the mapping strips it).
|
||||
custom_domain.add_base_path_mapping(api, stage=api.deployment_stage)
|
||||
|
||||
# --- Alarms (ALARM-only -> site-alerts, NOT_BREACHING) ---
|
||||
# common.add_standard_lambda_alarms is NOT used here: its duration
|
||||
# threshold is a fixed 45000 ms (75% of the processors' 60 s timeout),
|
||||
|
|
@ -404,24 +362,3 @@ class ProcurementApiStack(Stack):
|
|||
value=api_fn.function_arn,
|
||||
description="ARN of the procurement-api Lambda",
|
||||
)
|
||||
cdk.CfnOutput(
|
||||
self,
|
||||
"ProcurementApiCustomDomainUrl",
|
||||
value=f"https://{CUSTOM_DOMAIN_NAME}/",
|
||||
description="procurement-api custom domain (SHOC-facing endpoint)",
|
||||
)
|
||||
# A-alias target for the mgmt-account Route53 record. Create an ALIAS A
|
||||
# record: procurement-api.seahaven.com -> this regional domain name,
|
||||
# with this hosted-zone id, EvaluateTargetHealth=false.
|
||||
cdk.CfnOutput(
|
||||
self,
|
||||
"ProcurementApiAliasTarget",
|
||||
value=custom_domain.domain_name_alias_domain_name,
|
||||
description="Route53 A-alias target (add in the mgmt seahaven.com zone)",
|
||||
)
|
||||
cdk.CfnOutput(
|
||||
self,
|
||||
"ProcurementApiAliasHostedZoneId",
|
||||
value=custom_domain.domain_name_alias_hosted_zone_id,
|
||||
description="Route53 A-alias target hosted-zone id (mgmt zone record)",
|
||||
)
|
||||
|
|
|
|||
|
|
@ -1,2 +1,2 @@
|
|||
aws-cdk-lib==2.262.1
|
||||
aws-cdk-lib==2.261.0
|
||||
constructs==10.7.1
|
||||
|
|
|
|||
473
cdk/wo_stack.py
473
cdk/wo_stack.py
|
|
@ -8,17 +8,13 @@ from aws_cdk import (
|
|||
aws_cloudwatch as cloudwatch,
|
||||
aws_cloudwatch_actions as cw_actions,
|
||||
aws_dynamodb as dynamodb,
|
||||
aws_iam as iam,
|
||||
aws_kms as kms,
|
||||
aws_lambda as lambda_,
|
||||
aws_lambda_event_sources as lambda_event_sources,
|
||||
aws_s3 as s3,
|
||||
aws_s3_notifications as s3n,
|
||||
aws_ses as ses,
|
||||
aws_ses_actions as ses_actions,
|
||||
aws_secretsmanager as secretsmanager,
|
||||
aws_sns as sns,
|
||||
aws_sqs as sqs,
|
||||
)
|
||||
from constructs import Construct
|
||||
|
||||
|
|
@ -55,10 +51,6 @@ class WorkorderIngestStack(Stack):
|
|||
type=dynamodb.AttributeType.STRING,
|
||||
),
|
||||
billing_mode=dynamodb.BillingMode.PAY_PER_REQUEST,
|
||||
# NEW_AND_OLD_IMAGES: the SHOC emitter needs OLD.wo_status to
|
||||
# classify the cancelled transition (docs/shoc-webhook-plan.md
|
||||
# Phase 2). In-place CFN update -- no table replacement.
|
||||
stream=dynamodb.StreamViewType.NEW_AND_OLD_IMAGES,
|
||||
removal_policy=RemovalPolicy.RETAIN,
|
||||
)
|
||||
# site-code-index and status-index GSIs removed 2026-06-03 (audit M-20):
|
||||
|
|
@ -78,8 +70,6 @@ class WorkorderIngestStack(Stack):
|
|||
type=dynamodb.AttributeType.STRING,
|
||||
),
|
||||
billing_mode=dynamodb.BillingMode.PAY_PER_REQUEST,
|
||||
# Streamed for the SHOC emitter (docs/shoc-webhook-plan.md Phase 2).
|
||||
stream=dynamodb.StreamViewType.NEW_AND_OLD_IMAGES,
|
||||
removal_policy=RemovalPolicy.RETAIN,
|
||||
)
|
||||
|
||||
|
|
@ -420,466 +410,3 @@ class WorkorderIngestStack(Stack):
|
|||
# unauthenticated FunctionUrlAuthType.NONE URL was deleted out-of-band
|
||||
# via CLI. Removing the construct (and its auto-generated Principal:*
|
||||
# invoke permission) reconciles IaC with the live state.
|
||||
|
||||
# =====================================================================
|
||||
# --- SHOC webhook emitter (docs/shoc-webhook-plan.md) ---
|
||||
# Realtime work-order feed to the SHOC backend: DynamoDB Streams on the
|
||||
# two WO tables -> workorder-shoc-emitter -> HMAC-signed HTTPS POST.
|
||||
# Contract: docs/shoc-webhook-contract.md (Rev 2026-07-23). Built in
|
||||
# _add_shoc_webhook_emitter (module helper, common.py plain-helper
|
||||
# style) to keep __init__ under the PLR0915 statement ceiling; the
|
||||
# stack stays the construct scope, so extraction does not move any
|
||||
# logical ID.
|
||||
# =====================================================================
|
||||
_add_shoc_webhook_emitter(self, work_orders_table, comments_table, alarm_topic)
|
||||
|
||||
|
||||
def _scope_rotation_invoke_permission(stack, rotator_fn, secret):
|
||||
"""Add SourceAccount/SourceArn to the generated rotation invoke permission.
|
||||
|
||||
``add_rotation_schedule`` emits an ``AWS::Lambda::Permission`` for the
|
||||
``secretsmanager.amazonaws.com`` service principal with no source
|
||||
conditions. Rather than add a second (additive) permission, find that
|
||||
generated ``CfnPermission`` and pin it to this account + secret so only
|
||||
this secret's Secrets Manager can invoke the rotator.
|
||||
"""
|
||||
patched = False
|
||||
for child in stack.node.find_all():
|
||||
if (
|
||||
isinstance(child, lambda_.CfnPermission)
|
||||
and child.principal == "secretsmanager.amazonaws.com"
|
||||
and stack.resolve(child.function_name)
|
||||
== stack.resolve(rotator_fn.function_arn)
|
||||
):
|
||||
child.source_account = stack.account
|
||||
child.source_arn = secret.secret_arn
|
||||
patched = True
|
||||
if not patched: # fail loud if CDK changes the generated shape on upgrade
|
||||
raise RuntimeError(
|
||||
"rotation invoke CfnPermission not found; cannot scope source conditions"
|
||||
)
|
||||
|
||||
|
||||
def _add_shoc_webhook_emitter(stack, work_orders_table, comments_table, alarm_topic):
|
||||
"""SHOC webhook emitter (docs/shoc-webhook-plan.md Phases 1-4).
|
||||
|
||||
Dedicated HMAC CMK + secret with cross-account SHOC read grants, the
|
||||
30-day rotation Lambda, the two failure queues, the stream-driven emitter
|
||||
Lambda with its two (dark, enabled=False) event source mappings, and the
|
||||
full alarm set. `stack` is the construct scope for every child, exactly as
|
||||
if this code were inline in __init__.
|
||||
"""
|
||||
# SHOC consumer principal for the cross-account read grants. EXACT role
|
||||
# ARN only -- future shoc-backend-staging/-prod roles are each a
|
||||
# deliberate, individually-reviewed policy addition (no wildcard or
|
||||
# prefix trust).
|
||||
shoc_consumer_principal = iam.ArnPrincipal(
|
||||
"arn:aws:iam::396287094661:role/shoc-backend-dev"
|
||||
)
|
||||
|
||||
# --- Dedicated CMK for the HMAC secret ---
|
||||
# Dedicated key, NOT alias/seahaven-dynamodb: reusing the DynamoDB CMK
|
||||
# would hand the SHOC cross-account grant decrypt reach over the PO
|
||||
# table's encryption key -- the dedicated key scopes the grant to
|
||||
# exactly this secret (plan Phase 1).
|
||||
shoc_webhook_key = kms.Key(
|
||||
stack,
|
||||
"ShocWebhookHmacKey",
|
||||
alias="workorder-ingest-shoc-webhook-kms",
|
||||
description=(
|
||||
"Dedicated CMK for the workorder-ingest/shoc-webhook-hmac "
|
||||
"secret (cross-account readable by the SHOC backend)"
|
||||
),
|
||||
enable_key_rotation=True,
|
||||
# GOTCHA: DESTROY is deliberate -- do not "harden" this to RETAIN.
|
||||
# The key protects only machine-generated HMAC material that is
|
||||
# fully regenerable by one rotation, and DESTROY avoids the
|
||||
# fixed-name RETAIN-orphan deadlock on the alias (mirrors the
|
||||
# secret's rationale below).
|
||||
removal_policy=RemovalPolicy.DESTROY,
|
||||
pending_window=Duration.days(7),
|
||||
)
|
||||
# Key-policy half of the cross-account read grant (the secret resource
|
||||
# policy below is the other half; either one alone fails silently at
|
||||
# the receiver). resources=["*"] is key-scoped, not account-wide --
|
||||
# KMS key policies only ever apply to this key.
|
||||
# kms:ViaService pins the grant to Secrets Manager decrypt paths only
|
||||
# (GPT-4.1 cross-review FIX): a compromised shoc-backend-dev cannot use
|
||||
# this key for arbitrary KMS operations outside the secret fetch.
|
||||
shoc_webhook_key.add_to_resource_policy(
|
||||
iam.PolicyStatement(
|
||||
actions=["kms:Decrypt"],
|
||||
principals=[shoc_consumer_principal],
|
||||
resources=["*"],
|
||||
conditions={
|
||||
"StringEquals": {
|
||||
"kms:ViaService": (f"secretsmanager.{stack.region}.amazonaws.com")
|
||||
}
|
||||
},
|
||||
)
|
||||
)
|
||||
|
||||
# --- HMAC signing secret ---
|
||||
# Value shape (contract section 6.1):
|
||||
# {"keys": [{"kid": "<YYYY-MM-DDTHH>", "secret": "<64 hex>"}, ...]},
|
||||
# newest first, max 2; the producer signs with keys[0]. The
|
||||
# generate_secret_string below is BOOTSTRAP shape only ({"keys": []}
|
||||
# plus throwaway entropy the rotator ignores); the first rotation
|
||||
# (rotate_immediately default) populates the real keys.
|
||||
shoc_hmac_secret = secretsmanager.Secret(
|
||||
stack,
|
||||
"ShocWebhookHmacSecret",
|
||||
secret_name="workorder-ingest/shoc-webhook-hmac",
|
||||
encryption_key=shoc_webhook_key,
|
||||
description=(
|
||||
"HMAC signing keys for the SHOC work-order webhook "
|
||||
"(docs/shoc-webhook-contract.md section 6)"
|
||||
),
|
||||
# GOTCHA: DESTROY is deliberate -- do not "harden" this to RETAIN.
|
||||
# The value is machine-generated HMAC material with no operator-set
|
||||
# content, fully regenerable by one rotation, so RETAIN buys
|
||||
# nothing and would expose the fixed-name RETAIN orphan deadlock
|
||||
# (a failed first create orphans an empty shell holding the global
|
||||
# name; see reference_secret_retain_orphan_deadlock).
|
||||
removal_policy=RemovalPolicy.DESTROY,
|
||||
generate_secret_string=secretsmanager.SecretStringGenerator(
|
||||
secret_string_template='{"keys": []}',
|
||||
generate_string_key="bootstrap_entropy",
|
||||
password_length=32,
|
||||
exclude_punctuation=True,
|
||||
),
|
||||
)
|
||||
cdk.Tags.of(shoc_hmac_secret).add("Purpose", "shoc-webhook-hmac")
|
||||
cdk.Tags.of(shoc_hmac_secret).add("ManagedBy", "procurement-ingest-cdk")
|
||||
# Secret-resource-policy half of the cross-account read grant (the key
|
||||
# policy above is the other half). DescribeSecret lets the receiver
|
||||
# resolve secret metadata without any broader list permission.
|
||||
shoc_hmac_secret.add_to_resource_policy(
|
||||
iam.PolicyStatement(
|
||||
actions=[
|
||||
"secretsmanager:GetSecretValue",
|
||||
"secretsmanager:DescribeSecret",
|
||||
],
|
||||
principals=[shoc_consumer_principal],
|
||||
resources=["*"],
|
||||
)
|
||||
)
|
||||
|
||||
# --- HMAC rotation Lambda ---
|
||||
# 30-day schedule: generates a new key, prepends as keys[0], truncates
|
||||
# to 2 entries. Single-user rotation (receivers re-fetch on a <=5-min
|
||||
# TTL), so the standard 4-step rotation collapses to
|
||||
# createSecret/finishSecret.
|
||||
shoc_hmac_rotator_log_group = common.make_function_log_group(
|
||||
stack, "ShocHmacRotator", "workorder-shoc-hmac-rotator"
|
||||
)
|
||||
shoc_hmac_rotator = lambda_.Function(
|
||||
stack,
|
||||
"ShocHmacRotator",
|
||||
function_name="workorder-shoc-hmac-rotator",
|
||||
runtime=lambda_.Runtime.PYTHON_3_12,
|
||||
architecture=lambda_.Architecture.ARM_64,
|
||||
handler="handler.handler",
|
||||
code=lambda_.Code.from_asset(
|
||||
"../lambdas",
|
||||
exclude=["**/__pycache__/**", "**/tests/**", "**/package/**"],
|
||||
bundling=cdk.BundlingOptions(
|
||||
image=lambda_.Runtime.PYTHON_3_12.bundling_image,
|
||||
command=[
|
||||
"bash",
|
||||
"-c",
|
||||
# Stdlib + boto3-from-runtime only; nothing installed.
|
||||
"cp wo/shoc_hmac_rotator/*.py /asset-output/",
|
||||
],
|
||||
),
|
||||
),
|
||||
timeout=Duration.seconds(60),
|
||||
memory_size=128,
|
||||
log_group=shoc_hmac_rotator_log_group,
|
||||
)
|
||||
# Rotation permissions, scoped to the one secret. CDK's secret_arn
|
||||
# token resolves to the full ARN including the -?????? suffix wildcard,
|
||||
# so no separate "*"-suffixed resource variant is needed.
|
||||
shoc_hmac_rotator.add_to_role_policy(
|
||||
iam.PolicyStatement(
|
||||
actions=[
|
||||
"secretsmanager:DescribeSecret",
|
||||
"secretsmanager:GetSecretValue",
|
||||
"secretsmanager:PutSecretValue",
|
||||
"secretsmanager:UpdateSecretVersionStage",
|
||||
],
|
||||
resources=[shoc_hmac_secret.secret_arn],
|
||||
)
|
||||
)
|
||||
# Explicit statement instead of grant_encrypt_decrypt so the grant can
|
||||
# carry kms:ViaService (cross-review FIX): the rotator only ever touches
|
||||
# this key through Secrets Manager put/get, never the KMS API directly.
|
||||
shoc_hmac_rotator.add_to_role_policy(
|
||||
iam.PolicyStatement(
|
||||
actions=[
|
||||
"kms:Decrypt",
|
||||
"kms:Encrypt",
|
||||
"kms:GenerateDataKey*",
|
||||
"kms:ReEncrypt*",
|
||||
],
|
||||
resources=[shoc_webhook_key.key_arn],
|
||||
conditions={
|
||||
"StringEquals": {
|
||||
"kms:ViaService": (f"secretsmanager.{stack.region}.amazonaws.com")
|
||||
}
|
||||
},
|
||||
)
|
||||
)
|
||||
shoc_hmac_secret.add_rotation_schedule(
|
||||
"Rotation",
|
||||
rotation_lambda=shoc_hmac_rotator,
|
||||
automatically_after=Duration.days(30),
|
||||
)
|
||||
# Scope the Secrets-Manager-service invoke permission to THIS secret
|
||||
# (cross-review FIX / confused-deputy): add_rotation_schedule emits an
|
||||
# AWS::Lambda::Permission for secretsmanager.amazonaws.com with no
|
||||
# SourceAccount/SourceArn, so any account's Secrets Manager could invoke
|
||||
# the rotator by pointing a foreign secret's RotationLambdaARN at it.
|
||||
# Lambda permissions are additive (OR), so a second scoped permission
|
||||
# would NOT revoke the unscoped one -- patch the generated permission in
|
||||
# place. source_arn pins the invoker to this secret; source_account is the
|
||||
# belt-and-braces account bound. (Blast radius was already contained by
|
||||
# the rotator role being resource-scoped to this secret, but this closes
|
||||
# the unauthenticated invoke primitive per AWS rotation guidance.)
|
||||
_scope_rotation_invoke_permission(stack, shoc_hmac_rotator, shoc_hmac_secret)
|
||||
|
||||
# --- Standard per-Lambda alarms: workorder-shoc-hmac-rotator ---
|
||||
# errors + throttles + p99 duration. No DLQ alarm: rotation is invoked
|
||||
# synchronously by Secrets Manager (dlq=None); a failed rotation
|
||||
# surfaces as an invocation error.
|
||||
common.add_standard_lambda_alarms(
|
||||
stack,
|
||||
"ShocHmacRotator",
|
||||
shoc_hmac_rotator,
|
||||
"workorder-shoc-hmac-rotator",
|
||||
alarm_topic,
|
||||
duration_statistic="p99",
|
||||
errors=True,
|
||||
dlq=None,
|
||||
descriptions={
|
||||
"errors": "workorder-shoc-hmac-rotator invocation errors",
|
||||
"throttles": "workorder-shoc-hmac-rotator invocation throttles",
|
||||
"duration": (
|
||||
"workorder-shoc-hmac-rotator p99 duration approaching the 60s timeout"
|
||||
),
|
||||
},
|
||||
)
|
||||
|
||||
# --- Emitter failure queues ---
|
||||
# Failures queue: ESM on_failure destination. It receives ESM failure
|
||||
# METADATA (shard/sequence pointers), not full payloads -- replay
|
||||
# rebuilds events from DynamoDB (contract section 8).
|
||||
shoc_emitter_failures_queue = sqs.Queue(
|
||||
stack,
|
||||
"ShocEmitterFailuresQueue",
|
||||
queue_name="workorder-shoc-emitter-failures",
|
||||
retention_period=Duration.days(14),
|
||||
enforce_ssl=True,
|
||||
)
|
||||
# Rejected queue: full {envelope, response_status} payloads parked by
|
||||
# the handler on non-retryable 4xx responses (contract section 7).
|
||||
shoc_emitter_rejected_queue = sqs.Queue(
|
||||
stack,
|
||||
"ShocEmitterRejectedQueue",
|
||||
queue_name="workorder-shoc-emitter-rejected",
|
||||
retention_period=Duration.days(14),
|
||||
enforce_ssl=True,
|
||||
)
|
||||
|
||||
# --- Emitter Lambda ---
|
||||
shoc_emitter_log_group = common.make_function_log_group(
|
||||
stack, "ShocEmitter", "workorder-shoc-emitter"
|
||||
)
|
||||
shoc_emitter = lambda_.Function(
|
||||
stack,
|
||||
"ShocEmitter",
|
||||
function_name="workorder-shoc-emitter",
|
||||
runtime=lambda_.Runtime.PYTHON_3_12,
|
||||
architecture=lambda_.Architecture.ARM_64,
|
||||
handler="handler.handler",
|
||||
code=lambda_.Code.from_asset(
|
||||
"../lambdas",
|
||||
exclude=["**/__pycache__/**", "**/tests/**", "**/package/**"],
|
||||
bundling=cdk.BundlingOptions(
|
||||
image=lambda_.Runtime.PYTHON_3_12.bundling_image,
|
||||
command=[
|
||||
"bash",
|
||||
"-c",
|
||||
# Stdlib HTTP (urllib.request) + boto3-from-runtime
|
||||
# only; nothing installed.
|
||||
"cp wo/shoc_emitter/*.py /asset-output/",
|
||||
],
|
||||
),
|
||||
),
|
||||
timeout=Duration.seconds(60),
|
||||
memory_size=256,
|
||||
log_group=shoc_emitter_log_group,
|
||||
environment={
|
||||
# Non-sensitive endpoint URL (HMAC is the auth, the URL is
|
||||
# not). path TBD by SHOC -- confirmed in the activation PR.
|
||||
"SHOC_WEBHOOK_URL": (
|
||||
"https://api.dev.seahaven.com/api/webhooks/work-orders"
|
||||
),
|
||||
"HMAC_SECRET_ARN": shoc_hmac_secret.secret_arn,
|
||||
"REJECTED_QUEUE_URL": shoc_emitter_rejected_queue.queue_url,
|
||||
},
|
||||
)
|
||||
|
||||
work_orders_table.grant_stream_read(shoc_emitter)
|
||||
comments_table.grant_stream_read(shoc_emitter)
|
||||
shoc_hmac_secret.grant_read(shoc_emitter)
|
||||
# Explicit statement instead of grant_decrypt so the grant carries
|
||||
# kms:ViaService (cross-review FIX): the emitter only decrypts this key
|
||||
# through Secrets Manager GetSecretValue.
|
||||
shoc_emitter.add_to_role_policy(
|
||||
iam.PolicyStatement(
|
||||
actions=["kms:Decrypt"],
|
||||
resources=[shoc_webhook_key.key_arn],
|
||||
conditions={
|
||||
"StringEquals": {
|
||||
"kms:ViaService": (f"secretsmanager.{stack.region}.amazonaws.com")
|
||||
}
|
||||
},
|
||||
)
|
||||
)
|
||||
shoc_emitter_rejected_queue.grant_send_messages(shoc_emitter)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# SHIPS DARK: both event source mappings deploy with enabled=False,
|
||||
# deliberately. The full stack (Lambda, queues, alarms, secret,
|
||||
# rotation) deploys and is testable with zero deliveries while SHOC
|
||||
# has no receiver, so our merge cadence never depends on Luby's.
|
||||
# Activation is a one-line enabled=True PR gated on the SHOC receiver
|
||||
# passing the shared HMAC test vectors (plan Phase 3). LATEST start
|
||||
# position => no historical flood at activation; SHOC loads history
|
||||
# via the procurement read API instead.
|
||||
# Ordering knobs: parallelization_factor=1, bisect_batch_on_error=
|
||||
# False and retry_attempts=-1 (retry until the 24h record age) are
|
||||
# REQUIRED for strict per-work-order in-order delivery -- a retryable
|
||||
# failure blocks the shard rather than skipping ahead, and
|
||||
# report_batch_item_failures keeps earlier in-batch successes from
|
||||
# being re-delivered.
|
||||
# ------------------------------------------------------------------
|
||||
shoc_emitter.add_event_source(
|
||||
lambda_event_sources.DynamoEventSource(
|
||||
work_orders_table,
|
||||
starting_position=lambda_.StartingPosition.LATEST,
|
||||
batch_size=10,
|
||||
bisect_batch_on_error=False,
|
||||
retry_attempts=-1,
|
||||
max_record_age=Duration.hours(24),
|
||||
parallelization_factor=1,
|
||||
report_batch_item_failures=True,
|
||||
enabled=False,
|
||||
on_failure=lambda_event_sources.SqsDlq(shoc_emitter_failures_queue),
|
||||
)
|
||||
)
|
||||
shoc_emitter.add_event_source(
|
||||
lambda_event_sources.DynamoEventSource(
|
||||
comments_table,
|
||||
starting_position=lambda_.StartingPosition.LATEST,
|
||||
batch_size=10,
|
||||
bisect_batch_on_error=False,
|
||||
retry_attempts=-1,
|
||||
max_record_age=Duration.hours(24),
|
||||
parallelization_factor=1,
|
||||
report_batch_item_failures=True,
|
||||
enabled=False,
|
||||
on_failure=lambda_event_sources.SqsDlq(shoc_emitter_failures_queue),
|
||||
)
|
||||
)
|
||||
|
||||
# --- Standard per-Lambda alarms: workorder-shoc-emitter ---
|
||||
# errors + throttles + p99 duration. No DLQ alarm here: the emitter is
|
||||
# a stream consumer with no async DLQ (dlq=None); its failure surfaces
|
||||
# are the two SQS queues alarmed bespoke below.
|
||||
common.add_standard_lambda_alarms(
|
||||
stack,
|
||||
"ShocEmitter",
|
||||
shoc_emitter,
|
||||
"workorder-shoc-emitter",
|
||||
alarm_topic,
|
||||
duration_statistic="p99",
|
||||
errors=True,
|
||||
dlq=None,
|
||||
descriptions={
|
||||
"errors": "workorder-shoc-emitter invocation errors",
|
||||
"throttles": "workorder-shoc-emitter invocation throttles",
|
||||
"duration": (
|
||||
"workorder-shoc-emitter p99 duration approaching the 60s timeout"
|
||||
),
|
||||
},
|
||||
)
|
||||
|
||||
# --- Bespoke emitter alarms (plan Phase 4) ---
|
||||
# These don't fit add_standard_lambda_alarms' shape and stay bespoke.
|
||||
# Iterator age >= 10 min sustained means SHOC is likely down and the
|
||||
# shard is blocking (exactly the ordered-backpressure design working);
|
||||
# the two SQS-visible alarms page the operator replay runbook.
|
||||
shoc_emitter.metric(
|
||||
"IteratorAge",
|
||||
statistic="Maximum",
|
||||
period=Duration.minutes(5),
|
||||
).create_alarm(
|
||||
stack,
|
||||
"ShocEmitterIteratorAgeAlarm",
|
||||
alarm_name="workorder-shoc-emitter-iterator-age",
|
||||
alarm_description=(
|
||||
"workorder-shoc-emitter stream lag >= 10 min "
|
||||
"(SHOC receiver likely down; shard blocking on retries)"
|
||||
),
|
||||
threshold=600000,
|
||||
evaluation_periods=3,
|
||||
datapoints_to_alarm=2,
|
||||
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_OR_EQUAL_TO_THRESHOLD,
|
||||
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
|
||||
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
|
||||
|
||||
shoc_emitter_failures_queue.metric_approximate_number_of_messages_visible(
|
||||
period=Duration.minutes(5),
|
||||
statistic="Maximum",
|
||||
).create_alarm(
|
||||
stack,
|
||||
"ShocEmitterFailuresMessagesAlarm",
|
||||
alarm_name="workorder-shoc-emitter-failures-messages",
|
||||
alarm_description=(
|
||||
"workorder-shoc-emitter retry-exhausted stream records parked "
|
||||
"(ESM failure metadata; replay rebuilds from DynamoDB)"
|
||||
),
|
||||
threshold=0,
|
||||
evaluation_periods=1,
|
||||
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
|
||||
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
|
||||
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
|
||||
|
||||
shoc_emitter_rejected_queue.metric_approximate_number_of_messages_visible(
|
||||
period=Duration.minutes(5),
|
||||
statistic="Maximum",
|
||||
).create_alarm(
|
||||
stack,
|
||||
"ShocEmitterRejectedMessagesAlarm",
|
||||
alarm_name="workorder-shoc-emitter-rejected-messages",
|
||||
alarm_description=(
|
||||
"workorder-shoc-emitter parked non-retryable 4xx deliveries "
|
||||
"(contract bug; inspect payloads and replay)"
|
||||
),
|
||||
threshold=0,
|
||||
evaluation_periods=1,
|
||||
comparison_operator=cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
|
||||
treat_missing_data=cloudwatch.TreatMissingData.NOT_BREACHING,
|
||||
).add_alarm_action(cw_actions.SnsAction(alarm_topic))
|
||||
|
||||
cdk.CfnOutput(
|
||||
stack,
|
||||
"ShocWebhookHmacSecretArn",
|
||||
value=shoc_hmac_secret.secret_arn,
|
||||
description=(
|
||||
"SHOC webhook HMAC secret ARN -- hand off to Luby (SHOC team) "
|
||||
"for the cross-account receiver fetch"
|
||||
),
|
||||
)
|
||||
|
|
|
|||
|
|
@ -1,167 +0,0 @@
|
|||
# SHOC Work-Order Webhook — Delivery Contract (v1)
|
||||
|
||||
**Status:** DRAFT — for SHOC team (Luby) review. **Rev 2026-07-23** (supersedes the 2026-07-16 draft: producer account corrected to seahaven-prod; reconciliation backstop changed to the new read API; §4.1 `unknown` status note; §3 `write_origin` forward-compat note. Sections 2, 5, 6, 7, and 10 are unchanged from the 07-16 draft).
|
||||
**Producer:** `workorder-shoc-emitter` Lambda, Sea Haven **seahaven-prod** AWS account (**011934824531**, us-east-1). The former management account (328440206208) is frozen/rollback-only and will never host this feed, its secret, or its streams.
|
||||
**Consumer:** SHOC backend (.NET 8), initially `https://api.dev.seahaven.com`. Endpoint path is SHOC's choice — suggested `POST /api/webhooks/work-orders`; map entities onto the work-orders domain from shoc-backend PR #10.
|
||||
|
||||
## 1. Overview
|
||||
|
||||
Every work-order mutation the procurement-ingest pipeline writes to DynamoDB is pushed to SHOC as an HTTPS POST within seconds. The feed is driven by DynamoDB Streams, so events are emitted **in the exact order the pipeline committed them**, per work order. DynamoDB remains the source of truth; this webhook is a realtime feed. The reconciliation backstop (and the initial-history load — the feed starts at activation time, not from history) is the **procurement read API** (`GET /work-orders`, `GET /work-orders/{id}/comments`, IAM SigV4 — see its OpenAPI doc). SHOC's existing SyncController DynamoDB scan is transitional: it points at the old management account and retires when those stacks are decommissioned. `SyncVendorReplies` should be dropped on the SHOC side — the `VendorReplies` table is dead (its writer was deleted), and no `vendor_reply` webhook event exists or is planned.
|
||||
|
||||
## 2. Transport
|
||||
|
||||
- HTTPS POST, `Content-Type: application/json; charset=utf-8`, body ≤ 256 KB.
|
||||
- Producer timeout is **10 seconds**. Respond `2xx` as fast as possible; if your processing is slow, accept-and-enqueue internally rather than processing inline.
|
||||
|
||||
## 3. Event envelope
|
||||
|
||||
```json
|
||||
{
|
||||
"schema_version": 1,
|
||||
"delivery_id": "4b7c2f0e-...",
|
||||
"event_type": "work_order.updated",
|
||||
"occurred_at": "2026-07-16T14:03:22.114208+00:00",
|
||||
"source": "procurement-ingest/workorder-shoc-emitter",
|
||||
"replay": false,
|
||||
"data": { ... }
|
||||
}
|
||||
```
|
||||
|
||||
| Field | Meaning |
|
||||
|---|---|
|
||||
| `schema_version` | Integer. Breaking payload changes increment it; SHOC should reject versions it doesn't know. |
|
||||
| `delivery_id` | Unique per source event, **stable across producer retries** — your idempotency key. Persist it; ignore any delivery whose id you've already processed. |
|
||||
| `event_type` | See §4. |
|
||||
| `occurred_at` | ISO 8601 **with UTC offset** (`+00:00`), when the pipeline committed the write. |
|
||||
| `replay` | `true` when re-sent by the operator replay tool after an outage. Same idempotency rules apply. |
|
||||
| `data` | Event-type-specific body, §4. |
|
||||
|
||||
> **Forward-compat (phase-2 write-back):** when SHOC later gains write endpoints on the read API, records SHOC itself wrote will carry a `write_origin` attribute (e.g. `"shoc-write-api"`), and the emitter will skip them so SHOC never receives an echo of its own write. Nothing to build now — just don't reject envelopes if a `data.write_origin` field appears later.
|
||||
|
||||
## 4. Event types
|
||||
|
||||
### 4.1 `work_order.created` / `work_order.updated` / `work_order.cancelled`
|
||||
|
||||
Emitted from `WorkOrders` table writes: `created` on first insert, `updated` on any subsequent change, `cancelled` when `wo_status` transitions to `cancelled` (a `cancelled` event is a specialization of `updated` — same body). `data` is the **full current work-order state** (not a diff); fields absent from the source email are `null`:
|
||||
|
||||
```json
|
||||
{
|
||||
"work_order_id": "11144580730",
|
||||
"wo_status": "assigned",
|
||||
"description": "Dock door 14 won't close",
|
||||
"customer": "AMAZON",
|
||||
"site_code": "JFK8",
|
||||
"building": "JFK8",
|
||||
"address": "546 Gulf Ave, Staten Island, NY 10314",
|
||||
"severity": "3-Normal",
|
||||
"priority": "Medium",
|
||||
"assigned_to": "Sea Haven Industries",
|
||||
"date_reported": "2026-07-14T09:12:00",
|
||||
"scheduled_start": "2026-07-17T08:00:00",
|
||||
"due_date": "2026-07-21T17:00:00",
|
||||
"record_type": "new_work_order",
|
||||
"created_at": "2026-07-16T14:03:22.114208+00:00",
|
||||
"updated_at": "2026-07-16T14:03:22.114208+00:00"
|
||||
}
|
||||
```
|
||||
|
||||
`wo_status` ∈ `new | assigned | in_progress | on_hold | completed | cancelled | unknown`. `record_type` (the triggering email's type) ∈ `new_work_order | update | comment | cancellation`.
|
||||
|
||||
> **`unknown` is a real value** the extractor emits when the source email doesn't state a status — SHOC must define an explicit mapping for it (and defensively for any unrecognized future value) rather than letting it fall through a switch. Field names and enums in this section mirror the producer's write path, `lambdas/wo/email_processor/persistence.py` (+ `prompts.py` for enums) on `main` — that code is the authoritative source if this doc ever drifts.
|
||||
|
||||
### 4.2 `work_order.comment_added`
|
||||
|
||||
Emitted from `WorkOrderComments` inserts — one per source email (comments, but also update/cancellation event records):
|
||||
|
||||
```json
|
||||
{
|
||||
"work_order_id": "11144580730",
|
||||
"comment_id": "11144580730#2026-04-27T23:51:48#a1b2c3d4e5f6",
|
||||
"record_type": "comment",
|
||||
"commenter": "APM Technician",
|
||||
"text": "Vendor dispatched, ETA tomorrow AM.",
|
||||
"created_at": "2026-04-27T23:51:48",
|
||||
"ingested_at": "2026-07-16T14:03:22.114208+00:00"
|
||||
}
|
||||
```
|
||||
|
||||
`comment_id` is unique per source email and stable across retries — use it (or `delivery_id`) for dedupe.
|
||||
|
||||
## 5. Ordering and delivery semantics
|
||||
|
||||
- **At-least-once.** Duplicates are possible on retry; dedupe on `delivery_id`.
|
||||
- **Per-work-order, per-event-family ordering is guaranteed on the live feed:** all `work_order.*` state events for a given `work_order_id` arrive in commit order (a `cancelled` never precedes its `created`). Comment events are likewise ordered among themselves per work order. **Caveat:** this holds for delivered events. A rare non-retryable `4xx` on one event (a contract bug) parks *that* event and lets later events proceed, so the receiver can in principle see a `work_order.updated`/`cancelled` for a WO whose `created` was parked. Because every `work_order.*` body is **full current state** (not a diff), the two requirements below make this safe regardless.
|
||||
- **Requirement — skeleton-upsert on ANY unknown `work_order_id`:** on any `work_order.*` event (state or comment) referencing a `work_order_id` you have not seen, upsert the record from the event's own `data` (or a skeleton for a bare comment) rather than dropping it. Full-state bodies mean a later event fully reconstructs a WO whose `created` never arrived — the same out-of-order semantics the pipeline itself uses for source emails.
|
||||
- **Requirement — monotonicity (do not regress state):** ignore any `work_order.*` state event whose `data.updated_at` is **older** than the `updated_at` already stored for that WO. This makes an operator replay (see §8) or a rare out-of-order delivery idempotent and non-regressing: a stale snapshot can never overwrite newer state. Comments are append-only (dedupe on `comment_id`), so this applies only to WO state events.
|
||||
- **Cross-family ordering is NOT guaranteed:** a `comment_added` may occasionally arrive before the `created` for its work order (separate streams) — covered by the skeleton-upsert requirement above.
|
||||
- Expected volume ≈ 22,900 events/month (~760/day); >90% are comment events. Bursts of a few events/second are possible.
|
||||
|
||||
## 6. Authentication — HMAC signature
|
||||
|
||||
Every request carries:
|
||||
|
||||
```
|
||||
X-SH-Timestamp: 1784642602 (unix seconds, producer clock)
|
||||
X-SH-Key-Id: 2026-07-20T00
|
||||
X-SH-Signature: v1=hex(HMAC_SHA256(secret, "{timestamp}.{raw_body}"))
|
||||
```
|
||||
|
||||
Verification requirements (fail closed — reject with `401` on any failure):
|
||||
|
||||
1. Reject if `|now - X-SH-Timestamp| > 300s` (replay window).
|
||||
2. Look up the secret for `X-SH-Key-Id` from the shared secret material (§6.1). Reject unknown key ids.
|
||||
3. Recompute `HMAC-SHA256(secret, timestamp + "." + raw_body)` over the **raw request bytes** (before any JSON parsing/re-serialization) and compare **constant-time** against the `v1=` value.
|
||||
|
||||
### 6.1 Key material and rotation
|
||||
|
||||
The secret lives in **seahaven-prod (011934824531)** Secrets Manager: **`workorder-ingest/shoc-webhook-hmac`**, encrypted with a dedicated customer-managed KMS key so it is readable cross-account. The SHOC backend role (`arn:aws:iam::396287094661:role/shoc-backend-dev`) is granted `secretsmanager:GetSecretValue` + `kms:Decrypt` via resource policies — exact role ARN only; future `shoc-backend-staging`/`-prod` roles are each a deliberate, individually-reviewed policy addition (no wildcard/prefix trust). Secret value shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"keys": [
|
||||
{ "kid": "2026-07-20T00", "secret": "<64 hex chars>" },
|
||||
{ "kid": "2026-06-20T00", "secret": "<64 hex chars>" }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
- Rotation is automated (30-day schedule): a new key is prepended, the previous key is kept for one overlap cycle.
|
||||
- The producer always signs with `keys[0]`. The receiver must accept **any** listed `kid`.
|
||||
- **Receiver must re-fetch the secret at most every 5 minutes** (cache TTL ≤ 300s) so rotation propagates with zero downtime. Also re-fetch immediately on an unknown-`kid` before rejecting.
|
||||
|
||||
## 7. Response contract
|
||||
|
||||
| Receiver response | Producer behavior |
|
||||
|---|---|
|
||||
| `2xx` | Delivered. Body ignored. |
|
||||
| `429`, `5xx`, timeout, connection error | Retried with in-order blocking (that work order's events queue behind it) for up to **24 hours**; then parked for operator replay. |
|
||||
| Any other `4xx` | Not retried — parked immediately and alarmed on our side. A `4xx` means a contract bug; expect us to call you. |
|
||||
|
||||
## 8. Backstop / replay
|
||||
|
||||
If SHOC is down longer than the retry window or deliveries are parked, an operator replay tool re-sends events rebuilt from DynamoDB (marked `"replay": true`). Because DynamoDB stays the source of truth, the procurement read API can also fully reconcile at any time (paginated `GET /work-orders` + per-WO comments). Missed webhooks are therefore an availability inconvenience, never data loss.
|
||||
|
||||
## 9. SHOC-side checklist
|
||||
|
||||
- [ ] Endpoint path confirmed + implemented against PR #10 entity model
|
||||
- [ ] Raw-body HMAC verification, constant-time compare, ±300s timestamp window, fail-closed
|
||||
- [ ] Cross-account secret fetch with ≤5-min cache + refresh-on-unknown-kid
|
||||
- [ ] Dedupe store on `delivery_id` (and/or `comment_id`)
|
||||
- [ ] Skeleton-upsert on ANY unknown `work_order_id` (state events too, not just comment-before-create)
|
||||
- [ ] Monotonicity guard: ignore WO state events whose `data.updated_at` is older than stored (§5)
|
||||
- [ ] Fast `2xx` ack (<10s), internal queue if processing is slow
|
||||
- [ ] Timestamp parsing accepts `+00:00` offsets
|
||||
- [ ] Reject unknown `schema_version`
|
||||
- [ ] Explicit mapping for `wo_status: unknown` (and a safe default for unrecognized future values)
|
||||
- [ ] Initial history loaded via the read API before relying on the feed (feed starts at activation, not from history)
|
||||
- [ ] `SyncVendorReplies` dropped (dead table; no vendor-reply event exists)
|
||||
|
||||
## 10. Environments
|
||||
|
||||
| Env | URL | Status |
|
||||
|---|---|---|
|
||||
| dev | `https://api.dev.seahaven.com/<path TBD>` | first target (prod data in dev accepted 2026-07-16) |
|
||||
| staging | `https://api.staging.seahaven.com/<path TBD>` | when staging deploys |
|
||||
| prod | TBD | when SHOC prod exists |
|
||||
|
||||
The target URL is producer-side config (per-env), so promotion is a config change, not a code change.
|
||||
|
|
@ -1,108 +0,0 @@
|
|||
# Implementation Plan — SHOC Realtime Work-Order Webhook
|
||||
|
||||
**Branch:** `feat/shoc-wo-webhook` (re-cut on `main` 2026-07-23; all build work targets the post-refactor layout — `cdk/common.py` helpers, explicit LogGroups, decomposed handlers, consolidated test roots — not the pre-refactor style earlier drafts implied).
|
||||
**Companion docs:** `docs/shoc-webhook-contract.md` (Rev 2026-07-23 — the producer/consumer contract; Luby builds the receiver against it) and the procurement read API OpenAPI spec (`lambdas/api/openapi.json`, built in the sibling `procurement-api` effort — the reconciliation/backfill path).
|
||||
**Requestor:** Luby (SHOC team). Decisions captured 2026-07-16: all WO event types, seconds-latency, full-payload push, HMAC + automated rotation, in-order per WO, prod-data-in-dev accepted, dual-write retained (DynamoDB retirement deferred for scope discipline — the original blocker, seahaven-slack-bot, was decommissioned 2026-07-23; retirement is now sequenced after SHOC prod + mgmt decommission, not this PR).
|
||||
**Account (2026-07-23):** everything here builds in **seahaven-prod (011934824531)**. The old management account (328440206208) is frozen/rollback-only.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
SES → S3 → workorder-email-processor ──writes──▶ WorkOrders ─stream─▶
|
||||
└─writes──▶ WorkOrderComments ─stream─▶ workorder-shoc-emitter ──HMAC POST──▶ SHOC API
|
||||
│ (retry-exhausted / non-retryable)
|
||||
▼
|
||||
shoc-emitter failure queues → alarms → replay script
|
||||
```
|
||||
|
||||
Rationale (vs. inline call from the processor): DynamoDB Streams gives per-partition-key (per-`work_order_id`) commit-order delivery with shard-blocking retries — the ordering guarantee Luby requires — and keeps the email-ingestion hot path completely decoupled from SHOC availability. No change to PR #99's handler.
|
||||
|
||||
## Phase 0 — Prerequisites (blocking)
|
||||
|
||||
1. Branch re-cut on `main` (done 2026-07-23).
|
||||
2. Luby signs off `docs/shoc-webhook-contract.md` Rev 2026-07-23 (endpoint path, PR #10 entity mapping, the `unknown`-status mapping, `SyncVendorReplies` retirement).
|
||||
3. Confirm the SHOC receiver role ARN for the resource policies (`arn:aws:iam::396287094661:role/shoc-backend-dev`) — the role lives in SHOC's account, outside this repo's control; confirm it still exists before the CDK references it (the first cross-account `get-secret-value` test is the hard proof).
|
||||
4. **⚠️ ACCOUNT GATE: every deploy in this plan targets seahaven-prod (011934824531) ONLY.** Never deploy this stack to mgmt (328440206208), and never "temporarily" enable streams on mgmt's copies of the WO tables — that is the only path by which the SES dual-delivery bake could ever double-send to SHOC. Verify `aws sts get-caller-identity` resolves to 011934824531 before `cdk deploy`.
|
||||
5. Confirm with Luby which SHOC environment should receive the feed now — "prod-data-in-dev" was accepted 2026-07-16, pre-migration; re-confirm the target URL.
|
||||
|
||||
## Phase 1 — Secret + rotation (CDK, `wo_stack.py`)
|
||||
|
||||
This phase is an *incremental* deploy onto the already-migrated `workorder-ingest` stack in seahaven-prod — it is not part of the account-migration work.
|
||||
|
||||
- New customer-managed KMS key `workorder-ingest-shoc-webhook-kms` (kebab-case per naming-conventions.md). Required because the default `aws/secretsmanager` key cannot serve cross-account reads. **Deliberately a dedicated key, not `alias/seahaven-dynamodb`:** reusing the DynamoDB CMK would hand the SHOC cross-account grant decrypt reach over the PO table's encryption key — the dedicated key scopes the grant to exactly this secret.
|
||||
- New Secrets Manager secret **`workorder-ingest/shoc-webhook-hmac`** (naming per secrets-and-config.md `stack-name/value-name`), CDK-managed, encrypted with the CMK. Value shape: `{"keys": [{"kid", "secret"}, ...]}` (dual-key, contract §6.1).
|
||||
- **RemovalPolicy: `DESTROY`, deliberately.** The value is machine-generated HMAC material with no operator-set content — fully regenerable by one rotation — so `RETAIN` buys nothing and would expose the fixed-name RETAIN orphan deadlock (failed first create orphans an empty shell holding the global name → every later create fails `AlreadyExists`; see `reference_secret_retain_orphan_deadlock`). Add a GOTCHA comment at the removal-policy line so nobody "hardens" it to RETAIN later. If a first deploy does fail, check for and force-delete any **empty** orphaned shell before retrying. Accidental-deletion recovery: recreate + rotate (new keys), notify Luby (receivers refresh within their 5-min TTL), watch the `-failures` queue and replay the gap — documented in the README runbook. Secret gets a `description` and `Purpose`/`ManagedBy` tags for discoverability.
|
||||
- Rotation Lambda `workorder-shoc-hmac-rotator` (Python 3.12, ARM64, explicit LogGroup via `common.make_function_log_group`, alarms via the house `common.add_standard_lambda_alarms` idiom) on a 30-day `rotation_schedule`: generates a new 32-byte key, prepends as `keys[0]`, truncates to 2 entries. Single-user rotation (no external system holds the value — receivers re-fetch), so the standard 4-step rotation collapses to createSecret/finishSecret. Rotation-outage math: producer signs `keys[0]` with a ≤5-min cache; receiver keeps the previous key and refreshes on unknown `kid` (contract §6.1, receiver-side) — no delivery window where signatures can't verify.
|
||||
- Cross-account read grants — **both halves required, to the exact role ARN** (`arn:aws:iam::396287094661:role/shoc-backend-dev`), no wildcards: the secret **resource policy** grants `secretsmanager:GetSecretValue` AND the **KMS key policy** grants `kms:Decrypt`; either one alone fails silently at the receiver. Future staging/prod receiver roles are each an explicit policy addition with its own cross-family review — no prefix/wildcard trust. **IAM/resource-policy change → mandatory GPT-4.1 cross-family review (Phase 6).**
|
||||
- **Post-deploy verification:** assume `shoc-backend-dev` (or have Luby run it) and perform a real cross-account `get-secret-value` before declaring Phase 1 done (synth ≠ authz — the live call is the proof).
|
||||
- **Handoff:** send Luby the secret ARN, region, and the contract Rev pointer as an explicit step — the grant is useless if she doesn't have the ARN.
|
||||
|
||||
## Phase 2 — Streams on the WO tables (CDK)
|
||||
|
||||
- Enable `stream=NEW_AND_OLD_IMAGES` on `WorkOrders` and `WorkOrderComments`. In-place CFN update, no replacement, additive. Consumer audit (refreshed 2026-07-23): **neither table has a stream, so no stream consumers can exist, and the tables have NO cross-stack consumers at all** — seahaven-slack-bot (the former sole reader) was decommissioned 2026-07-23 and its grants dropped from CDK. Re-run this audit before deploy anyway; the reasoning pattern (re-audit before assuming safety) is the point. Note the change in the README data-contract section. **Pre-deploy checks:** (a) confirm the prod tables still carry `RemovalPolicy.RETAIN` and their pre-migration logical IDs; (b) `cdk diff` must show the table changes as in-place updates — if either table shows `[Replacement]` (e.g. from an accidental logical-ID change), abort; these are RETAIN production tables and a replacement would orphan them.
|
||||
- OLD_IMAGE is needed to detect the `wo_status → cancelled` transition (`work_order.cancelled` classification).
|
||||
|
||||
## Phase 3 — Emitter Lambda `workorder-shoc-emitter`
|
||||
|
||||
- Python 3.12, ARM64, explicit LogGroup via `common.make_function_log_group`, kebab-case name, `memory_size=256`, `timeout=60s` (POST timeout 10s per contract §2). Stdlib HTTP (`urllib.request`) — no new deps. Source at `lambdas/wo/shoc_emitter/`.
|
||||
- **Ships DARK:** both event-source mappings deploy with `enabled=False` — the full stack (Lambda, queues, alarms, secret, rotation) deploys and is testable with zero deliveries while SHOC has no receiver, so our merge cadence never depends on Luby's. Activation is a deliberate one-line `enabled=True` PR after her receiver passes the shared HMAC test vectors. While disabled, stream records simply age out unprocessed (24 h retention) — that's fine; SHOC loads history via the read API at activation anyway.
|
||||
- Two event-source mappings (one per stream): `parallelization_factor=1` and `bisect_batch_on_function_error=False` (both required for strict in-order), `batch_size=10`, `maximum_record_age` = 24h (stream retention), `retry_attempts=-1` (retry until expiry — blocking the shard is what preserves order), `on_failure` → SQS destination `workorder-shoc-emitter-failures` (14-day, SSL-enforced, house DLQ style).
|
||||
- Handler logic:
|
||||
1. Deserialize the Dynamo stream image → plain JSON (Decimal-safe).
|
||||
2. Classify: `WorkOrders` INSERT → `work_order.created`; MODIFY → `work_order.updated`, or `work_order.cancelled` when OLD.wo_status ≠ `cancelled` ∧ NEW.wo_status = `cancelled`; `WorkOrderComments` INSERT → `work_order.comment_added`. REMOVE events: skip (no deletes in this pipeline). **Echo guard (forward-compat):** if NEW.`write_origin == "shoc-write-api"`, skip — a no-op branch today (nothing writes that attribute yet) that keeps the phase-2 write-back API from echoing SHOC's own writes back at it without re-touching this loop.
|
||||
3. Build envelope (contract §3); `delivery_id` = the stream record `eventID` (unique, stable across ESM retries); `occurred_at` = `ApproximateCreationDateTime`.
|
||||
4. Sign (contract §6) with `keys[0]` from the secret — cached per handbook Lambda pattern with a **5-minute TTL** so rotation propagates.
|
||||
5. POST with a `User-Agent: workorder-shoc-emitter/1` header; classify response per contract §7: `2xx` → done; `429/5xx/timeout/conn-error` → raise (ESM blocks + retries in order); other `4xx` → send record to `workorder-shoc-emitter-rejected` SQS queue, log structured warning, return success (a contract bug must not block the shard for 24h).
|
||||
6. Structured JSON logging for every delivery attempt/outcome (delivery_id, event_type, work_order_id, status code, latency) — Logs-Insights-queryable, same style as the processor's `sender_auth_rejected` records.
|
||||
- Throughput note (`parallelization_factor=1`): ordering serializes per shard, but volume is ~760 events/day with bursts of a few events/second against a ~10s worst-case POST — a shard-blocking bottleneck only matters when SHOC is degraded, which is exactly when we *want* ordered blocking + the iterator-age alarm rather than parallel hammering. Accepted.
|
||||
- Env config: `SHOC_WEBHOOK_URL` (non-sensitive endpoint URL — env var/CDK context per secrets-and-config.md; HMAC is the auth, the URL is not), `HMAC_SECRET_ARN`.
|
||||
- IAM: stream read (`grant_stream_read` on both tables), `GetSecretValue` scoped to the one secret, `kms:Decrypt` on the CMK, SQS send to the two failure queues. **→ cross-family review.**
|
||||
- CI plumbing (post-refactor, easy to miss): the emitter's and rotator's bundling commands need **new AST-pin coverage in `tests/test_bundle_consistency.py`** (the test currently only recognizes email_processor/web_ui command shapes — an unrecognized bundle ships unchecked); add `lambdas/wo/shoc_emitter` (and the rotator dir) to `pytest.ini`'s explicit `--cov=` list (unlisted dirs are invisible to the 80% floor); touch `tests/support/loader.py` only if a real sibling-name collision exists (per-directory resolution usually isolates it).
|
||||
|
||||
## Phase 4 — Monitoring (house alarm style: ALARM-only → `site-alerts`, `NOT_BREACHING`)
|
||||
|
||||
| Alarm | Metric | Threshold |
|
||||
|---|---|---|
|
||||
| `workorder-shoc-emitter-errors` | Lambda Errors Sum 5 min | > 0, eval 1 |
|
||||
| `workorder-shoc-emitter-throttles` | Lambda Throttles Sum 5 min | > 0, eval 1 |
|
||||
| `workorder-shoc-emitter-iterator-age` | IteratorAge Max 5 min | ≥ 600,000 ms (10 min lag = SHOC likely down), eval 3/2 |
|
||||
| `workorder-shoc-emitter-failures-messages` | SQS visible Max 5 min | > 0, eval 1 |
|
||||
| `workorder-shoc-emitter-rejected-messages` | SQS visible Max 5 min | > 0, eval 1 |
|
||||
|
||||
House rules made explicit: every alarm sets **only** `AlarmActions` (no `OKActions`, no `InsufficientDataActions` — `feedback_cloudwatch_alarms`), and every alarm must actually carry the `site-alerts` action (no action = monitoring theater). Errors/throttles/duration come from `common.add_standard_lambda_alarms`; iterator-age and the two SQS-visible alarms don't fit that helper's shape and stay bespoke — don't misread the helper as covering everything. `site-alerts` exists in seahaven-prod (live-verified 2026-07-23) via `Topic.from_topic_arn`, correctly on the `alias/seahaven-alarm-topics` CMK — never `alias/aws/sns`, whose key policy silently blocks CloudWatch publishes (`feedback_sns_alarm_topic_kms`). Post-deploy, verify delivery on one new alarm with `set-alarm-state` + `describe-alarm-history --history-item-type Action`.
|
||||
|
||||
Optional EMF `DeliveryOutcome` metric (delivered/rejected, mirroring the ParseMethod pattern) for a delivery-rate dashboard — cheap, same zero-latency EMF approach.
|
||||
|
||||
## Phase 5 — Replay tool + tests
|
||||
|
||||
- `scripts/replay_shoc_webhooks.py`: rebuild events from `WorkOrders`/`WorkOrderComments` for a `--work-order-id` list or `--since` window and re-POST with `"replay": true` (contract §8). Dry-run default, house script style. Documented with usage examples in the README Scripts section (operator runbook for the `-failures`/`-rejected` alarms).
|
||||
- Unit tests (offline, no AWS — extend the pytest suite under the consolidated `tests/` conventions): signature generation golden vectors (shareable with Luby so both sides verify against identical vectors); stream-record → envelope mapping incl. Decimal handling and `null` fields; `cancelled` transition classification incl. already-cancelled MODIFYs (no false `cancelled` events); the `write_origin == "shoc-write-api"` skip branch; an enum golden test pinning emitted `wo_status`/`record_type` values to `lambdas/wo/email_processor/prompts.py` (enum drift fails CI here, not in SHOC prod); response-classification matrix (2xx/429/5xx/timeout/4xx); secret-cache TTL + unknown-kid refresh.
|
||||
- Interaction note: PR #99 advisory A1 (AI-fallback `comment_time` nondeterminism) can duplicate comment rows on async retries → duplicate `comment_added` webhooks. SHOC dedupe on `comment_id`/`delivery_id` absorbs it; fixing A1 stays tracked separately.
|
||||
|
||||
## Phase 6 — Gates (before merge, in order)
|
||||
|
||||
1. `ruff check` + `ruff format --check` + `pytest` locally (pre-push hook enforced). Pre-flight: deploy credentials resolve to seahaven-prod (Phase 0 gate 4).
|
||||
2. **GPT-4.1 cross-family review** (`cross_review.py`) of the complete policy surface **as a single diff** — Lambda execution-role policy + KMS key policy + secret resource policy + rotation-Lambda role together, never piecemeal (cross-policy gaps hide between separately-reviewed fragments). Mandatory. All policy changes ship in this one PR so the review sees the complete surface.
|
||||
3. **`/sh-security-review`** — mandatory (auth surface: HMAC design + rotation; untrusted-input: email-derived data serialized into outbound requests; cross-account exposure). Resolve every confirmed critical/high.
|
||||
4. PR against `main`, CI green (`gh pr checks`), no admin-merge.
|
||||
|
||||
## Phase 7 — Docs + memory (same conversation as ship, per global instructions)
|
||||
|
||||
- README: new emitter Lambda, streams, secret + CMK, alarms, data-flow diagram, SHOC consumer contract section, replay-script runbook, the DESTROY-not-RETAIN rationale on the secret — and **remove the stale "Consumer (read-only) — data contract" paragraph naming seahaven-slack-bot** (decommissioned).
|
||||
- Confluence **AWS Architecture Map** (id 1540098): add emitter, streams, the `workorder-ingest/shoc-webhook-hmac` secret + CMK, the two failure SQS queues, and the cross-account SHOC edge (both the webhook POST and the secret-read trust) to the WorkorderIngestStack subgraph. Read the page first — the 2026-07-23 migration already edited it (v35); don't clobber those edits.
|
||||
- Memory: update `project_procurement_ingest.md` and `project_shoc.md` (cross-referenced) — webhook shipped, contract Rev, secret/CMK names + exact cross-account grant ARNs, rotation design, DESTROY-policy rationale, dark-ship/activation state, read API replaces SyncController, echo-guard design.
|
||||
|
||||
## Deploy/cutover order
|
||||
|
||||
1. `cdk deploy` (streams + secret + emitter ship in one deploy, **ESMs disabled** — zero deliveries, zero alarm noise; our merge cadence never depends on SHOC's readiness).
|
||||
2. Luby builds the receiver against the contract + shared test vectors and loads history via the read API; receiver passes the vectors at the target env.
|
||||
3. Activation: one-line `enabled=True` PR. The ESM starts at `LATEST` — no historical flood.
|
||||
4. Verify end-to-end with a live APM email; watch `iterator-age` + `rejected` alarms for 48h.
|
||||
5. SyncController's Dynamo scan (incl. `SyncVendorReplies`) retires with the mgmt decommission; the read API is the standing reconciliation path.
|
||||
|
||||
## Explicit non-goals (this PR)
|
||||
|
||||
- No DynamoDB retirement / write-path switch (kept out for scope discipline — the former blocker, seahaven-slack-bot, is decommissioned; revisit after SHOC prod + mgmt decommission).
|
||||
- No PO-pipeline webhook.
|
||||
- No change to `workorder-email-processor` or its parser.
|
||||
- No write-back endpoints (phase 2 of the read API, own PR + own IAM cross-review).
|
||||
|
|
@ -1,33 +0,0 @@
|
|||
{
|
||||
"_readme": "Shared HMAC signing test vectors for the SHOC work-order webhook (docs/shoc-webhook-contract.md section 6). string_to_sign = \"{timestamp}.{raw_body}\" computed over the RAW UTF-8 body bytes; signature = lowercase hex of HMAC-SHA256(secret, string_to_sign), sent as header X-SH-Signature: \"v1=<hex>\" alongside X-SH-Timestamp: <unix seconds> and X-SH-Key-Id: <kid>. The HMAC key is the UTF-8 bytes of the 64-hex 'secret' string exactly as stored in the workorder-ingest/shoc-webhook-hmac secret (NO hex-decoding on either side). 'body' is the exact raw JSON string to sign, byte-for-byte: vector 2 contains non-ASCII UTF-8 (multi-byte characters must be signed as their UTF-8 bytes), vector 3 is an empty JSON object. Producer pins: lambdas/wo/shoc_emitter/delivery.py sign_body and scripts/replay_shoc_webhooks.py sign_body, both enforced by tests/test_shoc_emitter_delivery.py. The SHOC receiver should verify its implementation against every vector before activation.",
|
||||
"vectors": [
|
||||
{
|
||||
"kid": "2026-07-20T00",
|
||||
"secret_hex": "3afc6cf9cc5782b304fda7efa7f0a77c4b67ab7336536daa228960dae34aa2fe",
|
||||
"timestamp": 1784642602,
|
||||
"body": "{\"schema_version\": 1, \"delivery_id\": \"f2a9c1de-7b34-4d5c-9e01-8a6b5c4d3e2f\", \"event_type\": \"work_order.created\", \"occurred_at\": \"2026-07-16T14:03:22.114208+00:00\", \"source\": \"procurement-ingest/workorder-shoc-emitter\", \"replay\": false, \"data\": {\"work_order_id\": \"11144580730\", \"wo_status\": \"new\", \"description\": \"Dock door 14 won't close\", \"customer\": \"AMAZON\", \"site_code\": \"JFK8\", \"building\": \"JFK8\", \"address\": \"546 Gulf Ave, Staten Island, NY 10314\", \"severity\": \"3-Normal\", \"priority\": \"Medium\", \"assigned_to\": \"Sea Haven Industries\", \"date_reported\": \"2026-07-14T09:12:00\", \"scheduled_start\": null, \"due_date\": null, \"record_type\": \"new_work_order\", \"created_at\": \"2026-07-16T14:03:22.114208+00:00\", \"updated_at\": \"2026-07-16T14:03:22.114208+00:00\"}}",
|
||||
"expected_signature": "4e6e171480e80eed333c966743c9da46595df86b4b65b76926e0781d2d3b0b46"
|
||||
},
|
||||
{
|
||||
"kid": "2026-07-20T00",
|
||||
"secret_hex": "3afc6cf9cc5782b304fda7efa7f0a77c4b67ab7336536daa228960dae34aa2fe",
|
||||
"timestamp": 1784642700,
|
||||
"body": "{\"schema_version\": 1, \"delivery_id\": \"0d1e2f3a-4b5c-6d7e-8f90-a1b2c3d4e5f6\", \"event_type\": \"work_order.comment_added\", \"occurred_at\": \"2026-07-16T14:05:00+00:00\", \"source\": \"procurement-ingest/workorder-shoc-emitter\", \"replay\": true, \"data\": {\"work_order_id\": \"11144580730\", \"comment_id\": \"11144580730#2026-04-27T23:51:48#a1b2c3d4e5f6\", \"record_type\": \"comment\", \"commenter\": \"APM Technician\", \"text\": \"Vendor dispatched — café access via süd door ✓\", \"created_at\": \"2026-04-27T23:51:48\", \"ingested_at\": \"2026-07-16T14:05:00+00:00\"}}",
|
||||
"expected_signature": "6b61d5ac09bbf032b1a9c9b651bd7d5ea2ec925f94a857eeade52995801479e5"
|
||||
},
|
||||
{
|
||||
"kid": "2026-06-20T00",
|
||||
"secret_hex": "737719b2c437aa252a0db5f65411f828f244d403f7e625f7336da10ee985b2e4",
|
||||
"timestamp": 1784000000,
|
||||
"body": "{}",
|
||||
"expected_signature": "ebe65980194b494c8de8d40cd6b8a42ff64c26749cf40fc6f36f59426dafa3c7"
|
||||
},
|
||||
{
|
||||
"kid": "2026-06-20T00",
|
||||
"secret_hex": "737719b2c437aa252a0db5f65411f828f244d403f7e625f7336da10ee985b2e4",
|
||||
"timestamp": 1784650000,
|
||||
"body": "{\"schema_version\": 1, \"event_type\": \"work_order.updated\", \"data\": {\"work_order_id\": \"999\"}}",
|
||||
"expected_signature": "626c5d241887c6186fe021907b1a67a0f14ff2286d19df97785a33994c56ca54"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -1,81 +0,0 @@
|
|||
# shoc-assessment-dynamo-reader (TEMPORARY)
|
||||
|
||||
Time-boxed cross-account read role in **seahaven-prod (011934824531)** that lets
|
||||
the Luby team assess the procurement-ingest DynamoDB tables directly from
|
||||
**seahaven-external-dev (396287094661)** during their initial assessment.
|
||||
|
||||
**This is not the steady-state access path.** SHOC's durable integration is the
|
||||
`procurement-api` SigV4 read API plus the `workorder-shoc-emitter` webhook.
|
||||
This role exists only so the team can eyeball raw tables while scoping that
|
||||
integration, and it self-expires.
|
||||
|
||||
| Item | Value |
|
||||
|---|---|
|
||||
| Role | `arn:aws:iam::011934824531:role/shoc-assessment-dynamo-reader` |
|
||||
| Trusts | `arn:aws:iam::396287094661:root` restricted by `ArnLike aws:PrincipalArn` to `role/aws-reserved/sso.amazonaws.com/*` — human Identity Center sessions in external-dev only (no EC2/service/created roles) |
|
||||
| Hard expiry | `2026-08-23T00:00:00Z` — `DateLessThan` on `aws:CurrentTime` in BOTH the trust policy (new AssumeRole calls fail) and every permissions statement (in-flight sessions lose all data access at the same instant) |
|
||||
| Actions | `dynamodb:GetItem`, `Query`, `Scan`, `DescribeTable` |
|
||||
| Tables | `purchase-orders`, `verified-sites`, `pending-site-review`, `WorkOrders`, `WorkOrderComments` (+ `index/*` on each) |
|
||||
| KMS | `kms:Decrypt` on `seahaven-dynamodb-cmk` (`be5fa4cb-c546-40fe-a13d-c7bec79f5d12`), `kms:ViaService`-pinned to DynamoDB (the three po-ingest tables are CMK-encrypted); create script validates the key ARN against SSM `/seahaven/dynamodb/cmk-arn` before applying |
|
||||
| Session | 1h (chained sessions from SSO cap at 1h regardless) |
|
||||
|
||||
## Deliberate design choices (re-review before changing)
|
||||
|
||||
- **Root trust + `aws:PrincipalArn` condition, no ExternalId.** The assessors
|
||||
are humans on Identity Center sessions in external-dev; console Switch-Role
|
||||
cannot pass an ExternalId, and there is no service deputy in this path. The
|
||||
`ArnLike` condition on the SSO role path restricts to human sessions without
|
||||
pinning a specific role ARN in `Principal` (role principals resolve to
|
||||
unique IDs and silently break when Identity Center reprovisions the
|
||||
permission set — a string condition survives that).
|
||||
- **`DescribeTable` added** beyond the requested GetItem/Query/Scan: the
|
||||
DynamoDB console item explorer and SDK table bindings require it
|
||||
(metadata-only).
|
||||
- **Not in CDK, not a table resource policy.** Table resource policies would
|
||||
re-open the cross-account grant surface deliberately zeroed on 2026-07-23 and
|
||||
trip `tests/test_cross_account_principal_pin.py`. A standalone role keeps the
|
||||
grant in one deletable object; the pin test's scope (cdk/) is intact.
|
||||
- **Reads are not access-logged** (no CloudTrail data events on these tables) —
|
||||
accepted for a 30-day read-only window.
|
||||
|
||||
## Data handling (communicate to Luby with the grant)
|
||||
|
||||
Table contents are Sea Haven confidential and the work-order comment bodies are
|
||||
free text that may contain personal data. View/query for the assessment only;
|
||||
no bulk export or copies outside the AWS session; delete any local extracts
|
||||
when the assessment ends. The governing vendor agreement applies.
|
||||
|
||||
## Emergency revocation
|
||||
|
||||
Fastest kill: run `./teardown-assessment-reader.sh`. A successful `DeleteRole`
|
||||
immediately invalidates all outstanding session credentials — and the script
|
||||
strips every inline/attached policy first, so `DeleteRole` cannot fail on
|
||||
`DeleteConflict` (including the `AWSRevokeOlderSessions` inline policy the
|
||||
console's "Revoke active sessions" button attaches).
|
||||
|
||||
## Usage (Luby side)
|
||||
|
||||
Console: Switch Role → account `011934824531`, role
|
||||
`shoc-assessment-dynamo-reader`. Direct table deep-links:
|
||||
`https://us-east-1.console.aws.amazon.com/dynamodbv2/home?region=us-east-1#item-explorer?table=<TableName>`
|
||||
|
||||
CLI:
|
||||
|
||||
```bash
|
||||
aws sts assume-role \
|
||||
--role-arn arn:aws:iam::011934824531:role/shoc-assessment-dynamo-reader \
|
||||
--role-session-name luby-assessment
|
||||
aws dynamodb scan --table-name WorkOrders --max-items 25 # with the returned creds
|
||||
```
|
||||
|
||||
## Lifecycle
|
||||
|
||||
- Create/update: `./create-assessment-reader.sh` (gated on GPT-4.1 cross-family
|
||||
review + `/sh-security-review` of these exact files).
|
||||
- Remove: `./teardown-assessment-reader.sh` when the assessment is done; the
|
||||
trust expiry is the backstop, not the plan.
|
||||
- Extend: edit the `DateLessThan` timestamp in **both** `trust-policy.json`
|
||||
AND every statement of `permissions-policy.json` (the expiry is enforced in
|
||||
both layers — extending only the trust policy yields a role that assumes but
|
||||
cannot read), then re-run both review gates before
|
||||
`create-assessment-reader.sh`.
|
||||
|
|
@ -1,116 +0,0 @@
|
|||
#!/usr/bin/env bash
|
||||
###############################################################################
|
||||
# create-assessment-reader.sh
|
||||
#
|
||||
# Creates / updates the TEMPORARY cross-account read role
|
||||
# `shoc-assessment-dynamo-reader` in AWS account 011934824531 (seahaven-prod),
|
||||
# us-east-1, for the Luby team's initial data assessment from
|
||||
# seahaven-external-dev (396287094661).
|
||||
#
|
||||
# TEMPORARY BY DESIGN:
|
||||
# - Hard expiry 2026-08-23T00:00:00Z enforced in BOTH layers: the trust
|
||||
# policy (new AssumeRole calls fail) and every permissions statement
|
||||
# (in-flight sessions lose data access at the same instant), so access
|
||||
# dies at the deadline even if teardown never runs.
|
||||
# - Steady-state SHOC access is the procurement-api SigV4 read API plus the
|
||||
# shoc-webhook emitter, NOT this role. Tear this down (or let it expire)
|
||||
# once the assessment is done; extend only by a deliberate edit to
|
||||
# trust-policy.json re-run through the review gates.
|
||||
#
|
||||
# GATE - DO NOT EXECUTE until BOTH of the following have passed on these
|
||||
# exact artifact files:
|
||||
# 1. GPT-4.1 cross-family review (IAM trust/permissions change) via
|
||||
# cross_review.py in the security-review repo.
|
||||
# 2. /sh-security-review (deep agentic pass - IaC/IAM is a gated surface).
|
||||
#
|
||||
# Design notes (deliberate, do not "fix" without re-review):
|
||||
# - Account-root trust (396287094661:root), NOT a pinned role: the assessors
|
||||
# are the Luby humans on SSO Admin sessions in external-dev, and every
|
||||
# principal in that account is Luby-controlled + SCP/boundary-contained.
|
||||
# - NO sts:ExternalId condition: console Switch-Role cannot pass one, and
|
||||
# there is no third-party deputy in this human-driven path. (The old
|
||||
# mgmt-account `shoc-dynamo-reader` used ExternalId because it served an
|
||||
# EC2 role, i.e. an actual service deputy.)
|
||||
# - dynamodb:DescribeTable is included beyond the requested
|
||||
# GetItem/Query/Scan because the DynamoDB console item explorer and most
|
||||
# SDK table bindings require it (metadata only).
|
||||
# - kms:Decrypt is ViaService-pinned to DynamoDB: the three po-ingest tables
|
||||
# are CMK-encrypted (seahaven-dynamodb-cmk); the WO tables are
|
||||
# AWS-owned-key encrypted and need no KMS grant.
|
||||
# - Max session 3600s: sessions from external-dev are role-chained (SSO ->
|
||||
# Admin role -> this role), and chained sessions cap at 1h regardless.
|
||||
###############################################################################
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
PROFILE="seahaven-prod"
|
||||
ROLE_NAME="shoc-assessment-dynamo-reader"
|
||||
POLICY_NAME="shoc-assessment-dynamo-read"
|
||||
ACCOUNT_ID="011934824531"
|
||||
SCRIPT_DIR="$(CDPATH= cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P)"
|
||||
TRUST_POLICY="file://${SCRIPT_DIR}/trust-policy.json"
|
||||
PERMS_POLICY="file://${SCRIPT_DIR}/permissions-policy.json"
|
||||
|
||||
echo "==> Verifying active account for profile '${PROFILE}'..."
|
||||
CALLER_ACCOUNT="$(aws --profile "${PROFILE}" sts get-caller-identity --query Account --output text)"
|
||||
if [[ "${CALLER_ACCOUNT}" != "${ACCOUNT_ID}" ]]; then
|
||||
echo "ERROR: profile '${PROFILE}' resolves to account ${CALLER_ACCOUNT}, expected ${ACCOUNT_ID}. Aborting." >&2
|
||||
exit 1
|
||||
fi
|
||||
CALLER_ARN="$(aws --profile "${PROFILE}" sts get-caller-identity --query Arn --output text)"
|
||||
echo " Audit: run by ${CALLER_ARN} at $(date -u +%Y-%m-%dT%H:%M:%SZ)"
|
||||
|
||||
echo "==> Validating the KMS key ARN in permissions-policy.json against SSM /seahaven/dynamodb/cmk-arn..."
|
||||
LIVE_CMK_ARN="$(aws --profile "${PROFILE}" ssm get-parameter \
|
||||
--name /seahaven/dynamodb/cmk-arn --query Parameter.Value --output text)"
|
||||
if ! grep -qF "\"${LIVE_CMK_ARN}\"" "${SCRIPT_DIR}/permissions-policy.json"; then
|
||||
echo "ERROR: permissions-policy.json does not reference the live DynamoDB CMK (${LIVE_CMK_ARN})." >&2
|
||||
echo " Copy/paste drift would make CMK-table reads fail silently at runtime. Aborting." >&2
|
||||
exit 1
|
||||
fi
|
||||
echo " OK - policy references the live CMK."
|
||||
|
||||
echo "==> Ensuring role '${ROLE_NAME}' exists with the correct trust policy..."
|
||||
if aws --profile "${PROFILE}" iam get-role --role-name "${ROLE_NAME}" >/dev/null 2>&1; then
|
||||
echo " Role exists - updating assume-role (trust) policy + re-asserting session cap."
|
||||
aws --profile "${PROFILE}" iam update-assume-role-policy \
|
||||
--role-name "${ROLE_NAME}" \
|
||||
--policy-document "${TRUST_POLICY}"
|
||||
aws --profile "${PROFILE}" iam update-role \
|
||||
--role-name "${ROLE_NAME}" \
|
||||
--max-session-duration 3600
|
||||
else
|
||||
echo " Role absent - creating."
|
||||
aws --profile "${PROFILE}" iam create-role \
|
||||
--role-name "${ROLE_NAME}" \
|
||||
--assume-role-policy-document "${TRUST_POLICY}" \
|
||||
--description "TEMPORARY read-only DynamoDB access for Luby initial assessment from seahaven-external-dev; trust self-expires 2026-08-23" \
|
||||
--max-session-duration 3600 \
|
||||
--tags Key=project,Value=procurement-ingest Key=managed-by,Value=create-assessment-reader.sh Key=temporary,Value=expires-2026-08-23
|
||||
fi
|
||||
|
||||
echo "==> Putting inline permissions policy '${POLICY_NAME}' (create-or-replace)..."
|
||||
aws --profile "${PROFILE}" iam put-role-policy \
|
||||
--role-name "${ROLE_NAME}" \
|
||||
--policy-name "${POLICY_NAME}" \
|
||||
--policy-document "${PERMS_POLICY}"
|
||||
|
||||
echo "==> Checking for unexpected policies on the role..."
|
||||
EXTRA_INLINE="$(aws --profile "${PROFILE}" iam list-role-policies \
|
||||
--role-name "${ROLE_NAME}" --query "PolicyNames[?@!='${POLICY_NAME}']" --output text)"
|
||||
EXTRA_ATTACHED="$(aws --profile "${PROFILE}" iam list-attached-role-policies \
|
||||
--role-name "${ROLE_NAME}" --query 'AttachedPolicies[].PolicyName' --output text)"
|
||||
if [[ -n "${EXTRA_INLINE}" || -n "${EXTRA_ATTACHED}" ]]; then
|
||||
echo " WARNING: unreviewed policies present on ${ROLE_NAME} (not in these artifacts):" >&2
|
||||
[[ -n "${EXTRA_INLINE}" ]] && echo " inline: ${EXTRA_INLINE}" >&2
|
||||
[[ -n "${EXTRA_ATTACHED}" ]] && echo " attached: ${EXTRA_ATTACHED}" >&2
|
||||
echo " Review and remove them (delete-role-policy / detach-role-policy) if unexpected." >&2
|
||||
else
|
||||
echo " OK - only ${POLICY_NAME} present."
|
||||
fi
|
||||
|
||||
ROLE_ARN="$(aws --profile "${PROFILE}" iam get-role --role-name "${ROLE_NAME}" \
|
||||
--query Role.Arn --output text)"
|
||||
|
||||
echo "==> Done. Assessment reader ready (expires 2026-08-23T00:00:00Z):"
|
||||
echo " ${ROLE_ARN}"
|
||||
|
|
@ -1,46 +0,0 @@
|
|||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Sid": "ReadProcurementTables",
|
||||
"Effect": "Allow",
|
||||
"Action": [
|
||||
"dynamodb:GetItem",
|
||||
"dynamodb:Query",
|
||||
"dynamodb:Scan",
|
||||
"dynamodb:DescribeTable"
|
||||
],
|
||||
"Resource": [
|
||||
"arn:aws:dynamodb:us-east-1:011934824531:table/purchase-orders",
|
||||
"arn:aws:dynamodb:us-east-1:011934824531:table/purchase-orders/index/*",
|
||||
"arn:aws:dynamodb:us-east-1:011934824531:table/verified-sites",
|
||||
"arn:aws:dynamodb:us-east-1:011934824531:table/verified-sites/index/*",
|
||||
"arn:aws:dynamodb:us-east-1:011934824531:table/pending-site-review",
|
||||
"arn:aws:dynamodb:us-east-1:011934824531:table/pending-site-review/index/*",
|
||||
"arn:aws:dynamodb:us-east-1:011934824531:table/WorkOrders",
|
||||
"arn:aws:dynamodb:us-east-1:011934824531:table/WorkOrders/index/*",
|
||||
"arn:aws:dynamodb:us-east-1:011934824531:table/WorkOrderComments",
|
||||
"arn:aws:dynamodb:us-east-1:011934824531:table/WorkOrderComments/index/*"
|
||||
],
|
||||
"Condition": {
|
||||
"DateLessThan": {
|
||||
"aws:CurrentTime": "2026-08-23T00:00:00Z"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"Sid": "DecryptTableCmkViaDynamo",
|
||||
"Effect": "Allow",
|
||||
"Action": "kms:Decrypt",
|
||||
"Resource": "arn:aws:kms:us-east-1:011934824531:key/be5fa4cb-c546-40fe-a13d-c7bec79f5d12",
|
||||
"Condition": {
|
||||
"StringEquals": {
|
||||
"kms:ViaService": "dynamodb.us-east-1.amazonaws.com"
|
||||
},
|
||||
"DateLessThan": {
|
||||
"aws:CurrentTime": "2026-08-23T00:00:00Z"
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -1,61 +0,0 @@
|
|||
#!/usr/bin/env bash
|
||||
###############################################################################
|
||||
# teardown-assessment-reader.sh
|
||||
#
|
||||
# Deletes the temporary `shoc-assessment-dynamo-reader` role in seahaven-prod
|
||||
# (011934824531). Run once the Luby initial assessment is complete — and this
|
||||
# is ALSO the emergency kill switch: a successful DeleteRole immediately
|
||||
# invalidates all outstanding session credentials for the role.
|
||||
#
|
||||
# DeleteRole fails with DeleteConflict while ANY policy remains on the role
|
||||
# (including the inline `AWSRevokeOlderSessions` policy the IAM console's
|
||||
# "Revoke active sessions" button attaches), so this script enumerates and
|
||||
# strips ALL inline policies, attached managed policies, and instance-profile
|
||||
# memberships first. Idempotent: a rerun after the role is gone exits 0.
|
||||
###############################################################################
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
PROFILE="seahaven-prod"
|
||||
ROLE_NAME="shoc-assessment-dynamo-reader"
|
||||
ACCOUNT_ID="011934824531"
|
||||
|
||||
CALLER_ACCOUNT="$(aws --profile "${PROFILE}" sts get-caller-identity --query Account --output text)"
|
||||
if [[ "${CALLER_ACCOUNT}" != "${ACCOUNT_ID}" ]]; then
|
||||
echo "ERROR: profile '${PROFILE}' resolves to account ${CALLER_ACCOUNT}, expected ${ACCOUNT_ID}. Aborting." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! aws --profile "${PROFILE}" iam get-role --role-name "${ROLE_NAME}" >/dev/null 2>&1; then
|
||||
echo "==> Role '${ROLE_NAME}' already absent in ${ACCOUNT_ID} - nothing to do."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "==> Deleting ALL inline policies on '${ROLE_NAME}'..."
|
||||
for POLICY in $(aws --profile "${PROFILE}" iam list-role-policies \
|
||||
--role-name "${ROLE_NAME}" --query 'PolicyNames[]' --output text); do
|
||||
echo " delete-role-policy ${POLICY}"
|
||||
aws --profile "${PROFILE}" iam delete-role-policy \
|
||||
--role-name "${ROLE_NAME}" --policy-name "${POLICY}"
|
||||
done
|
||||
|
||||
echo "==> Detaching ALL managed policies on '${ROLE_NAME}'..."
|
||||
for POLICY_ARN in $(aws --profile "${PROFILE}" iam list-attached-role-policies \
|
||||
--role-name "${ROLE_NAME}" --query 'AttachedPolicies[].PolicyArn' --output text); do
|
||||
echo " detach-role-policy ${POLICY_ARN}"
|
||||
aws --profile "${PROFILE}" iam detach-role-policy \
|
||||
--role-name "${ROLE_NAME}" --policy-arn "${POLICY_ARN}"
|
||||
done
|
||||
|
||||
echo "==> Removing '${ROLE_NAME}' from any instance profiles..."
|
||||
for IP in $(aws --profile "${PROFILE}" iam list-instance-profiles-for-role \
|
||||
--role-name "${ROLE_NAME}" --query 'InstanceProfiles[].InstanceProfileName' --output text); do
|
||||
echo " remove-role-from-instance-profile ${IP}"
|
||||
aws --profile "${PROFILE}" iam remove-role-from-instance-profile \
|
||||
--instance-profile-name "${IP}" --role-name "${ROLE_NAME}"
|
||||
done
|
||||
|
||||
echo "==> Deleting role '${ROLE_NAME}' (this invalidates all outstanding sessions)..."
|
||||
aws --profile "${PROFILE}" iam delete-role --role-name "${ROLE_NAME}"
|
||||
|
||||
echo "==> Done. ${ROLE_NAME} removed from ${ACCOUNT_ID}; all live sessions are dead."
|
||||
|
|
@ -1,21 +0,0 @@
|
|||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Sid": "ExternalDevAssessmentAssume",
|
||||
"Effect": "Allow",
|
||||
"Principal": {
|
||||
"AWS": "arn:aws:iam::396287094661:root"
|
||||
},
|
||||
"Action": "sts:AssumeRole",
|
||||
"Condition": {
|
||||
"DateLessThan": {
|
||||
"aws:CurrentTime": "2026-08-23T00:00:00Z"
|
||||
},
|
||||
"ArnLike": {
|
||||
"aws:PrincipalArn": "arn:aws:iam::396287094661:role/aws-reserved/sso.amazonaws.com/*"
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -82,16 +82,7 @@ def _load_spec() -> str:
|
|||
def _base_url(event: dict) -> str | None:
|
||||
rc = event.get("requestContext") or {}
|
||||
domain, stage = rc.get("domainName"), rc.get("stage")
|
||||
if not domain:
|
||||
return None
|
||||
# A custom domain (base-path mapping) serves the stage at the root, so the
|
||||
# public URL has NO /{stage} segment; only the raw execute-api host carries
|
||||
# it. Emitting /{stage} for a custom-domain request would advertise a
|
||||
# broken server URL in the docs. All API Gateway default domains end with
|
||||
# .amazonaws.com, so check the suffix rather than a substring.
|
||||
if domain.endswith(".amazonaws.com"):
|
||||
return f"https://{domain}/{stage}" if stage else None
|
||||
return f"https://{domain}"
|
||||
return f"https://{domain}/{stage}" if domain and stage else None
|
||||
|
||||
|
||||
def _spec_for_request(event: dict) -> str:
|
||||
|
|
|
|||
|
|
@ -9,10 +9,6 @@
|
|||
"description": "Read API over the procurement-ingest pipelines (work orders and purchase orders). The outbound SHOC work-order webhook feed is documented under **webhooks** below.\n\nData endpoints use AWS IAM SigV4; the `/docs` and `/openapi.json` routes use a shared token. Listings are unordered, cursor-paginated scans. Endpoints tagged **x-planned** are phase 2 and currently answer `501`.\n\nThis API replaces SHOC's retired SyncController DynamoDB scan as the reconciliation and backfill path. Full detail: the repo README and `docs/shoc-webhook-contract.md`."
|
||||
},
|
||||
"servers": [
|
||||
{
|
||||
"url": "https://procurement-api.seahaven.com",
|
||||
"description": "seahaven-prod (011934824531) custom domain — recommended."
|
||||
},
|
||||
{
|
||||
"url": "https://mvul1efda2.execute-api.us-east-1.amazonaws.com/prod",
|
||||
"description": "seahaven-prod (011934824531)."
|
||||
|
|
|
|||
|
|
@ -1 +1 @@
|
|||
boto3==1.43.56
|
||||
boto3==1.43.55
|
||||
|
|
|
|||
|
|
@ -1 +1 @@
|
|||
boto3==1.43.55
|
||||
boto3==1.43.53
|
||||
|
|
|
|||
|
|
@ -1,218 +0,0 @@
|
|||
"""HMAC-signed webhook delivery to the SHOC receiver (contract sections 6-7).
|
||||
|
||||
Owns the signed POST and its response classification. The signing key comes
|
||||
from Secrets Manager (``workorder-ingest/shoc-webhook-hmac``, dual-key shape
|
||||
``{"keys": [{"kid", "secret"}, ...]}``), cached in the warm container for a
|
||||
short TTL so 30-day rotation propagates without waiting for the execution
|
||||
environment to recycle -- the producer always signs with ``keys[0]``.
|
||||
|
||||
Response contract (section 7): 2xx -> delivered; 429/5xx/timeout/connection
|
||||
error -> RetryableDeliveryError (the handler surfaces it as a batch item
|
||||
failure so the ESM blocks the shard and retries in order); any other status ->
|
||||
("rejected", code) for the handler to park (a contract bug must not block the
|
||||
shard for 24 hours). Secret material and signatures are never logged.
|
||||
|
||||
``sign_body`` is a module-level pure function on purpose: the golden-vector
|
||||
test suite and the replay script both pin against it, and it is the shared
|
||||
definition Luby's receiver verifies with.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import hmac
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from http import HTTPStatus
|
||||
|
||||
import boto3
|
||||
from botocore.config import Config
|
||||
|
||||
logger = logging.getLogger()
|
||||
logger.setLevel(logging.INFO)
|
||||
|
||||
# Required, NO default (Open SWE #0): a hardcoded fallback URL would silently
|
||||
# ship production work-order state to that endpoint if the CDK-set env var were
|
||||
# ever dropped. Unset -> deliver() fails closed (retries in order) rather than
|
||||
# defaulting anywhere. The single source of the URL is the Lambda environment
|
||||
# (CDK), confirmed by the activation PR.
|
||||
SHOC_WEBHOOK_URL = os.environ.get("SHOC_WEBHOOK_URL")
|
||||
HMAC_SECRET_ARN = os.environ.get("HMAC_SECRET_ARN")
|
||||
|
||||
POST_TIMEOUT_SECONDS = 10
|
||||
USER_AGENT = "workorder-shoc-emitter/1"
|
||||
|
||||
# Bound the Secrets Manager client's timeouts (Open SWE #16): default botocore
|
||||
# timeouts can run to ~60s, which combined with the 10s POST could exhaust the
|
||||
# Lambda budget and make a slow AWS call re-deliver the whole batch. Fast, few
|
||||
# retries -- the key is 300s-cached so this fetch is rare.
|
||||
_BOTO_CONFIG = Config(
|
||||
connect_timeout=3, read_timeout=5, retries={"max_attempts": 2, "mode": "standard"}
|
||||
)
|
||||
|
||||
# 2xx = delivered; 429 and 5xx retry; everything else parks (contract sec. 7).
|
||||
_HTTP_SUCCESS_RANGE = range(HTTPStatus.OK, HTTPStatus.MULTIPLE_CHOICES)
|
||||
_HTTP_SERVER_ERROR_MIN = HTTPStatus.INTERNAL_SERVER_ERROR
|
||||
# Transient auth failures: a stale cached key mid-rotation, a receiver
|
||||
# secret-fetch blip, or clock skew past the +/-300s window. These are
|
||||
# availability events, not payload contract bugs, so they retry (in order)
|
||||
# after the key cache is dropped -- never park, which would strand the
|
||||
# delivery out of the ordered feed until a manual replay.
|
||||
_HTTP_AUTH_FAILURES = (HTTPStatus.UNAUTHORIZED, HTTPStatus.FORBIDDEN)
|
||||
|
||||
|
||||
class _NoRedirectHandler(urllib.request.HTTPRedirectHandler):
|
||||
"""Refuse to follow receiver redirects.
|
||||
|
||||
The default opener follows 3xx transparently, which would (a) forward the
|
||||
live X-SH-* auth headers to a receiver-chosen Location and (b) let an
|
||||
http:// Location slip past the https-only guard on the configured URL. We
|
||||
only ever POST to the one configured endpoint; a redirect is a receiver
|
||||
misconfiguration, so surface the 3xx as an HTTPError and let it park.
|
||||
"""
|
||||
|
||||
def redirect_request(self, *args, **kwargs):
|
||||
return None
|
||||
|
||||
|
||||
_opener = urllib.request.build_opener(_NoRedirectHandler)
|
||||
|
||||
# Refresh the cached key material this often so a rotated secret propagates
|
||||
# without waiting for the execution environment to recycle (contract requires
|
||||
# receiver-side TTL <= 300s; the producer matches it).
|
||||
_HMAC_KEYS_CACHE_TTL_SECONDS = 300
|
||||
_hmac_keys_cache = None
|
||||
_hmac_keys_cached_at = 0.0
|
||||
|
||||
|
||||
class RetryableDeliveryError(Exception):
|
||||
"""Delivery failed in a way the ESM should retry in order (shard-blocking).
|
||||
|
||||
``status_code`` carries the HTTP status when one exists (429/5xx); it is
|
||||
None for timeouts, connection errors, and secret-fetch failures.
|
||||
"""
|
||||
|
||||
def __init__(self, message: str, status_code: int | None = None):
|
||||
super().__init__(message)
|
||||
self.status_code = status_code
|
||||
|
||||
|
||||
def _get_hmac_keys() -> list[dict]:
|
||||
"""Fetch the signing-key list from Secrets Manager, TTL-cached.
|
||||
|
||||
An empty/missing ``keys`` list is the bootstrap state before the first
|
||||
rotation has run -- retryable, not a crash: the shard blocks until the
|
||||
rotator populates the secret. Fetch failures (throttle, transient IAM/KMS
|
||||
denial) are likewise retryable so a Secrets Manager blip blocks in order
|
||||
instead of failing the whole batch. Key material is never logged.
|
||||
"""
|
||||
global _hmac_keys_cache, _hmac_keys_cached_at
|
||||
now = time.monotonic()
|
||||
if (
|
||||
_hmac_keys_cache is not None
|
||||
and now - _hmac_keys_cached_at < _HMAC_KEYS_CACHE_TTL_SECONDS
|
||||
):
|
||||
return _hmac_keys_cache
|
||||
return _refresh_hmac_keys()
|
||||
|
||||
|
||||
def _invalidate_hmac_keys() -> None:
|
||||
"""Drop the cached key material so the next sign re-fetches (rotation)."""
|
||||
global _hmac_keys_cache
|
||||
_hmac_keys_cache = None
|
||||
|
||||
|
||||
def _refresh_hmac_keys() -> list[dict]:
|
||||
"""Force a fresh Secrets Manager fetch, bypassing the TTL cache."""
|
||||
global _hmac_keys_cache, _hmac_keys_cached_at
|
||||
now = time.monotonic()
|
||||
secrets = boto3.client("secretsmanager", config=_BOTO_CONFIG)
|
||||
try:
|
||||
secret = secrets.get_secret_value(SecretId=HMAC_SECRET_ARN)
|
||||
keys = json.loads(secret["SecretString"]).get("keys") or []
|
||||
except Exception as exc:
|
||||
raise RetryableDeliveryError(f"hmac secret fetch failed: {exc}") from exc
|
||||
if not keys:
|
||||
raise RetryableDeliveryError("hmac secret not yet rotated")
|
||||
_hmac_keys_cache = keys
|
||||
_hmac_keys_cached_at = now
|
||||
return keys
|
||||
|
||||
|
||||
def sign_body(secret_hex: str, timestamp: int, raw_body: bytes) -> str:
|
||||
"""HMAC-SHA256 hex digest over ``f"{timestamp}.{raw_body}"`` (raw bytes).
|
||||
|
||||
The key is the UTF-8 bytes of the secret string exactly as stored in the
|
||||
secret's ``secret`` field (the receiver reads the same JSON field -- no
|
||||
hex-decoding on either side). Pure function; the golden-vector tests and
|
||||
Luby's receiver both pin against this definition.
|
||||
"""
|
||||
string_to_sign = f"{timestamp}.".encode() + raw_body
|
||||
return hmac.new(
|
||||
secret_hex.encode("utf-8"), string_to_sign, hashlib.sha256
|
||||
).hexdigest()
|
||||
|
||||
|
||||
def deliver(envelope: dict) -> tuple[str, int]:
|
||||
"""POST one envelope to SHOC. Returns ("delivered"|"rejected", status).
|
||||
|
||||
Raises RetryableDeliveryError for 429/5xx/timeout/connection failures so
|
||||
the caller can block the shard (in-order retry, contract section 7).
|
||||
"""
|
||||
if not SHOC_WEBHOOK_URL or not SHOC_WEBHOOK_URL.startswith("https://"):
|
||||
# Fail closed if the URL is unset (no hardcoded fallback -- Open SWE
|
||||
# #0) or is any non-HTTPS scheme (file://, http://, ...): urllib would
|
||||
# otherwise follow it, and the HMAC only protects an HTTPS transport.
|
||||
# Retryable, so a config gap blocks the shard (visible via iterator-age)
|
||||
# rather than shipping data somewhere unintended.
|
||||
raise RetryableDeliveryError(
|
||||
"SHOC_WEBHOOK_URL is unset or not an https:// URL; refusing to deliver"
|
||||
)
|
||||
raw_body = json.dumps(envelope).encode("utf-8")
|
||||
timestamp = int(time.time())
|
||||
signing_key = _get_hmac_keys()[0]
|
||||
signature = sign_body(signing_key["secret"], timestamp, raw_body)
|
||||
request = urllib.request.Request(
|
||||
SHOC_WEBHOOK_URL,
|
||||
data=raw_body,
|
||||
headers={
|
||||
"Content-Type": "application/json; charset=utf-8",
|
||||
"User-Agent": USER_AGENT,
|
||||
"X-SH-Timestamp": str(timestamp),
|
||||
"X-SH-Key-Id": signing_key["kid"],
|
||||
"X-SH-Signature": f"v1={signature}",
|
||||
},
|
||||
method="POST",
|
||||
)
|
||||
try:
|
||||
# _opener refuses redirects, so a 3xx surfaces here as an HTTPError
|
||||
# rather than silently re-issuing the request (with its auth headers)
|
||||
# to a receiver-chosen Location.
|
||||
with _opener.open(request, timeout=POST_TIMEOUT_SECONDS) as response:
|
||||
status_code = response.status
|
||||
except urllib.error.HTTPError as exc:
|
||||
status_code = exc.code
|
||||
if (
|
||||
status_code == HTTPStatus.TOO_MANY_REQUESTS
|
||||
or status_code >= _HTTP_SERVER_ERROR_MIN
|
||||
):
|
||||
raise RetryableDeliveryError(
|
||||
f"receiver returned {status_code}", status_code=status_code
|
||||
) from exc
|
||||
if status_code in _HTTP_AUTH_FAILURES:
|
||||
# Drop the cached key so an in-order retry re-signs with the
|
||||
# current secret (handles a rotation that outran the TTL cache).
|
||||
_invalidate_hmac_keys()
|
||||
raise RetryableDeliveryError(
|
||||
f"receiver auth failure {status_code}", status_code=status_code
|
||||
) from exc
|
||||
return ("rejected", status_code)
|
||||
except (TimeoutError, urllib.error.URLError, OSError) as exc:
|
||||
raise RetryableDeliveryError(f"connection error: {exc}") from exc
|
||||
if status_code in _HTTP_SUCCESS_RANGE:
|
||||
return ("delivered", status_code)
|
||||
# A non-2xx that the opener did not raise for -- contract bug on
|
||||
# someone's side. Park it, never block the shard.
|
||||
return ("rejected", status_code)
|
||||
|
|
@ -1,163 +0,0 @@
|
|||
"""Stream-record -> SHOC webhook envelope mapping (contract sections 3-4).
|
||||
|
||||
Pure and total: no boto3 clients, no environment reads, no network. Every
|
||||
function either returns a value or returns None (skip) -- a malformed stream
|
||||
record must never raise here; classification gaps fall through to None so the
|
||||
shard is never blocked by an unmappable record. ``build_event`` is the single
|
||||
entry point the handler calls per record.
|
||||
|
||||
Classification (docs/shoc-webhook-contract.md section 4):
|
||||
WorkOrders INSERT -> work_order.created
|
||||
WorkOrders MODIFY -> work_order.cancelled iff OLD wo_status was not
|
||||
"cancelled" AND NEW wo_status is "cancelled",
|
||||
else work_order.updated
|
||||
WorkOrderComments INSERT -> work_order.comment_added
|
||||
REMOVE (either table) -> skip (no deletes in this pipeline)
|
||||
NEW write_origin == "shoc-write-api" -> skip (echo guard; a no-op branch
|
||||
today -- nothing writes that attribute yet -- kept so the phase-2
|
||||
write-back API never echoes SHOC's own writes back at it)
|
||||
Anything else (unknown table, comment MODIFYs) -> skip.
|
||||
"""
|
||||
|
||||
from datetime import datetime, timezone
|
||||
from decimal import Decimal
|
||||
|
||||
from boto3.dynamodb.types import TypeDeserializer
|
||||
|
||||
SCHEMA_VERSION = 1
|
||||
SOURCE = "procurement-ingest/workorder-shoc-emitter"
|
||||
CANCELLED_STATUS = "cancelled"
|
||||
SHOC_WRITE_ORIGIN = "shoc-write-api"
|
||||
WORK_ORDERS_TABLE = "WorkOrders"
|
||||
COMMENTS_TABLE = "WorkOrderComments"
|
||||
|
||||
# data payload field lists (contract sections 4.1 / 4.2). Absent attributes are
|
||||
# null-filled; source_email_s3_key is deliberately EXCLUDED (internal key, not
|
||||
# part of the contract).
|
||||
WO_DATA_FIELDS = (
|
||||
"work_order_id",
|
||||
"wo_status",
|
||||
"description",
|
||||
"customer",
|
||||
"site_code",
|
||||
"building",
|
||||
"address",
|
||||
"severity",
|
||||
"priority",
|
||||
"assigned_to",
|
||||
"date_reported",
|
||||
"scheduled_start",
|
||||
"due_date",
|
||||
"record_type",
|
||||
"created_at",
|
||||
"updated_at",
|
||||
)
|
||||
COMMENT_DATA_FIELDS = (
|
||||
"work_order_id",
|
||||
"comment_id",
|
||||
"record_type",
|
||||
"commenter",
|
||||
"text",
|
||||
"created_at",
|
||||
"ingested_at",
|
||||
)
|
||||
|
||||
_deserializer = TypeDeserializer()
|
||||
# arn:aws:dynamodb:...:table/NAME[/stream/LABEL] -- slash-split needs at least
|
||||
# the ":table" head plus the name segment for the parse to be meaningful.
|
||||
_MIN_ARN_SLASH_SEGMENTS = 2
|
||||
|
||||
|
||||
def _plain(value):
|
||||
"""Convert TypeDeserializer output to json.dumps-safe plain Python.
|
||||
|
||||
Mirrors lambdas/api/serialization.py: integral Decimals become JSON
|
||||
integers (exact), non-integral Decimals become floats, sets become sorted
|
||||
lists. Local copy on purpose -- the emitter bundles flat and must not grow
|
||||
a cross-package import for twelve lines of logic.
|
||||
"""
|
||||
if isinstance(value, Decimal):
|
||||
if value == value.to_integral_value():
|
||||
return int(value)
|
||||
return float(value)
|
||||
if isinstance(value, dict):
|
||||
return {key: _plain(item) for key, item in value.items()}
|
||||
if isinstance(value, list):
|
||||
return [_plain(item) for item in value]
|
||||
if isinstance(value, set):
|
||||
return sorted(_plain(item) for item in value)
|
||||
return value
|
||||
|
||||
|
||||
def _deserialize_image(image: dict) -> dict:
|
||||
"""DynamoDB stream image (attribute-value encoded) -> plain dict."""
|
||||
return {key: _plain(_deserializer.deserialize(av)) for key, av in image.items()}
|
||||
|
||||
|
||||
def _table_name(event_source_arn: str) -> str | None:
|
||||
"""Extract the exact table-name segment from a stream eventSourceARN.
|
||||
|
||||
ARN format: arn:aws:dynamodb:region:acct:table/NAME/stream/LABEL. Parsing
|
||||
the segment exactly (not a loose substring) matters: "WorkOrders" is a
|
||||
prefix of nothing, but a substring test for it WOULD match
|
||||
"WorkOrderComments"-adjacent names -- segment equality can't.
|
||||
"""
|
||||
parts = event_source_arn.split("/")
|
||||
if len(parts) >= _MIN_ARN_SLASH_SEGMENTS and parts[0].endswith(":table"):
|
||||
return parts[1]
|
||||
return None
|
||||
|
||||
|
||||
def _classify(table, event_name, new_image, old_image) -> str | None:
|
||||
if table == WORK_ORDERS_TABLE:
|
||||
if event_name == "INSERT":
|
||||
return "work_order.created"
|
||||
if event_name == "MODIFY":
|
||||
was_cancelled = old_image.get("wo_status") == CANCELLED_STATUS
|
||||
if not was_cancelled and new_image.get("wo_status") == CANCELLED_STATUS:
|
||||
return "work_order.cancelled"
|
||||
return "work_order.updated"
|
||||
return None
|
||||
if table == COMMENTS_TABLE and event_name == "INSERT":
|
||||
return "work_order.comment_added"
|
||||
return None
|
||||
|
||||
|
||||
def build_event(record: dict) -> dict | None:
|
||||
"""Map one DynamoDB stream record to a webhook envelope, or None to skip."""
|
||||
event_name = record.get("eventName")
|
||||
if event_name == "REMOVE":
|
||||
return None
|
||||
stream = record.get("dynamodb") or {}
|
||||
new_image = _deserialize_image(stream.get("NewImage") or {})
|
||||
if new_image.get("write_origin") == SHOC_WRITE_ORIGIN:
|
||||
return None # echo guard (forward-compat, contract section 3)
|
||||
table = _table_name(record.get("eventSourceARN") or "")
|
||||
old_image = _deserialize_image(stream.get("OldImage") or {})
|
||||
event_type = _classify(table, event_name, new_image, old_image)
|
||||
if event_type is None:
|
||||
return None
|
||||
if event_type == "work_order.comment_added":
|
||||
fields = COMMENT_DATA_FIELDS
|
||||
else:
|
||||
fields = WO_DATA_FIELDS
|
||||
# eventID and ApproximateCreationDateTime are present on every real
|
||||
# DynamoDB stream record; guarding them keeps this function total (the
|
||||
# "never raise" contract above) rather than trusting a hard subscript.
|
||||
event_id = record.get("eventID")
|
||||
approx_creation = stream.get("ApproximateCreationDateTime")
|
||||
if event_id is None or approx_creation is None:
|
||||
return None
|
||||
# ApproximateCreationDateTime arrives as epoch seconds (float/Decimal).
|
||||
occurred_at = datetime.fromtimestamp(
|
||||
float(approx_creation), tz=timezone.utc
|
||||
).isoformat()
|
||||
return {
|
||||
"schema_version": SCHEMA_VERSION,
|
||||
"delivery_id": event_id,
|
||||
"event_type": event_type,
|
||||
"occurred_at": occurred_at,
|
||||
"source": SOURCE,
|
||||
"replay": False,
|
||||
"data": {field: new_image.get(field) for field in fields},
|
||||
}
|
||||
|
|
@ -1,180 +0,0 @@
|
|||
"""
|
||||
SHOC work-order webhook emitter Lambda.
|
||||
|
||||
Consumes the WorkOrders/WorkOrderComments DynamoDB streams and pushes each
|
||||
mutation to SHOC as an HMAC-signed HTTPS POST (docs/shoc-webhook-contract.md).
|
||||
This handler is the thin event loop; the work lives in flat sibling modules
|
||||
(bare-name imports resolve via the same flat-landing bundling as the email
|
||||
processor's siblings):
|
||||
envelope.py -- stream record -> envelope mapping + event classification
|
||||
delivery.py -- secret cache, signing, POST, response classification
|
||||
|
||||
Ships DARK: both event-source mappings deploy with enabled=False, so this code
|
||||
runs zero deliveries until the activation PR flips them on after SHOC's
|
||||
receiver passes the shared HMAC test vectors.
|
||||
|
||||
Ordering semantics: the ESMs run parallelization_factor=1 with bisect-on-error
|
||||
off, and this loop processes records strictly in order. A retryable failure
|
||||
(429/5xx/timeout/connection error) stops the batch immediately and reports
|
||||
that record via report_batch_item_failures -- earlier successes are not
|
||||
re-delivered, and the ESM blocks the shard and retries from the failed record,
|
||||
preserving per-work-order commit order. Non-retryable 4xx responses are a
|
||||
contract bug, not an availability blip: the full envelope is parked on the
|
||||
rejected queue (alarmed) and the loop continues, so a bad payload can never
|
||||
block the shard for 24 hours.
|
||||
|
||||
No healthcheck branch on purpose: stream consumers are not smoke-gated
|
||||
(po-ingest-site-extractor precedent) -- there is no direct-invoke path to
|
||||
probe, and a synthetic stream record would be a real delivery.
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import time
|
||||
|
||||
import boto3
|
||||
import delivery
|
||||
import envelope
|
||||
from botocore.config import Config
|
||||
|
||||
logger = logging.getLogger()
|
||||
logger.setLevel(logging.INFO)
|
||||
|
||||
REJECTED_QUEUE_URL = os.environ.get("REJECTED_QUEUE_URL")
|
||||
|
||||
# Bound the SQS client's timeouts (Open SWE #17): a slow SQS response while
|
||||
# parking a rejected envelope must not hang the invocation toward its 60s
|
||||
# timeout and re-deliver the whole batch.
|
||||
_BOTO_CONFIG = Config(
|
||||
connect_timeout=3, read_timeout=5, retries={"max_attempts": 2, "mode": "standard"}
|
||||
)
|
||||
|
||||
# Lazy cached SQS client. Keeps the public attribute name ``sqs`` so the test
|
||||
# monkeypatch target changes module only, not attribute name.
|
||||
sqs = None
|
||||
|
||||
|
||||
def _get_sqs():
|
||||
global sqs
|
||||
if sqs is None:
|
||||
sqs = boto3.client("sqs", config=_BOTO_CONFIG)
|
||||
return sqs
|
||||
|
||||
|
||||
def _delivery_log(event: dict, status_code, latency_ms: int, outcome: str) -> str:
|
||||
"""One structured record per delivery attempt (Logs-Insights-queryable)."""
|
||||
return json.dumps(
|
||||
{
|
||||
"event": "shoc_delivery",
|
||||
"delivery_id": event["delivery_id"],
|
||||
"event_type": event["event_type"],
|
||||
"work_order_id": event["data"].get("work_order_id"),
|
||||
"status_code": status_code,
|
||||
"latency_ms": latency_ms,
|
||||
"outcome": outcome,
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def _park_rejected(event: dict, status_code: int):
|
||||
"""Send the full envelope to the rejected queue for operator replay."""
|
||||
_get_sqs().send_message(
|
||||
QueueUrl=REJECTED_QUEUE_URL,
|
||||
MessageBody=json.dumps({"envelope": event, "response_status": status_code}),
|
||||
)
|
||||
logger.warning(
|
||||
json.dumps(
|
||||
{
|
||||
"event": "shoc_delivery_rejected",
|
||||
"delivery_id": event["delivery_id"],
|
||||
"event_type": event["event_type"],
|
||||
"work_order_id": event["data"].get("work_order_id"),
|
||||
"response_status": status_code,
|
||||
}
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _batch_summary(records: int, emitted: int, skipped: int) -> str:
|
||||
"""Per-invocation emit/skip tally.
|
||||
|
||||
Skips (REMOVE, echo guard, comment MODIFYs, unmappable records) return no
|
||||
delivery and would otherwise be invisible: a systemic classification break
|
||||
-- e.g. a table rename desyncing the eventSourceARN parse -- would drop
|
||||
100%% of events while every batch still reports success. Logging the ratio
|
||||
makes that queryable in Logs Insights and alarmable.
|
||||
"""
|
||||
return json.dumps(
|
||||
{
|
||||
"event": "shoc_batch_summary",
|
||||
"records": records,
|
||||
"emitted": emitted,
|
||||
"skipped": skipped,
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def handler(event, context):
|
||||
"""Lambda entry point. Triggered by the two WO-table stream ESMs."""
|
||||
records = event.get("Records", [])
|
||||
emitted = 0
|
||||
skipped = 0
|
||||
for record in records:
|
||||
webhook_event = envelope.build_event(record)
|
||||
if webhook_event is None:
|
||||
skipped += 1
|
||||
continue
|
||||
# Capture the sequence number BEFORE any delivery attempt so the
|
||||
# failure paths below can never KeyError on the subscript (Open SWE
|
||||
# #9/#26 -- the catch-all exists to keep an exception from re-delivering
|
||||
# the whole batch, so it must not itself raise). SequenceNumber is
|
||||
# present on every real DynamoDB stream record; a record lacking it is
|
||||
# unmappable-to-a-checkpoint, so log and skip rather than crash.
|
||||
seq = record.get("dynamodb", {}).get("SequenceNumber")
|
||||
if seq is None:
|
||||
logger.warning(
|
||||
json.dumps(
|
||||
{
|
||||
"event": "shoc_missing_sequence_number",
|
||||
"delivery_id": webhook_event["delivery_id"],
|
||||
}
|
||||
)
|
||||
)
|
||||
skipped += 1
|
||||
continue
|
||||
emitted += 1
|
||||
start = time.monotonic()
|
||||
try:
|
||||
outcome, status_code = delivery.deliver(webhook_event)
|
||||
latency_ms = int((time.monotonic() - start) * 1000)
|
||||
logger.info(_delivery_log(webhook_event, status_code, latency_ms, outcome))
|
||||
if outcome == "rejected":
|
||||
_park_rejected(webhook_event, status_code)
|
||||
except delivery.RetryableDeliveryError as exc:
|
||||
latency_ms = int((time.monotonic() - start) * 1000)
|
||||
logger.warning(
|
||||
_delivery_log(webhook_event, exc.status_code, latency_ms, "retryable")
|
||||
)
|
||||
logger.info(_batch_summary(len(records), emitted, skipped))
|
||||
# Stop here: reporting this record's sequence number makes the ESM
|
||||
# retry from it in order; earlier successes are not re-delivered.
|
||||
return {"batchItemFailures": [{"itemIdentifier": seq}]}
|
||||
except Exception:
|
||||
# Catch-all so an unexpected error (e.g. an SQS park failure)
|
||||
# cannot escape and fail the WHOLE invocation -- that would make
|
||||
# the ESM re-deliver every earlier success in the batch for up to
|
||||
# 24h. Report only THIS record so the ESM retries from it in order.
|
||||
logger.exception(
|
||||
json.dumps(
|
||||
{
|
||||
"event": "shoc_delivery_error",
|
||||
"delivery_id": webhook_event["delivery_id"],
|
||||
"event_type": webhook_event["event_type"],
|
||||
}
|
||||
)
|
||||
)
|
||||
logger.info(_batch_summary(len(records), emitted, skipped))
|
||||
return {"batchItemFailures": [{"itemIdentifier": seq}]}
|
||||
logger.info(_batch_summary(len(records), emitted, skipped))
|
||||
return {"batchItemFailures": []}
|
||||
|
|
@ -1,234 +0,0 @@
|
|||
"""Rotation Lambda for the SHOC webhook HMAC secret.
|
||||
|
||||
Secrets Manager invokes this on the 30-day rotation schedule for
|
||||
``workorder-ingest/shoc-webhook-hmac`` (docs/shoc-webhook-contract.md section
|
||||
6.1). Secret value shape::
|
||||
|
||||
{"keys": [{"kid": "<YYYY-MM-DDTHH>", "secret": "<64 hex chars>"}, ...]}
|
||||
|
||||
newest first, truncated to 2 entries (one overlap cycle). The emitter always
|
||||
signs with ``keys[0]``; the SHOC receiver accepts any listed ``kid``.
|
||||
|
||||
This is *single-user* rotation: no external system holds the value -- both
|
||||
sides re-fetch from Secrets Manager on their own <=5-minute cache TTLs -- so
|
||||
the standard 4-step protocol collapses to createSecret/finishSecret, with
|
||||
setSecret a logged no-op and testSecret validating the pending value's shape.
|
||||
|
||||
The secret material is machine-generated HMAC key bytes and is NEVER logged;
|
||||
structured logs carry only step names, version ids, and ``kid`` values.
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
import secrets
|
||||
from datetime import datetime, timezone
|
||||
|
||||
import boto3
|
||||
|
||||
logger = logging.getLogger()
|
||||
logger.setLevel(logging.INFO)
|
||||
|
||||
# One overlap cycle: the new key plus the previous key (contract section 6.1).
|
||||
MAX_KEYS = 2
|
||||
# 32 random bytes -> secrets.token_hex emits 64 lowercase hex chars.
|
||||
SECRET_NUM_BYTES = 32
|
||||
SECRET_HEX_PATTERN = re.compile(r"[0-9a-f]{64}")
|
||||
KID_FORMAT = "%Y-%m-%dT%H"
|
||||
# Forced re-rotation within the same hour needs a distinct kid.
|
||||
KID_FORMAT_INTRA_HOUR = "%Y-%m-%dT%H%M"
|
||||
|
||||
# Lazy cached Secrets Manager client. Keeps the public attribute name
|
||||
# ``secretsmanager_client`` so the test monkeypatch target is stable.
|
||||
secretsmanager_client = None
|
||||
|
||||
|
||||
def _get_client():
|
||||
global secretsmanager_client
|
||||
if secretsmanager_client is None:
|
||||
secretsmanager_client = boto3.client("secretsmanager")
|
||||
return secretsmanager_client
|
||||
|
||||
|
||||
def _current_keys(client, secret_id: str) -> list:
|
||||
"""Read the AWSCURRENT ``keys`` list, tolerating malformed prior values.
|
||||
|
||||
The CDK-seeded bootstrap value is ``{"keys": [], "bootstrap_entropy": ...}``;
|
||||
any unparseable or wrong-shape prior value likewise falls back to ``[]``
|
||||
(rotation then simply starts a fresh key list -- the material is fully
|
||||
regenerable, receivers refresh within their TTL).
|
||||
"""
|
||||
try:
|
||||
raw = client.get_secret_value(SecretId=secret_id, VersionStage="AWSCURRENT")
|
||||
parsed = json.loads(raw["SecretString"])
|
||||
except (client.exceptions.ResourceNotFoundException, json.JSONDecodeError):
|
||||
# Genuinely-absent or corrupt current value -> start a fresh list.
|
||||
# NB: this does NOT catch transient failures (throttling, KMS/IAM
|
||||
# blips): those re-raise so Secrets Manager marks the rotation failed
|
||||
# and retries with the prior AWSCURRENT intact, rather than silently
|
||||
# dropping the overlap key and stranding in-flight deliveries.
|
||||
logger.warning(
|
||||
json.dumps(
|
||||
{"event": "hmac_rotation_current_unreadable", "fallback": "empty"}
|
||||
)
|
||||
)
|
||||
return []
|
||||
keys = parsed.get("keys") if isinstance(parsed, dict) else None
|
||||
if not isinstance(keys, list) or not all(
|
||||
isinstance(key, dict)
|
||||
and isinstance(key.get("kid"), str)
|
||||
and isinstance(key.get("secret"), str)
|
||||
for key in keys
|
||||
):
|
||||
logger.warning(
|
||||
json.dumps(
|
||||
{"event": "hmac_rotation_current_malformed", "fallback": "empty"}
|
||||
)
|
||||
)
|
||||
return []
|
||||
return keys
|
||||
|
||||
|
||||
def _create_secret(client, secret_id: str, token: str) -> None:
|
||||
"""Stage a new AWSPENDING value: fresh key prepended, list truncated to 2."""
|
||||
versions = client.describe_secret(SecretId=secret_id).get("VersionIdsToStages", {})
|
||||
if "AWSCURRENT" in versions.get(token, []):
|
||||
logger.info(
|
||||
json.dumps(
|
||||
{"event": "hmac_rotation_create_already_current", "version_id": token}
|
||||
)
|
||||
)
|
||||
return
|
||||
try:
|
||||
client.get_secret_value(
|
||||
SecretId=secret_id, VersionId=token, VersionStage="AWSPENDING"
|
||||
)
|
||||
# This token's version already holds a value: retried createSecret
|
||||
# invocation, nothing to do.
|
||||
logger.info(
|
||||
json.dumps(
|
||||
{"event": "hmac_rotation_create_idempotent", "version_id": token}
|
||||
)
|
||||
)
|
||||
return
|
||||
except client.exceptions.ResourceNotFoundException:
|
||||
pass
|
||||
current_keys = _current_keys(client, secret_id)
|
||||
now = datetime.now(timezone.utc)
|
||||
existing_kids = {key["kid"] for key in current_keys}
|
||||
new_kid = now.strftime(KID_FORMAT)
|
||||
if new_kid in existing_kids:
|
||||
# Collision against ANY retained kid (not just keys[0]): repeated
|
||||
# same-hour forced rotations must never reissue a kid for a different
|
||||
# secret, or a receiver's cached kid->secret map desyncs. Append a
|
||||
# random suffix guaranteed distinct from the retained set.
|
||||
new_kid = f"{now.strftime(KID_FORMAT_INTRA_HOUR)}{secrets.token_hex(2)}"
|
||||
new_key = {"kid": new_kid, "secret": secrets.token_hex(SECRET_NUM_BYTES)}
|
||||
client.put_secret_value(
|
||||
SecretId=secret_id,
|
||||
ClientRequestToken=token,
|
||||
SecretString=json.dumps({"keys": [new_key] + current_keys[: MAX_KEYS - 1]}),
|
||||
VersionStages=["AWSPENDING"],
|
||||
)
|
||||
logger.info(
|
||||
json.dumps(
|
||||
{
|
||||
"event": "hmac_rotation_pending_staged",
|
||||
"version_id": token,
|
||||
"kid": new_kid,
|
||||
"key_count": 1 + len(current_keys[: MAX_KEYS - 1]),
|
||||
}
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _set_secret(client, secret_id: str, token: str) -> None:
|
||||
"""No-op: no external system holds the value (single-user rotation)."""
|
||||
logger.info(
|
||||
json.dumps({"event": "hmac_rotation_set_secret_noop", "version_id": token})
|
||||
)
|
||||
|
||||
|
||||
def _test_secret(client, secret_id: str, token: str) -> None:
|
||||
"""Validate the AWSPENDING value's shape; raise ValueError to fail rotation."""
|
||||
pending = client.get_secret_value(
|
||||
SecretId=secret_id, VersionId=token, VersionStage="AWSPENDING"
|
||||
)
|
||||
# json.JSONDecodeError is a ValueError subclass: bad JSON fails the rotation.
|
||||
parsed = json.loads(pending["SecretString"])
|
||||
keys = parsed.get("keys") if isinstance(parsed, dict) else None
|
||||
if not isinstance(keys, list) or not keys:
|
||||
raise ValueError("AWSPENDING value has no keys list")
|
||||
head = keys[0]
|
||||
if not isinstance(head, dict) or not isinstance(head.get("kid"), str):
|
||||
raise ValueError("AWSPENDING keys[0] has no string kid")
|
||||
secret_value = head.get("secret")
|
||||
if not isinstance(secret_value, str) or not SECRET_HEX_PATTERN.fullmatch(
|
||||
secret_value
|
||||
):
|
||||
raise ValueError("AWSPENDING keys[0].secret is not 64 lowercase hex chars")
|
||||
# Log only the rotation token, never a value pulled from the parsed
|
||||
# secret dict. kid is non-sensitive (it rides X-SH-Key-Id in the clear)
|
||||
# and is already logged at stage time in _create_secret, but subscripting
|
||||
# the secret-bearing dict here trips CodeQL's clear-text-logging taint
|
||||
# (py/clear-text-logging-sensitive-data) and is fragile if a later edit
|
||||
# swaps the field -- version_id already correlates this step to the stage.
|
||||
logger.info(json.dumps({"event": "hmac_rotation_test_ok", "version_id": token}))
|
||||
|
||||
|
||||
def _finish_secret(client, secret_id: str, token: str) -> None:
|
||||
"""Move AWSCURRENT to the pending version (idempotent on retry)."""
|
||||
versions = client.describe_secret(SecretId=secret_id).get("VersionIdsToStages", {})
|
||||
current_version_id = None
|
||||
for version_id, stages in versions.items():
|
||||
if "AWSCURRENT" in stages:
|
||||
current_version_id = version_id
|
||||
break
|
||||
if current_version_id == token:
|
||||
logger.info(
|
||||
json.dumps(
|
||||
{"event": "hmac_rotation_finish_idempotent", "version_id": token}
|
||||
)
|
||||
)
|
||||
return
|
||||
stage_move = {
|
||||
"SecretId": secret_id,
|
||||
"VersionStage": "AWSCURRENT",
|
||||
"MoveToVersionId": token,
|
||||
}
|
||||
if current_version_id is not None:
|
||||
stage_move["RemoveFromVersionId"] = current_version_id
|
||||
client.update_secret_version_stage(**stage_move)
|
||||
logger.info(
|
||||
json.dumps(
|
||||
{
|
||||
"event": "hmac_rotation_finished",
|
||||
"moved_to": token,
|
||||
"removed_from": current_version_id,
|
||||
}
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def handler(event, context):
|
||||
"""Secrets Manager rotation entry point.
|
||||
|
||||
``event`` = ``{"SecretId", "ClientRequestToken", "Step"}``; dispatches the
|
||||
standard rotation steps. Unknown steps raise (rotation marked failed).
|
||||
"""
|
||||
secret_id = event["SecretId"]
|
||||
token = event["ClientRequestToken"]
|
||||
step = event["Step"]
|
||||
logger.info(
|
||||
json.dumps({"event": "hmac_rotation_step", "step": step, "version_id": token})
|
||||
)
|
||||
client = _get_client()
|
||||
steps = {
|
||||
"createSecret": _create_secret,
|
||||
"setSecret": _set_secret,
|
||||
"testSecret": _test_secret,
|
||||
"finishSecret": _finish_secret,
|
||||
}
|
||||
if step not in steps:
|
||||
raise ValueError(f"Unknown rotation step: {step}")
|
||||
steps[step](client, secret_id, token)
|
||||
|
|
@ -1 +1 @@
|
|||
boto3>=1.43.55
|
||||
boto3>=1.43.53
|
||||
|
|
|
|||
|
|
@ -14,8 +14,6 @@ addopts =
|
|||
--cov=lambdas/po/site_extractor
|
||||
--cov=lambdas/api
|
||||
--cov=lambdas/shared
|
||||
--cov=lambdas/wo/shoc_emitter
|
||||
--cov=lambdas/wo/shoc_hmac_rotator
|
||||
--cov-report=term-missing
|
||||
--cov-fail-under=80
|
||||
; The 80% floor is an aggregate whole-suite gate, enforced in CI by the
|
||||
|
|
|
|||
|
|
@ -1,445 +0,0 @@
|
|||
"""
|
||||
Rebuild SHOC work-order webhook events from DynamoDB and re-POST them to a
|
||||
receiver endpoint (contract section 8 replay backstop, for the
|
||||
workorder-shoc-emitter-failures / -rejected alarm runbook).
|
||||
|
||||
Dry-run is the DEFAULT in every mode; nothing is POSTed unless --execute is
|
||||
passed. Validations (account gate, --since parsing, secret shape, unknown
|
||||
--work-order-id) run on dry-run too, so the rehearsal surfaces the same
|
||||
failures the real run would.
|
||||
|
||||
Semantics (deliberate divergences from the live emitter):
|
||||
|
||||
* Every envelope carries "replay": true.
|
||||
* Every WorkOrders state row is emitted as work_order.updated -- a replay
|
||||
reads current table state, not the stream, so it cannot distinguish the
|
||||
original created (or cancelled transition). The receiver upserts
|
||||
idempotently, so updated is always safe.
|
||||
* delivery_id is deterministic: "replay-" + sha256 over
|
||||
table#work_order_id#comment_id#occurred_at, truncated. Re-running the
|
||||
same replay yields the SAME ids -- receivers dedupe on delivery_id, so
|
||||
overlapping replay runs are harmless.
|
||||
* occurred_at is the row's updated_at (WO) / ingested_at (comment), falling
|
||||
back to now-UTC when absent.
|
||||
|
||||
Selection is exactly one of --work-order-id (repeatable; per-item get_item /
|
||||
Query) or --since (ISO-8601 UTC; a FULL TABLE Scan on WorkOrders filtered on
|
||||
updated_at >= since and on WorkOrderComments filtered on ingested_at >= since
|
||||
-- a count banner is printed per table). --events narrows to state rows,
|
||||
comments, or both (default).
|
||||
|
||||
Usage:
|
||||
# dry-run two work orders (state + comments)
|
||||
python scripts/replay_shoc_webhooks.py \
|
||||
--url https://api.dev.seahaven.com/api/webhooks/work-orders \
|
||||
--work-order-id 11144580730 --work-order-id 11144580731
|
||||
# replay everything touched since a timestamp (full table Scan)
|
||||
python scripts/replay_shoc_webhooks.py --url https://... \
|
||||
--since 2026-07-24T02:00:00Z --execute
|
||||
# comments only, for one work order
|
||||
python scripts/replay_shoc_webhooks.py --url https://... \
|
||||
--work-order-id 11144580730 --events comments --execute
|
||||
|
||||
--url is required with no default: replay must be a deliberate act against a
|
||||
known receiver. Signing mirrors the emitter's delivery module byte-for-byte
|
||||
(same string-to-sign, same header names; only the User-Agent gains a
|
||||
"-replay" suffix). Non-2xx responses are counted as failures; the script
|
||||
keeps going and exits 1 if any event failed.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import hmac
|
||||
import json
|
||||
import sys
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from decimal import Decimal
|
||||
|
||||
import boto3
|
||||
from boto3.dynamodb.conditions import Key
|
||||
|
||||
EXPECTED_ACCOUNT = "011934824531"
|
||||
REGION = "us-east-1"
|
||||
|
||||
WO_TABLE = "WorkOrders"
|
||||
COMMENTS_TABLE = "WorkOrderComments"
|
||||
SECRET_NAME = "workorder-ingest/shoc-webhook-hmac"
|
||||
|
||||
SCHEMA_VERSION = 1
|
||||
SOURCE = "procurement-ingest/workorder-shoc-emitter"
|
||||
USER_AGENT = "workorder-shoc-emitter/1-replay"
|
||||
POST_TIMEOUT_SECONDS = 10
|
||||
DELIVERY_ID_HASH_CHARS = 32
|
||||
|
||||
HTTP_2XX_MIN = 200
|
||||
HTTP_2XX_MAX = 300
|
||||
HTTP_TOO_MANY_REQUESTS = 429
|
||||
HTTP_5XX_MIN = 500
|
||||
|
||||
|
||||
class _NoRedirectHandler(urllib.request.HTTPRedirectHandler):
|
||||
"""Refuse to follow receiver redirects (matches the emitter's opener).
|
||||
|
||||
Following a 3xx would forward the live X-SH-* auth headers to a
|
||||
receiver-chosen Location and could downgrade an http:// target past the
|
||||
https-only --url check; a redirect is a receiver misconfiguration, so let
|
||||
it surface as an HTTPError status.
|
||||
"""
|
||||
|
||||
def redirect_request(self, *args, **kwargs):
|
||||
return None
|
||||
|
||||
|
||||
_opener = urllib.request.build_opener(_NoRedirectHandler)
|
||||
|
||||
# data payload field lists -- pinned to the emitter's envelope module. Absent
|
||||
# attributes are emitted as null. source_email_s3_key is deliberately
|
||||
# EXCLUDED (internal S3 pointer, never leaves the account).
|
||||
WO_DATA_FIELDS = [
|
||||
"work_order_id",
|
||||
"wo_status",
|
||||
"description",
|
||||
"customer",
|
||||
"site_code",
|
||||
"building",
|
||||
"address",
|
||||
"severity",
|
||||
"priority",
|
||||
"assigned_to",
|
||||
"date_reported",
|
||||
"scheduled_start",
|
||||
"due_date",
|
||||
"record_type",
|
||||
"created_at",
|
||||
"updated_at",
|
||||
]
|
||||
COMMENT_DATA_FIELDS = [
|
||||
"work_order_id",
|
||||
"comment_id",
|
||||
"record_type",
|
||||
"commenter",
|
||||
"text",
|
||||
"created_at",
|
||||
"ingested_at",
|
||||
]
|
||||
|
||||
|
||||
def parse_since(value):
|
||||
"""Parse the operator-supplied --since and re-emit the canonical
|
||||
zero-padded ``+00:00`` isoformat string that ``updated_at`` /
|
||||
``ingested_at`` are lexicographically comparable against (both are
|
||||
``datetime.now(timezone.utc).isoformat()`` strings). A raw operator
|
||||
string would compare WRONG under DynamoDB's byte-wise ``>=``, so parse
|
||||
strictly and require an aware UTC instant."""
|
||||
raw = value[:-1] + "+00:00" if value.endswith("Z") else value
|
||||
try:
|
||||
dt = datetime.fromisoformat(raw)
|
||||
except ValueError:
|
||||
sys.exit(
|
||||
f"ERROR: --since {value!r} is not ISO-8601 (e.g. 2026-07-24T02:00:00Z)."
|
||||
)
|
||||
if dt.tzinfo is None or dt.utcoffset() != timedelta(0):
|
||||
sys.exit(f"ERROR: --since {value!r} must be UTC (trailing 'Z' or '+00:00').")
|
||||
return dt.isoformat()
|
||||
|
||||
|
||||
def _json_safe(value):
|
||||
"""boto3's DynamoDB resource deserializes numbers as Decimal, which
|
||||
json.dumps rejects; convert recursively (defensive -- these fields are
|
||||
string-typed today)."""
|
||||
if isinstance(value, Decimal):
|
||||
return int(value) if value == value.to_integral_value() else float(value)
|
||||
if isinstance(value, list):
|
||||
return [_json_safe(v) for v in value]
|
||||
if isinstance(value, dict):
|
||||
return {k: _json_safe(v) for k, v in value.items()}
|
||||
return value
|
||||
|
||||
|
||||
def sign_body(secret_hex, timestamp, raw_body_bytes):
|
||||
"""HMAC-SHA256 over ``f"{timestamp}.{raw_body}"`` (contract section 6),
|
||||
computed over the raw UTF-8 body bytes exactly as the emitter's delivery
|
||||
module does -- the shared test vectors pin both implementations to
|
||||
identical output. Never log the secret."""
|
||||
string_to_sign = f"{timestamp}.".encode("utf-8") + raw_body_bytes
|
||||
mac = hmac.new(secret_hex.encode("utf-8"), string_to_sign, hashlib.sha256)
|
||||
return mac.hexdigest()
|
||||
|
||||
|
||||
def build_headers(kid, signature, timestamp):
|
||||
return {
|
||||
"Content-Type": "application/json; charset=utf-8",
|
||||
"User-Agent": USER_AGENT,
|
||||
"X-SH-Timestamp": str(timestamp),
|
||||
"X-SH-Key-Id": kid,
|
||||
"X-SH-Signature": f"v1={signature}",
|
||||
}
|
||||
|
||||
|
||||
def post_event(url, raw_body_bytes, kid, secret_hex):
|
||||
"""POST one signed envelope. Returns the HTTP status code, or None on
|
||||
timeout / connection error."""
|
||||
timestamp = int(time.time())
|
||||
signature = sign_body(secret_hex, timestamp, raw_body_bytes)
|
||||
request = urllib.request.Request(
|
||||
url,
|
||||
data=raw_body_bytes,
|
||||
headers=build_headers(kid, signature, timestamp),
|
||||
method="POST",
|
||||
)
|
||||
try:
|
||||
with _opener.open(request, timeout=POST_TIMEOUT_SECONDS) as resp:
|
||||
return resp.status
|
||||
except urllib.error.HTTPError as exc:
|
||||
return exc.code
|
||||
except (urllib.error.URLError, TimeoutError, OSError):
|
||||
return None
|
||||
|
||||
|
||||
def classify_response(status):
|
||||
"""Contract section 7 classification, printed per event for the operator
|
||||
(the replay tool itself never retries -- re-run it instead)."""
|
||||
if status is not None and HTTP_2XX_MIN <= status < HTTP_2XX_MAX:
|
||||
return "delivered"
|
||||
if status is None or status == HTTP_TOO_MANY_REQUESTS or status >= HTTP_5XX_MIN:
|
||||
return "retryable"
|
||||
return "non-retryable"
|
||||
|
||||
|
||||
def fetch_signing_key(session, secret_id):
|
||||
"""Fetch the HMAC secret once and return (kid, secret_hex) from keys[0]
|
||||
(the producer signing key). Secret material is never printed."""
|
||||
client = session.client("secretsmanager")
|
||||
value = json.loads(client.get_secret_value(SecretId=secret_id)["SecretString"])
|
||||
keys = value.get("keys") or []
|
||||
if not keys:
|
||||
sys.exit(
|
||||
f"ERROR: secret {secret_id!r} has no keys (bootstrap state); run the "
|
||||
"workorder-shoc-hmac-rotator rotation first."
|
||||
)
|
||||
return keys[0]["kid"], keys[0]["secret"]
|
||||
|
||||
|
||||
def replay_delivery_id(table, work_order_id, comment_id, occurred_at):
|
||||
"""Deterministic delivery_id, stable across replay runs, so receivers
|
||||
dedupe repeated replays on it."""
|
||||
seed = f"{table}#{work_order_id}#{comment_id}#{occurred_at}"
|
||||
digest = hashlib.sha256(seed.encode("utf-8")).hexdigest()
|
||||
return f"replay-{digest[:DELIVERY_ID_HASH_CHARS]}"
|
||||
|
||||
|
||||
def _envelope(event_type, delivery_id, occurred_at, data):
|
||||
return {
|
||||
"schema_version": SCHEMA_VERSION,
|
||||
"delivery_id": delivery_id,
|
||||
"event_type": event_type,
|
||||
"occurred_at": occurred_at,
|
||||
"source": SOURCE,
|
||||
"replay": True,
|
||||
"data": data,
|
||||
}
|
||||
|
||||
|
||||
def build_state_event(item):
|
||||
# Always work_order.updated: current-state replay cannot distinguish the
|
||||
# original created/cancelled, and the receiver upserts idempotently.
|
||||
occurred_at = item.get("updated_at") or datetime.now(timezone.utc).isoformat()
|
||||
return _envelope(
|
||||
"work_order.updated",
|
||||
replay_delivery_id(WO_TABLE, item["work_order_id"], "", occurred_at),
|
||||
occurred_at,
|
||||
{f: _json_safe(item.get(f)) for f in WO_DATA_FIELDS},
|
||||
)
|
||||
|
||||
|
||||
def build_comment_event(item):
|
||||
occurred_at = item.get("ingested_at") or datetime.now(timezone.utc).isoformat()
|
||||
delivery_id = replay_delivery_id(
|
||||
COMMENTS_TABLE, item["work_order_id"], item["comment_id"], occurred_at
|
||||
)
|
||||
return _envelope(
|
||||
"work_order.comment_added",
|
||||
delivery_id,
|
||||
occurred_at,
|
||||
{f: _json_safe(item.get(f)) for f in COMMENT_DATA_FIELDS},
|
||||
)
|
||||
|
||||
|
||||
def query_comments(table, work_order_id):
|
||||
kwargs = {"KeyConditionExpression": Key("work_order_id").eq(work_order_id)}
|
||||
while True:
|
||||
page = table.query(**kwargs)
|
||||
yield from page.get("Items", [])
|
||||
lek = page.get("LastEvaluatedKey")
|
||||
if not lek:
|
||||
return
|
||||
kwargs["ExclusiveStartKey"] = lek
|
||||
|
||||
|
||||
def scan_since(table, attr, since):
|
||||
kwargs = {
|
||||
"FilterExpression": "#a >= :since",
|
||||
"ExpressionAttributeNames": {"#a": attr},
|
||||
"ExpressionAttributeValues": {":since": since},
|
||||
}
|
||||
while True:
|
||||
page = table.scan(**kwargs)
|
||||
yield from page.get("Items", [])
|
||||
lek = page.get("LastEvaluatedKey")
|
||||
if not lek:
|
||||
return
|
||||
kwargs["ExclusiveStartKey"] = lek
|
||||
|
||||
|
||||
def select_by_ids(dynamodb, work_order_ids, want_state, want_comments):
|
||||
envelopes = []
|
||||
if want_state:
|
||||
table = dynamodb.Table(WO_TABLE)
|
||||
for wid in work_order_ids:
|
||||
item = table.get_item(Key={"work_order_id": wid}).get("Item")
|
||||
if item is None:
|
||||
sys.exit(f"ERROR: work order {wid!r} not found in {WO_TABLE}.")
|
||||
envelopes.append(build_state_event(item))
|
||||
if want_comments:
|
||||
table = dynamodb.Table(COMMENTS_TABLE)
|
||||
for wid in work_order_ids:
|
||||
envelopes.extend(build_comment_event(i) for i in query_comments(table, wid))
|
||||
return envelopes
|
||||
|
||||
|
||||
def select_since(dynamodb, since, want_state, want_comments):
|
||||
envelopes = []
|
||||
if want_state:
|
||||
table = dynamodb.Table(WO_TABLE)
|
||||
items = list(scan_since(table, "updated_at", since))
|
||||
print(
|
||||
f"[{WO_TABLE}] full table Scan (updated_at >= {since}): {len(items)} row(s)"
|
||||
)
|
||||
envelopes.extend(build_state_event(i) for i in items)
|
||||
if want_comments:
|
||||
table = dynamodb.Table(COMMENTS_TABLE)
|
||||
items = list(scan_since(table, "ingested_at", since))
|
||||
print(
|
||||
f"[{COMMENTS_TABLE}] full table Scan (ingested_at >= {since}): "
|
||||
f"{len(items)} row(s)"
|
||||
)
|
||||
envelopes.extend(build_comment_event(i) for i in items)
|
||||
return envelopes
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__)
|
||||
ap.add_argument(
|
||||
"--work-order-id",
|
||||
action="append",
|
||||
help="Replay this work order (repeatable). Exactly one of "
|
||||
"--work-order-id or --since is required.",
|
||||
)
|
||||
ap.add_argument(
|
||||
"--since",
|
||||
help="Replay every row with updated_at/ingested_at >= this ISO-8601 "
|
||||
"UTC timestamp (FULL TABLE Scan on both tables).",
|
||||
)
|
||||
ap.add_argument(
|
||||
"--events",
|
||||
choices=["state", "comments", "both"],
|
||||
default="both",
|
||||
help="Which event families to replay (default: both).",
|
||||
)
|
||||
ap.add_argument(
|
||||
"--url",
|
||||
required=True,
|
||||
help="Receiver endpoint URL. Required, no default -- replay must be "
|
||||
"deliberate.",
|
||||
)
|
||||
ap.add_argument(
|
||||
"--secret-arn",
|
||||
default=SECRET_NAME,
|
||||
help=f"HMAC secret to sign with (default: resolve {SECRET_NAME!r} by name).",
|
||||
)
|
||||
ap.add_argument("--profile", help="AWS profile (default: seahaven-prod).")
|
||||
ap.add_argument(
|
||||
"--execute",
|
||||
action="store_true",
|
||||
help="Actually POST. Default is dry-run (list only).",
|
||||
)
|
||||
args = ap.parse_args()
|
||||
|
||||
if bool(args.work_order_id) == bool(args.since):
|
||||
sys.exit("ERROR: pass exactly one of --work-order-id or --since.")
|
||||
|
||||
if not args.url.startswith("https://"):
|
||||
# urllib follows file:// and http:// too; the feed is HTTPS-only.
|
||||
sys.exit("ERROR: --url must be an https:// URL.")
|
||||
|
||||
session = boto3.Session(
|
||||
profile_name=args.profile or "seahaven-prod", region_name=REGION
|
||||
)
|
||||
acct = session.client("sts").get_caller_identity()["Account"]
|
||||
if acct != EXPECTED_ACCOUNT:
|
||||
sys.exit(
|
||||
f"ERROR: profile resolves to account {acct}, expected "
|
||||
f"{EXPECTED_ACCOUNT}. Aborting."
|
||||
)
|
||||
|
||||
want_state = args.events in ("state", "both")
|
||||
want_comments = args.events in ("comments", "both")
|
||||
dynamodb = session.resource("dynamodb")
|
||||
if args.work_order_id:
|
||||
events = select_by_ids(dynamodb, args.work_order_id, want_state, want_comments)
|
||||
else:
|
||||
since = parse_since(args.since)
|
||||
events = select_since(dynamodb, since, want_state, want_comments)
|
||||
|
||||
# Fetch on dry-run too: an empty/missing secret should surface on the
|
||||
# rehearsal, not first appear when the operator adds --execute.
|
||||
kid, secret_hex = fetch_signing_key(session, args.secret_arn)
|
||||
|
||||
if not events:
|
||||
print("No matching events -- nothing to replay.")
|
||||
return
|
||||
|
||||
# In-order per work order: state and comments interleave by occurred_at
|
||||
# (receiver upserts + skeleton-upserts, so cross-family order is a
|
||||
# nicety, not a requirement).
|
||||
events.sort(
|
||||
key=lambda e: (e["data"]["work_order_id"], e["occurred_at"], e["event_type"])
|
||||
)
|
||||
state_count = sum(1 for e in events if e["event_type"] == "work_order.updated")
|
||||
print(
|
||||
f"{len(events)} event(s) selected ({state_count} state, "
|
||||
f"{len(events) - state_count} comment) -> {args.url}\n"
|
||||
)
|
||||
|
||||
failed = 0
|
||||
for event in events:
|
||||
wid = event["data"]["work_order_id"]
|
||||
if not args.execute:
|
||||
print(
|
||||
f"[{event['delivery_id']}] DRY-RUN: would POST {event['event_type']} wo={wid}"
|
||||
)
|
||||
continue
|
||||
raw_body = json.dumps(event).encode("utf-8")
|
||||
status = post_event(args.url, raw_body, kid, secret_hex)
|
||||
outcome = classify_response(status)
|
||||
status_str = "conn-error/timeout" if status is None else str(status)
|
||||
print(
|
||||
f"[{event['delivery_id']}] POST {event['event_type']} wo={wid} -> "
|
||||
f"{status_str} ({outcome})"
|
||||
)
|
||||
if outcome != "delivered":
|
||||
failed += 1
|
||||
|
||||
if not args.execute:
|
||||
print("\nDry run complete. Re-run with --execute to POST.")
|
||||
return
|
||||
print(f"\n{len(events) - failed} delivered, {failed} failed.")
|
||||
if failed:
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
|
@ -1,128 +0,0 @@
|
|||
#!/usr/bin/env bash
|
||||
###############################################################################
|
||||
# setup_procurement_api_domain.sh
|
||||
#
|
||||
# One-time (idempotent) wiring for the procurement-api custom domain
|
||||
# (procurement-api.seahaven.com). CROSS-ACCOUNT: the API + ACM cert live in
|
||||
# seahaven-prod (011934824531), but the seahaven.com public zone lives in the
|
||||
# mgmt account (328440206208), so the cert's DNS-validation record and the
|
||||
# final A-alias are added to the mgmt zone.
|
||||
#
|
||||
# Order of operations:
|
||||
# 1. ./setup_procurement_api_domain.sh cert
|
||||
# - requests (or reuses) the ACM cert in prod, us-east-1
|
||||
# - adds its DNS-validation CNAME to the mgmt seahaven.com zone
|
||||
# - waits for ISSUED, then writes the cert ARN to prod SSM
|
||||
# (/procurement-api/custom-domain/certificate-arn)
|
||||
# 2. deploy the procurement-api stack (cdk deploy procurement-api) -- it
|
||||
# reads the SSM param and creates the API Gateway DomainName + mapping
|
||||
# 3. ./setup_procurement_api_domain.sh alias
|
||||
# - reads the stack's regional alias target from the outputs
|
||||
# - adds the A-alias (procurement-api.seahaven.com -> API GW) to the
|
||||
# mgmt zone
|
||||
#
|
||||
# Requires SSO sessions for BOTH profiles (prod for ACM/SSM, mgmt for Route53).
|
||||
###############################################################################
|
||||
set -euo pipefail
|
||||
|
||||
DOMAIN="procurement-api.seahaven.com"
|
||||
REGION="us-east-1"
|
||||
PROD_PROFILE="${PROD_PROFILE:-seahaven-prod}"
|
||||
MGMT_PROFILE="${MGMT_PROFILE:-seahaven-mgmt}"
|
||||
PROD_ACCOUNT="011934824531"
|
||||
MGMT_ACCOUNT="328440206208"
|
||||
ZONE_ID="Z06652411XKH89KTZD3XA" # seahaven.com public zone, in the mgmt account
|
||||
SSM_PARAM="/procurement-api/custom-domain/certificate-arn"
|
||||
STACK_NAME="procurement-api"
|
||||
|
||||
_verify_account() {
|
||||
local profile="$1" expected="$2"
|
||||
local got
|
||||
got="$(aws sts get-caller-identity --profile "${profile}" --query Account --output text)"
|
||||
if [[ "${got}" != "${expected}" ]]; then
|
||||
echo "ERROR: profile ${profile} resolves to ${got}, expected ${expected}. Aborting." >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
cmd_cert() {
|
||||
_verify_account "${PROD_PROFILE}" "${PROD_ACCOUNT}"
|
||||
_verify_account "${MGMT_PROFILE}" "${MGMT_ACCOUNT}"
|
||||
|
||||
echo "==> Finding or requesting ACM cert for ${DOMAIN} in ${PROD_ACCOUNT}/${REGION}"
|
||||
local cert_arn
|
||||
cert_arn="$(aws acm list-certificates --profile "${PROD_PROFILE}" --region "${REGION}" \
|
||||
--query "CertificateSummaryList[?DomainName=='${DOMAIN}'].CertificateArn | [0]" --output text)"
|
||||
if [[ "${cert_arn}" == "None" || -z "${cert_arn}" ]]; then
|
||||
cert_arn="$(aws acm request-certificate --profile "${PROD_PROFILE}" --region "${REGION}" \
|
||||
--domain-name "${DOMAIN}" --validation-method DNS \
|
||||
--query CertificateArn --output text)"
|
||||
echo " requested ${cert_arn}; waiting for the validation record to populate..."
|
||||
sleep 8
|
||||
else
|
||||
echo " reusing existing ${cert_arn}"
|
||||
fi
|
||||
|
||||
echo "==> Reading the DNS-validation record"
|
||||
local rec_name rec_value
|
||||
rec_name="$(aws acm describe-certificate --profile "${PROD_PROFILE}" --region "${REGION}" \
|
||||
--certificate-arn "${cert_arn}" \
|
||||
--query "Certificate.DomainValidationOptions[0].ResourceRecord.Name" --output text)"
|
||||
rec_value="$(aws acm describe-certificate --profile "${PROD_PROFILE}" --region "${REGION}" \
|
||||
--certificate-arn "${cert_arn}" \
|
||||
--query "Certificate.DomainValidationOptions[0].ResourceRecord.Value" --output text)"
|
||||
|
||||
echo "==> Upserting validation CNAME in the mgmt seahaven.com zone"
|
||||
aws route53 change-resource-record-sets --profile "${MGMT_PROFILE}" \
|
||||
--hosted-zone-id "${ZONE_ID}" --change-batch "$(cat <<JSON
|
||||
{"Changes":[{"Action":"UPSERT","ResourceRecordSet":{
|
||||
"Name":"${rec_name}","Type":"CNAME","TTL":300,
|
||||
"ResourceRecords":[{"Value":"${rec_value}"}]}}]}
|
||||
JSON
|
||||
)" >/dev/null
|
||||
|
||||
echo "==> Waiting for cert to reach ISSUED (can take a few minutes)"
|
||||
aws acm wait certificate-validated --profile "${PROD_PROFILE}" --region "${REGION}" \
|
||||
--certificate-arn "${cert_arn}"
|
||||
|
||||
echo "==> Writing cert ARN to prod SSM ${SSM_PARAM}"
|
||||
aws ssm put-parameter --profile "${PROD_PROFILE}" --region "${REGION}" \
|
||||
--name "${SSM_PARAM}" --type String --overwrite --value "${cert_arn}" >/dev/null
|
||||
|
||||
echo "OK: cert ISSUED and SSM param set. Now: cdk deploy ${STACK_NAME}, then '$0 alias'."
|
||||
}
|
||||
|
||||
cmd_alias() {
|
||||
_verify_account "${PROD_PROFILE}" "${PROD_ACCOUNT}"
|
||||
_verify_account "${MGMT_PROFILE}" "${MGMT_ACCOUNT}"
|
||||
|
||||
echo "==> Reading regional alias target from the ${STACK_NAME} stack outputs"
|
||||
local target zone
|
||||
target="$(aws cloudformation describe-stacks --profile "${PROD_PROFILE}" --region "${REGION}" \
|
||||
--stack-name "${STACK_NAME}" \
|
||||
--query "Stacks[0].Outputs[?OutputKey=='ProcurementApiAliasTarget'].OutputValue | [0]" --output text)"
|
||||
zone="$(aws cloudformation describe-stacks --profile "${PROD_PROFILE}" --region "${REGION}" \
|
||||
--stack-name "${STACK_NAME}" \
|
||||
--query "Stacks[0].Outputs[?OutputKey=='ProcurementApiAliasHostedZoneId'].OutputValue | [0]" --output text)"
|
||||
if [[ -z "${target}" || "${target}" == "None" ]]; then
|
||||
echo "ERROR: no alias target output; deploy the stack first." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "==> Upserting A-alias ${DOMAIN} -> ${target} in the mgmt zone"
|
||||
aws route53 change-resource-record-sets --profile "${MGMT_PROFILE}" \
|
||||
--hosted-zone-id "${ZONE_ID}" --change-batch "$(cat <<JSON
|
||||
{"Changes":[{"Action":"UPSERT","ResourceRecordSet":{
|
||||
"Name":"${DOMAIN}","Type":"A",
|
||||
"AliasTarget":{"DNSName":"${target}","HostedZoneId":"${zone}","EvaluateTargetHealth":false}}}]}
|
||||
JSON
|
||||
)" >/dev/null
|
||||
|
||||
echo "OK: A-alias set. Verify: curl -sS -o /dev/null -w '%{http_code}\\n' https://${DOMAIN}/docs (expect 401 without a token)."
|
||||
}
|
||||
|
||||
case "${1:-}" in
|
||||
cert) cmd_cert ;;
|
||||
alias) cmd_alias ;;
|
||||
*) echo "usage: $0 {cert|alias}" >&2; exit 2 ;;
|
||||
esac
|
||||
|
|
@ -66,13 +66,6 @@ _SIBLING_MODULES = (
|
|||
"extraction",
|
||||
"enrichment",
|
||||
"persistence",
|
||||
# SHOC emitter siblings (lambdas/wo/shoc_emitter/): envelope and delivery
|
||||
# are both standalone (no sibling deps, stdlib + boto3 only), so appending
|
||||
# them at the end keeps the tuple dependency-topological; the order between
|
||||
# the two is irrelevant. They exist only under wo/shoc_emitter/, so every
|
||||
# other handler load skips them via the exists() check.
|
||||
"envelope",
|
||||
"delivery",
|
||||
)
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -130,47 +130,6 @@ def test_docs_served_when_authenticated(api, monkeypatch):
|
|||
assert json.loads(spec["body"])["openapi"] == "3.1.0"
|
||||
|
||||
|
||||
def test_base_url_custom_domain_omits_stage(api):
|
||||
mod, _ = api
|
||||
# execute-api host keeps the /{stage} segment...
|
||||
execapi = {
|
||||
"requestContext": {
|
||||
"domainName": "mvul1efda2.execute-api.us-east-1.amazonaws.com",
|
||||
"stage": "prod",
|
||||
}
|
||||
}
|
||||
assert mod._base_url(execapi) == (
|
||||
"https://mvul1efda2.execute-api.us-east-1.amazonaws.com/prod"
|
||||
)
|
||||
# ...but a custom domain (base-path mapping) serves the stage at the root,
|
||||
# so no /{stage} is appended (it would advertise a broken server URL).
|
||||
custom = {
|
||||
"requestContext": {
|
||||
"domainName": "procurement-api.seahaven.com",
|
||||
"stage": "prod",
|
||||
}
|
||||
}
|
||||
assert mod._base_url(custom) == "https://procurement-api.seahaven.com"
|
||||
assert mod._base_url({}) is None
|
||||
|
||||
|
||||
def test_spec_injects_custom_domain_server(api):
|
||||
mod, _ = api
|
||||
event = {
|
||||
"requestContext": {
|
||||
"domainName": "procurement-api.seahaven.com",
|
||||
"stage": "prod",
|
||||
}
|
||||
}
|
||||
spec = json.loads(mod._spec_for_request(event))
|
||||
urls = [s["url"] for s in spec["servers"]]
|
||||
# Accessed via the custom domain, every server URL resolves to the custom
|
||||
# domain (the static entry plus the injected one), and none carry a /prod
|
||||
# suffix. Exact-equality checks (no substring-in-URL) so this reads
|
||||
# cleanly and doesn't trip CodeQL's URL-sanitization heuristic.
|
||||
assert set(urls) == {"https://procurement-api.seahaven.com"}
|
||||
|
||||
|
||||
def test_token_query_shim_synthesizes_header(api, monkeypatch):
|
||||
mod, _ = api
|
||||
seen = {}
|
||||
|
|
|
|||
|
|
@ -22,8 +22,6 @@ REPO_ROOT = Path(__file__).resolve().parents[1]
|
|||
PO_HANDLER = REPO_ROOT / "lambdas" / "po" / "email_processor" / "handler.py"
|
||||
WO_HANDLER = REPO_ROOT / "lambdas" / "wo" / "email_processor" / "handler.py"
|
||||
API_HANDLER = REPO_ROOT / "lambdas" / "api" / "handler.py"
|
||||
SHOC_EMITTER_HANDLER = REPO_ROOT / "lambdas" / "wo" / "shoc_emitter" / "handler.py"
|
||||
SHOC_ROTATOR_HANDLER = REPO_ROOT / "lambdas" / "wo" / "shoc_hmac_rotator" / "handler.py"
|
||||
PO_STACK = REPO_ROOT / "cdk" / "po_stack.py"
|
||||
WO_STACK = REPO_ROOT / "cdk" / "wo_stack.py"
|
||||
API_STACK = REPO_ROOT / "cdk" / "procurement_api_stack.py"
|
||||
|
|
@ -118,15 +116,6 @@ API_DATA_FILES_CP_RES = (
|
|||
r"(?:^|\s)api/fonts\.css(?:\s|$)",
|
||||
)
|
||||
|
||||
# SHOC webhook emitter + HMAC rotator (lambdas/wo/shoc_emitter|shoc_hmac_rotator,
|
||||
# bundled by cdk/wo_stack.py's _add_shoc_webhook_emitter). Same exact-set
|
||||
# discipline as the other bundles: `cp wo/shoc_emitter/*.py` /
|
||||
# `cp wo/shoc_hmac_rotator/*.py` ship every top-level .py in those dirs
|
||||
# (untracked strays included -- Code.from_asset does not honor .gitignore), so
|
||||
# any new module is a deliberate addition to these pins.
|
||||
SHOC_EMITTER_MODULES = frozenset({"__init__", "handler", "envelope", "delivery"})
|
||||
SHOC_ROTATOR_MODULES = frozenset({"__init__", "handler"})
|
||||
|
||||
# Pipeline-scoped glob shapes the per-stack ships-all pins accept. Phase 2
|
||||
# widened the bundling cwd to the shared ../lambdas asset root, so the executed
|
||||
# glob carries its own pipeline's path prefix (`po/email_processor/*.py`); a
|
||||
|
|
@ -779,158 +768,3 @@ def test_api_bundle_stages_spec_and_docs_page():
|
|||
f"cdk/procurement_api_stack.py bundling command must EXECUTE a cp "
|
||||
f"matching {pattern!r}. Executed cp commands: {executed_cps}"
|
||||
)
|
||||
|
||||
|
||||
def _extract_shoc_emitter_command(stack_path: Path) -> str:
|
||||
"""Extract the SHOC emitter's bash -c bundling command string.
|
||||
|
||||
Mirrors _extract_api_command but selects the command whose text mentions
|
||||
``shoc_emitter`` (its cp is ``cp wo/shoc_emitter/*.py``). The plan's CI
|
||||
note (docs/shoc-webhook-plan.md Phase 3) is explicit that an unrecognized
|
||||
bundle ships unchecked -- the email/web_ui selectors deliberately do not
|
||||
match the emitter command, so it needs its own selector. Demands exactly
|
||||
one match so an ambiguity surfaces loudly instead of silently checking the
|
||||
wrong command.
|
||||
"""
|
||||
tree = ast.parse(stack_path.read_text())
|
||||
commands: list[str] = []
|
||||
for node in ast.walk(tree):
|
||||
if isinstance(node, ast.keyword) and node.arg == "command":
|
||||
list_node = node.value
|
||||
if isinstance(list_node, ast.List) and list_node.elts:
|
||||
last = list_node.elts[-1]
|
||||
if isinstance(last, ast.Constant) and isinstance(last.value, str):
|
||||
commands.append(last.value)
|
||||
shoc_cmds = [c for c in commands if "shoc_emitter" in c]
|
||||
if len(shoc_cmds) != 1:
|
||||
raise AssertionError(
|
||||
f"expected exactly one shoc_emitter bundling command=[...] list in "
|
||||
f"{stack_path}, found {len(shoc_cmds)} (of {len(commands)} total "
|
||||
"command lists) — update this test to select the intended bundling "
|
||||
"command explicitly"
|
||||
)
|
||||
return shoc_cmds[0]
|
||||
|
||||
|
||||
def _extract_shoc_rotator_command(stack_path: Path) -> str:
|
||||
"""Extract the SHOC HMAC rotator's bash -c bundling command string.
|
||||
|
||||
Mirrors _extract_shoc_emitter_command with the ``shoc_hmac_rotator``
|
||||
selector (its cp is ``cp wo/shoc_hmac_rotator/*.py``); the emitter command
|
||||
does not contain that substring, so exactly-one holds.
|
||||
"""
|
||||
tree = ast.parse(stack_path.read_text())
|
||||
commands: list[str] = []
|
||||
for node in ast.walk(tree):
|
||||
if isinstance(node, ast.keyword) and node.arg == "command":
|
||||
list_node = node.value
|
||||
if isinstance(list_node, ast.List) and list_node.elts:
|
||||
last = list_node.elts[-1]
|
||||
if isinstance(last, ast.Constant) and isinstance(last.value, str):
|
||||
commands.append(last.value)
|
||||
rotator_cmds = [c for c in commands if "shoc_hmac_rotator" in c]
|
||||
if len(rotator_cmds) != 1:
|
||||
raise AssertionError(
|
||||
f"expected exactly one shoc_hmac_rotator bundling command=[...] list "
|
||||
f"in {stack_path}, found {len(rotator_cmds)} (of {len(commands)} "
|
||||
"total command lists) — update this test to select the intended "
|
||||
"bundling command explicitly"
|
||||
)
|
||||
return rotator_cmds[0]
|
||||
|
||||
|
||||
def test_shoc_emitter_bundling_ships_all_first_party_siblings():
|
||||
"""The SHOC emitter bundle must ship every sibling handler.py imports.
|
||||
|
||||
Same PR #105 ImportError class as the other bundles: the emitter handler
|
||||
imports envelope and delivery as bare-name flat siblings
|
||||
(lambdas/wo/shoc_emitter/); a narrowed cp would cold-start ImportError on
|
||||
the first stream invocation with green CI.
|
||||
"""
|
||||
required = _first_party_sibling_imports(SHOC_EMITTER_HANDLER)
|
||||
# Sanity: the emitter handler is known to import envelope + delivery. If
|
||||
# this ever collapses to empty, ships-all would vacuously pass.
|
||||
assert required == {"envelope", "delivery"}, (
|
||||
f"expected the emitter handler's first-party siblings to be envelope + "
|
||||
f"delivery, found {sorted(required)} — update this pin deliberately"
|
||||
)
|
||||
command = _extract_shoc_emitter_command(WO_STACK)
|
||||
assert _bundling_ships_all(command, required), (
|
||||
f"cdk/wo_stack.py shoc_emitter bundling does not ship all first-party "
|
||||
f"siblings {sorted(required)}. Executed cp commands: "
|
||||
f"{_executed_cp_commands(command)}"
|
||||
)
|
||||
|
||||
|
||||
def test_shoc_rotator_bundling_ships_handler():
|
||||
"""The rotator bundle must ship handler.py (it has no first-party siblings).
|
||||
|
||||
The rotator is stdlib + boto3-from-runtime only, so its required sibling
|
||||
set is empty -- pin that fact (a future sibling import must extend this
|
||||
test), then prove the executed glob actually ships handler.py itself.
|
||||
"""
|
||||
required = _first_party_sibling_imports(SHOC_ROTATOR_HANDLER)
|
||||
assert required == set(), (
|
||||
f"the rotator handler grew first-party sibling imports "
|
||||
f"{sorted(required)} — extend this ships-all test to require them"
|
||||
)
|
||||
command = _extract_shoc_rotator_command(WO_STACK)
|
||||
assert _bundling_ships_all(command, {"handler"}), (
|
||||
f"cdk/wo_stack.py shoc_hmac_rotator bundling does not ship handler.py. "
|
||||
f"Executed cp commands: {_executed_cp_commands(command)}"
|
||||
)
|
||||
|
||||
|
||||
def test_shoc_emitter_dir_ships_no_unexpected_top_level_modules():
|
||||
"""Exact-set pin on lambdas/wo/shoc_emitter/*.py -- both bounds.
|
||||
|
||||
`cp wo/shoc_emitter/*.py` ships every top-level .py in the dir (untracked
|
||||
strays included, since Code.from_asset does not honor .gitignore), so a
|
||||
stray scratch/secrets module would silently ship into the production zip.
|
||||
Any new module must be deliberately added to SHOC_EMITTER_MODULES.
|
||||
"""
|
||||
top_level = {p.stem for p in SHOC_EMITTER_HANDLER.parent.glob("*.py")}
|
||||
assert top_level == set(SHOC_EMITTER_MODULES), (
|
||||
f"lambdas/wo/shoc_emitter/ top-level modules {sorted(top_level)} != "
|
||||
f"pinned {sorted(SHOC_EMITTER_MODULES)}. If a new module is intended, "
|
||||
"add it to SHOC_EMITTER_MODULES."
|
||||
)
|
||||
|
||||
|
||||
def test_shoc_rotator_dir_ships_no_unexpected_top_level_modules():
|
||||
"""Exact-set pin on lambdas/wo/shoc_hmac_rotator/*.py -- both bounds.
|
||||
|
||||
Same rationale as the emitter pin: `cp wo/shoc_hmac_rotator/*.py` ships
|
||||
every top-level .py in the dir, strays included.
|
||||
"""
|
||||
top_level = {p.stem for p in SHOC_ROTATOR_HANDLER.parent.glob("*.py")}
|
||||
assert top_level == set(SHOC_ROTATOR_MODULES), (
|
||||
f"lambdas/wo/shoc_hmac_rotator/ top-level modules {sorted(top_level)} "
|
||||
f"!= pinned {sorted(SHOC_ROTATOR_MODULES)}. If a new module is "
|
||||
"intended, add it to SHOC_ROTATOR_MODULES."
|
||||
)
|
||||
|
||||
|
||||
def test_shoc_commands_do_not_collide_with_existing_selectors():
|
||||
"""The SHOC bundling commands must not trip the other exactly-one selectors.
|
||||
|
||||
_extract_bundling_command selects on the substring ``email_processor`` and
|
||||
_extract_web_ui_command on ``web_ui``, each demanding exactly one match in
|
||||
wo_stack.py. If either SHOC command ever grew one of those substrings
|
||||
(e.g. a copy-pasted comment folded into the command string), those
|
||||
selectors would find two commands and every email/web_ui pin would error.
|
||||
Assert the invariant here so the failure names the actual cause.
|
||||
"""
|
||||
for command in (
|
||||
_extract_shoc_emitter_command(WO_STACK),
|
||||
_extract_shoc_rotator_command(WO_STACK),
|
||||
):
|
||||
assert "email_processor" not in command, (
|
||||
f"SHOC bundling command contains 'email_processor', which would "
|
||||
f"break _extract_bundling_command's exactly-one selection: "
|
||||
f"{command!r}"
|
||||
)
|
||||
assert "web_ui" not in command, (
|
||||
f"SHOC bundling command contains 'web_ui', which would break "
|
||||
f"_extract_web_ui_command's exactly-one selection: {command!r}"
|
||||
)
|
||||
|
|
|
|||
|
|
@ -1,57 +0,0 @@
|
|||
"""Pin the cross-account principal surface of the CDK app.
|
||||
|
||||
The SHOC integration deliberately trusts EXACTLY ONE foreign principal:
|
||||
``arn:aws:iam::396287094661:role/shoc-backend-dev`` (read API resource
|
||||
policy in procurement_api_stack.py, HMAC secret + KMS grants in
|
||||
wo_stack.py). Future shoc-backend-staging/-prod roles are each a
|
||||
deliberate, individually-reviewed policy addition — so any new foreign
|
||||
account id or role ARN appearing in cdk/ must consciously update this
|
||||
pin (and go through the mandatory GPT-4.1 cross-family IAM review).
|
||||
|
||||
Raised as a QUESTION in the 2026-07-24 cross-family review of the
|
||||
webhook emitter policy surface: "how is the exact-one-principal
|
||||
invariant enforced over time?" — this test is the answer.
|
||||
"""
|
||||
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
CDK_DIR = REPO_ROOT / "cdk"
|
||||
|
||||
# The one foreign principal the app may reference, and the only files
|
||||
# allowed to reference it.
|
||||
ALLOWED_FOREIGN_PRINCIPAL = "arn:aws:iam::396287094661:role/shoc-backend-dev"
|
||||
ALLOWED_FILES = {"procurement_api_stack.py", "wo_stack.py"}
|
||||
|
||||
# Accounts that are not "foreign": seahaven-prod (the deploy target).
|
||||
HOME_ACCOUNTS = {"011934824531"}
|
||||
|
||||
_IAM_ARN_RE = re.compile(r"arn:aws:iam::(\d{12}):\S*?(?=[\"'\s])")
|
||||
|
||||
|
||||
def _cdk_sources():
|
||||
return sorted(CDK_DIR.glob("*.py"))
|
||||
|
||||
|
||||
def test_only_the_pinned_foreign_principal_appears_in_cdk_sources():
|
||||
findings = []
|
||||
for path in _cdk_sources():
|
||||
for match in _IAM_ARN_RE.finditer(path.read_text()):
|
||||
account = match.group(1)
|
||||
if account in HOME_ACCOUNTS:
|
||||
continue
|
||||
findings.append((path.name, match.group(0)))
|
||||
|
||||
unexpected = [
|
||||
(name, arn)
|
||||
for name, arn in findings
|
||||
if arn != ALLOWED_FOREIGN_PRINCIPAL or name not in ALLOWED_FILES
|
||||
]
|
||||
assert not unexpected, (
|
||||
"Unexpected foreign IAM principal(s) in cdk/ — every cross-account "
|
||||
f"trust addition must update this pin deliberately: {unexpected}"
|
||||
)
|
||||
# Both grant sites must still reference the pinned role (deleting one
|
||||
# half of the secret/KMS grant pair fails silently at the receiver).
|
||||
assert {name for name, _ in findings} == ALLOWED_FILES
|
||||
|
|
@ -1,222 +0,0 @@
|
|||
"""Contract tests for scripts/replay_shoc_webhooks.py (plan Phase 5).
|
||||
|
||||
Pins the replay tool's operator-safety and envelope contracts: dry-run is
|
||||
the default and performs NO HTTP, the sts account gate rejects any profile
|
||||
that does not resolve to seahaven-prod, the envelope data field lists stay
|
||||
byte-identical to the emitter's (the receiver sees one schema regardless of
|
||||
path), every replay envelope carries "replay": true, and delivery_id is
|
||||
deterministic across runs (receivers dedupe overlapping replays on it).
|
||||
|
||||
The script is loaded by file path (scripts/ is not a package -- mirrors
|
||||
tests/test_reprocess_contract.py); boto3.Session is monkeypatched to a plain
|
||||
fake so main() runs fully offline.
|
||||
"""
|
||||
|
||||
import importlib.util
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
import pytest
|
||||
|
||||
from tests.support import load_lambda_module
|
||||
|
||||
# replay builds its boto3 session inside main(), so import is side-effect-free.
|
||||
_p = pathlib.Path(__file__).resolve().parents[1] / "scripts" / "replay_shoc_webhooks.py"
|
||||
_spec = importlib.util.spec_from_file_location("replay_shoc_webhooks", _p)
|
||||
replay = importlib.util.module_from_spec(_spec)
|
||||
_spec.loader.exec_module(replay)
|
||||
|
||||
WO_ITEM = {
|
||||
"work_order_id": "11144580730",
|
||||
"wo_status": "assigned",
|
||||
"record_type": "new_work_order",
|
||||
"updated_at": "2026-07-01T00:00:00+00:00",
|
||||
"source_email_s3_key": "inbound/2026/07/abc.eml",
|
||||
}
|
||||
COMMENT_ITEM = {
|
||||
"work_order_id": "11144580730",
|
||||
"comment_id": "11144580730#2026-04-27T23:51:48#a1b2c3d4e5f6",
|
||||
"record_type": "comment",
|
||||
"commenter": "APM Technician",
|
||||
"text": "Vendor dispatched.",
|
||||
"created_at": "2026-04-27T23:51:48",
|
||||
"ingested_at": "2026-07-02T00:00:00+00:00",
|
||||
}
|
||||
|
||||
|
||||
# --- Offline fakes for main() ------------------------------------------------
|
||||
|
||||
|
||||
class _FakeSTS:
|
||||
def __init__(self, account):
|
||||
self.account = account
|
||||
|
||||
def get_caller_identity(self):
|
||||
return {"Account": self.account}
|
||||
|
||||
|
||||
class _FakeSecrets:
|
||||
def get_secret_value(self, SecretId): # noqa: N803 (boto3 kwarg name)
|
||||
return {
|
||||
"SecretString": json.dumps(
|
||||
{"keys": [{"kid": "2026-07-20T00", "secret": "a" * 64}]}
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
class _FakeWOTable:
|
||||
def get_item(self, Key): # noqa: N803 (boto3 kwarg name)
|
||||
return {"Item": dict(WO_ITEM, work_order_id=Key["work_order_id"])}
|
||||
|
||||
|
||||
class _FakeCommentsTable:
|
||||
def query(self, **kwargs):
|
||||
return {"Items": [dict(COMMENT_ITEM)]}
|
||||
|
||||
|
||||
class _FakeDynamo:
|
||||
def Table(self, name): # noqa: N802 (boto3 method name)
|
||||
if name == replay.WO_TABLE:
|
||||
return _FakeWOTable()
|
||||
return _FakeCommentsTable()
|
||||
|
||||
|
||||
class _FakeSession:
|
||||
def __init__(self, account):
|
||||
self.account = account
|
||||
|
||||
def client(self, name):
|
||||
if name == "sts":
|
||||
return _FakeSTS(self.account)
|
||||
return _FakeSecrets()
|
||||
|
||||
def resource(self, name):
|
||||
return _FakeDynamo()
|
||||
|
||||
|
||||
def _run_main(monkeypatch, account, argv):
|
||||
monkeypatch.setattr(replay.boto3, "Session", lambda **kwargs: _FakeSession(account))
|
||||
|
||||
def _no_http(*args, **kwargs):
|
||||
raise AssertionError("urlopen must not be called")
|
||||
|
||||
monkeypatch.setattr(replay.urllib.request, "urlopen", _no_http)
|
||||
monkeypatch.setattr(sys, "argv", ["replay_shoc_webhooks.py", *argv])
|
||||
replay.main()
|
||||
|
||||
|
||||
# --- Dry-run default + account gate ------------------------------------------
|
||||
|
||||
|
||||
def test_dry_run_is_default_and_performs_no_http(monkeypatch, capsys):
|
||||
# No --execute: main() must complete without ever touching urlopen (the
|
||||
# patched opener raises if reached).
|
||||
_run_main(
|
||||
monkeypatch,
|
||||
replay.EXPECTED_ACCOUNT,
|
||||
["--url", "https://receiver.invalid/webhook", "--work-order-id", "wo-1"],
|
||||
)
|
||||
out = capsys.readouterr().out
|
||||
assert "DRY-RUN: would POST" in out
|
||||
assert "Dry run complete" in out
|
||||
|
||||
|
||||
def test_account_gate_rejects_wrong_account(monkeypatch):
|
||||
assert replay.EXPECTED_ACCOUNT == "011934824531"
|
||||
with pytest.raises(SystemExit) as exc:
|
||||
_run_main(
|
||||
monkeypatch,
|
||||
"999999999999",
|
||||
[
|
||||
"--url",
|
||||
"https://receiver.invalid/webhook",
|
||||
"--work-order-id",
|
||||
"wo-1",
|
||||
],
|
||||
)
|
||||
assert "999999999999" in str(exc.value)
|
||||
assert replay.EXPECTED_ACCOUNT in str(exc.value)
|
||||
|
||||
|
||||
def test_selector_is_exactly_one_of_id_or_since(monkeypatch):
|
||||
with pytest.raises(SystemExit, match="exactly one"):
|
||||
_run_main(
|
||||
monkeypatch,
|
||||
replay.EXPECTED_ACCOUNT,
|
||||
["--url", "https://receiver.invalid/webhook"],
|
||||
)
|
||||
with pytest.raises(SystemExit, match="exactly one"):
|
||||
_run_main(
|
||||
monkeypatch,
|
||||
replay.EXPECTED_ACCOUNT,
|
||||
[
|
||||
"--url",
|
||||
"https://receiver.invalid/webhook",
|
||||
"--work-order-id",
|
||||
"wo-1",
|
||||
"--since",
|
||||
"2026-07-24T02:00:00Z",
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
# --- Envelope contract vs the emitter ----------------------------------------
|
||||
|
||||
|
||||
def test_data_field_lists_match_emitter_exactly():
|
||||
envelope = load_lambda_module("wo", "shoc_emitter/envelope")
|
||||
# Same fields, same order -- the receiver sees one schema whether an
|
||||
# event arrived live or via replay.
|
||||
assert list(replay.WO_DATA_FIELDS) == list(envelope.WO_DATA_FIELDS)
|
||||
assert list(replay.COMMENT_DATA_FIELDS) == list(envelope.COMMENT_DATA_FIELDS)
|
||||
assert "source_email_s3_key" not in replay.WO_DATA_FIELDS
|
||||
assert replay.SCHEMA_VERSION == envelope.SCHEMA_VERSION
|
||||
assert replay.SOURCE == envelope.SOURCE
|
||||
|
||||
|
||||
def test_replay_envelopes_carry_replay_true_and_exclude_s3_key():
|
||||
state = replay.build_state_event(dict(WO_ITEM))
|
||||
comment = replay.build_comment_event(dict(COMMENT_ITEM))
|
||||
assert state["replay"] is True
|
||||
assert comment["replay"] is True
|
||||
# State replays are always work_order.updated: current-state rebuilds
|
||||
# cannot distinguish the original created/cancelled, and the receiver
|
||||
# upserts idempotently.
|
||||
assert state["event_type"] == "work_order.updated"
|
||||
assert comment["event_type"] == "work_order.comment_added"
|
||||
assert "source_email_s3_key" not in state["data"]
|
||||
assert set(state["data"]) == set(replay.WO_DATA_FIELDS)
|
||||
assert set(comment["data"]) == set(replay.COMMENT_DATA_FIELDS)
|
||||
|
||||
|
||||
def test_delivery_id_is_stable_across_builds():
|
||||
first = replay.build_comment_event(dict(COMMENT_ITEM))
|
||||
second = replay.build_comment_event(dict(COMMENT_ITEM))
|
||||
assert first["delivery_id"] == second["delivery_id"]
|
||||
assert first["delivery_id"].startswith("replay-")
|
||||
state_first = replay.build_state_event(dict(WO_ITEM))
|
||||
state_second = replay.build_state_event(dict(WO_ITEM))
|
||||
assert state_first["delivery_id"] == state_second["delivery_id"]
|
||||
# State and comment ids never collide (different table seeds).
|
||||
assert first["delivery_id"] != state_first["delivery_id"]
|
||||
|
||||
|
||||
# --- --since parsing ---------------------------------------------------------
|
||||
|
||||
|
||||
def test_parse_since_canonicalizes_utc():
|
||||
assert replay.parse_since("2026-07-24T02:00:00Z") == "2026-07-24T02:00:00+00:00"
|
||||
assert (
|
||||
replay.parse_since("2026-07-24T02:00:00+00:00") == "2026-07-24T02:00:00+00:00"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"value",
|
||||
["2026-07-24T02:00:00", "2026-07-24T02:00:00+02:00", "yesterday"],
|
||||
ids=["naive", "non-utc-offset", "garbage"],
|
||||
)
|
||||
def test_parse_since_rejects_non_utc_or_garbage(value):
|
||||
with pytest.raises(SystemExit):
|
||||
replay.parse_since(value)
|
||||
|
|
@ -1,353 +0,0 @@
|
|||
"""Signing + delivery tests for the SHOC webhook emitter (plan Phase 5).
|
||||
|
||||
Golden vectors: docs/shoc-webhook-test-vectors.json is the shared handoff
|
||||
artifact -- Luby's receiver verifies against the same vectors -- and BOTH
|
||||
producer-side sign_body implementations (the emitter's delivery module and
|
||||
the replay script) are pinned here against every vector, so they can never
|
||||
drift from each other or from the published vectors.
|
||||
|
||||
Also covers the contract section 7 response-classification matrix (2xx /
|
||||
429+5xx / timeout+connection error / other 4xx) with a monkeypatched urllib
|
||||
opener, and the Secrets Manager key cache: 5-minute TTL via time.monotonic,
|
||||
keys[0] selection, empty-keys (bootstrap) -> retryable.
|
||||
"""
|
||||
|
||||
import importlib.util
|
||||
import io
|
||||
import json
|
||||
import urllib.error
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from tests.support import REPO_ROOT, load_lambda_module
|
||||
|
||||
_VECTORS_PATH = Path(REPO_ROOT) / "docs" / "shoc-webhook-test-vectors.json"
|
||||
VECTORS = json.loads(_VECTORS_PATH.read_text(encoding="utf-8"))["vectors"]
|
||||
|
||||
TEST_KEYS = [
|
||||
{"kid": "2026-07-20T00", "secret": "ab" * 32},
|
||||
{"kid": "2026-06-20T00", "secret": "cd" * 32},
|
||||
]
|
||||
|
||||
ENVELOPE = {
|
||||
"schema_version": 1,
|
||||
"delivery_id": "evt-1",
|
||||
"event_type": "work_order.created",
|
||||
"occurred_at": "2026-07-16T14:03:22.114208+00:00",
|
||||
"source": "procurement-ingest/workorder-shoc-emitter",
|
||||
"replay": False,
|
||||
"data": {"work_order_id": "11144580730"},
|
||||
}
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def delivery():
|
||||
return load_lambda_module("wo", "shoc_emitter/delivery")
|
||||
|
||||
|
||||
def _load_replay_script():
|
||||
# scripts/ is not a package and not on sys.path; load by file path
|
||||
# (mirrors tests/test_reprocess_contract.py). Import is side-effect-free:
|
||||
# the script builds its boto3 session inside main().
|
||||
path = Path(REPO_ROOT) / "scripts" / "replay_shoc_webhooks.py"
|
||||
spec = importlib.util.spec_from_file_location(
|
||||
"replay_shoc_webhooks_for_vectors", path
|
||||
)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
# --- Golden vectors ----------------------------------------------------------
|
||||
|
||||
|
||||
def test_vector_file_covers_required_shapes():
|
||||
# The handoff artifact itself must keep its coverage promises: at least
|
||||
# one non-ASCII UTF-8 body (multi-byte signing) and one empty-object body.
|
||||
assert len(VECTORS) >= 4
|
||||
assert any(any(ord(ch) > 127 for ch in v["body"]) for v in VECTORS)
|
||||
assert any(v["body"] == "{}" for v in VECTORS)
|
||||
for vector in VECTORS:
|
||||
assert set(vector) == {
|
||||
"kid",
|
||||
"secret_hex",
|
||||
"timestamp",
|
||||
"body",
|
||||
"expected_signature",
|
||||
}
|
||||
|
||||
|
||||
def test_delivery_sign_body_matches_golden_vectors(delivery):
|
||||
for vector in VECTORS:
|
||||
signature = delivery.sign_body(
|
||||
vector["secret_hex"],
|
||||
vector["timestamp"],
|
||||
vector["body"].encode("utf-8"),
|
||||
)
|
||||
assert signature == vector["expected_signature"], (
|
||||
f"delivery.sign_body drifted from golden vector kid="
|
||||
f"{vector['kid']} ts={vector['timestamp']}"
|
||||
)
|
||||
|
||||
|
||||
def test_replay_sign_body_matches_golden_vectors():
|
||||
replay = _load_replay_script()
|
||||
for vector in VECTORS:
|
||||
signature = replay.sign_body(
|
||||
vector["secret_hex"],
|
||||
vector["timestamp"],
|
||||
vector["body"].encode("utf-8"),
|
||||
)
|
||||
assert signature == vector["expected_signature"], (
|
||||
f"replay sign_body drifted from golden vector kid="
|
||||
f"{vector['kid']} ts={vector['timestamp']}"
|
||||
)
|
||||
|
||||
|
||||
# --- Response classification matrix (contract section 7) ---------------------
|
||||
|
||||
|
||||
class _FakeResponse:
|
||||
def __init__(self, status):
|
||||
self.status = status
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *exc_info):
|
||||
return False
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _webhook_url(monkeypatch, delivery):
|
||||
# SHOC_WEBHOOK_URL has no default now (Open SWE #0): set it for tests that
|
||||
# exercise deliver(), which fails closed on an unset URL.
|
||||
monkeypatch.setattr(
|
||||
delivery, "SHOC_WEBHOOK_URL", "https://shoc.example/api/webhooks/work-orders"
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def signing_keys(monkeypatch, delivery):
|
||||
monkeypatch.setattr(delivery, "_get_hmac_keys", lambda: TEST_KEYS)
|
||||
|
||||
|
||||
def _patch_urlopen(monkeypatch, delivery, fn):
|
||||
# deliver() posts through the no-redirect opener, not the module-level
|
||||
# urlopen, so patch the opener's open method.
|
||||
monkeypatch.setattr(delivery._opener, "open", fn)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("status", [200, 204])
|
||||
def test_2xx_is_delivered(monkeypatch, delivery, signing_keys, status):
|
||||
_patch_urlopen(
|
||||
monkeypatch, delivery, lambda request, timeout: _FakeResponse(status)
|
||||
)
|
||||
assert delivery.deliver(ENVELOPE) == ("delivered", status)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("url", [None, "", "http://insecure.example/hook"])
|
||||
def test_unset_or_non_https_url_fails_closed(monkeypatch, delivery, signing_keys, url):
|
||||
# No hardcoded fallback (Open SWE #0): an unset/empty/non-https URL must
|
||||
# raise (retryable) and NOT sign or POST anything.
|
||||
monkeypatch.setattr(delivery, "SHOC_WEBHOOK_URL", url)
|
||||
called = {"opened": False}
|
||||
_patch_urlopen(
|
||||
monkeypatch,
|
||||
delivery,
|
||||
lambda request, timeout: called.__setitem__("opened", True),
|
||||
)
|
||||
with pytest.raises(delivery.RetryableDeliveryError):
|
||||
delivery.deliver(ENVELOPE)
|
||||
assert called["opened"] is False
|
||||
|
||||
|
||||
@pytest.mark.parametrize("status", [429, 500, 503])
|
||||
def test_429_and_5xx_raise_retryable(monkeypatch, delivery, signing_keys, status):
|
||||
def _raise(request, timeout):
|
||||
raise urllib.error.HTTPError(
|
||||
delivery.SHOC_WEBHOOK_URL, status, "boom", None, io.BytesIO(b"")
|
||||
)
|
||||
|
||||
_patch_urlopen(monkeypatch, delivery, _raise)
|
||||
with pytest.raises(delivery.RetryableDeliveryError) as exc:
|
||||
delivery.deliver(ENVELOPE)
|
||||
assert exc.value.status_code == status
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"error",
|
||||
[urllib.error.URLError(OSError("connection refused")), TimeoutError()],
|
||||
ids=["connection-error", "timeout"],
|
||||
)
|
||||
def test_connection_failures_raise_retryable(
|
||||
monkeypatch, delivery, signing_keys, error
|
||||
):
|
||||
def _raise(request, timeout):
|
||||
raise error
|
||||
|
||||
_patch_urlopen(monkeypatch, delivery, _raise)
|
||||
with pytest.raises(delivery.RetryableDeliveryError) as exc:
|
||||
delivery.deliver(ENVELOPE)
|
||||
assert exc.value.status_code is None
|
||||
|
||||
|
||||
@pytest.mark.parametrize("status", [400, 404, 422])
|
||||
def test_other_4xx_is_rejected_not_raised(monkeypatch, delivery, signing_keys, status):
|
||||
def _raise(request, timeout):
|
||||
raise urllib.error.HTTPError(
|
||||
delivery.SHOC_WEBHOOK_URL, status, "bad", None, io.BytesIO(b"")
|
||||
)
|
||||
|
||||
_patch_urlopen(monkeypatch, delivery, _raise)
|
||||
assert delivery.deliver(ENVELOPE) == ("rejected", status)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("status", [401, 403])
|
||||
def test_auth_failures_are_retryable_and_invalidate_cache(
|
||||
monkeypatch, delivery, signing_keys, status
|
||||
):
|
||||
# A transient auth failure (stale cached key mid-rotation, receiver
|
||||
# secret-fetch blip, clock skew) must retry in order -- NOT park -- and
|
||||
# drop the key cache so the retry re-signs with the current secret.
|
||||
invalidated = {"called": False}
|
||||
|
||||
def _mark(*_a, **_k):
|
||||
invalidated["called"] = True
|
||||
|
||||
monkeypatch.setattr(delivery, "_invalidate_hmac_keys", _mark)
|
||||
|
||||
def _raise(request, timeout):
|
||||
raise urllib.error.HTTPError(
|
||||
delivery.SHOC_WEBHOOK_URL, status, "unauthorized", None, io.BytesIO(b"")
|
||||
)
|
||||
|
||||
_patch_urlopen(monkeypatch, delivery, _raise)
|
||||
with pytest.raises(delivery.RetryableDeliveryError) as exc:
|
||||
delivery.deliver(ENVELOPE)
|
||||
assert exc.value.status_code == status
|
||||
assert invalidated["called"] is True
|
||||
|
||||
|
||||
def test_redirects_are_not_followed():
|
||||
# The opener must refuse 3xx so auth headers are never forwarded to a
|
||||
# receiver-chosen Location. redirect_request returning None makes urllib
|
||||
# raise instead of following.
|
||||
delivery = load_lambda_module("wo", "shoc_emitter/delivery")
|
||||
handler = delivery._NoRedirectHandler()
|
||||
assert handler.redirect_request(None, None, 302, "Found", {}, "http://evil") is None
|
||||
|
||||
|
||||
def test_request_signed_with_keys0_and_contract_headers(
|
||||
monkeypatch, delivery, signing_keys
|
||||
):
|
||||
captured = {}
|
||||
|
||||
def _capture(request, timeout):
|
||||
captured["request"] = request
|
||||
captured["timeout"] = timeout
|
||||
return _FakeResponse(200)
|
||||
|
||||
_patch_urlopen(monkeypatch, delivery, _capture)
|
||||
assert delivery.deliver(ENVELOPE) == ("delivered", 200)
|
||||
|
||||
request = captured["request"]
|
||||
assert captured["timeout"] == delivery.POST_TIMEOUT_SECONDS
|
||||
assert request.get_method() == "POST"
|
||||
assert request.data == json.dumps(ENVELOPE).encode("utf-8")
|
||||
assert request.get_header("Content-type") == "application/json; charset=utf-8"
|
||||
assert request.get_header("User-agent") == "workorder-shoc-emitter/1"
|
||||
# The producer always signs with keys[0] (contract section 6.1).
|
||||
assert request.get_header("X-sh-key-id") == TEST_KEYS[0]["kid"]
|
||||
timestamp = request.get_header("X-sh-timestamp")
|
||||
assert timestamp.isdigit()
|
||||
expected = delivery.sign_body(TEST_KEYS[0]["secret"], int(timestamp), request.data)
|
||||
assert request.get_header("X-sh-signature") == f"v1={expected}"
|
||||
|
||||
|
||||
# --- Secret cache ------------------------------------------------------------
|
||||
|
||||
|
||||
class _FakeSecretsManager:
|
||||
"""Returns payloads in sequence (last one repeats); counts fetches."""
|
||||
|
||||
def __init__(self, payloads):
|
||||
self.payloads = list(payloads)
|
||||
self.calls = 0
|
||||
self.secret_ids = []
|
||||
|
||||
def get_secret_value(self, SecretId): # noqa: N803 (boto3 kwarg name)
|
||||
self.calls += 1
|
||||
self.secret_ids.append(SecretId)
|
||||
payload = self.payloads.pop(0) if len(self.payloads) > 1 else self.payloads[0]
|
||||
return {"SecretString": json.dumps(payload)}
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def cache_reset(monkeypatch, delivery):
|
||||
monkeypatch.setattr(delivery, "_hmac_keys_cache", None)
|
||||
monkeypatch.setattr(delivery, "_hmac_keys_cached_at", 0.0)
|
||||
monkeypatch.setattr(
|
||||
delivery,
|
||||
"HMAC_SECRET_ARN",
|
||||
"arn:aws:secretsmanager:us-east-1:011934824531:secret:"
|
||||
"workorder-ingest/shoc-webhook-hmac-AbCdEf",
|
||||
)
|
||||
|
||||
|
||||
def _wire_cache(monkeypatch, delivery, fake, clock):
|
||||
# client() is now called with a config= kwarg (bounded timeouts), so accept
|
||||
# and ignore it.
|
||||
monkeypatch.setattr(delivery.boto3, "client", lambda service, **kwargs: fake)
|
||||
monkeypatch.setattr(delivery.time, "monotonic", lambda: clock["t"])
|
||||
|
||||
|
||||
def test_cache_honors_ttl_and_refreshes_after_expiry(
|
||||
monkeypatch, delivery, cache_reset
|
||||
):
|
||||
rotated = [
|
||||
{"keys": [{"kid": "2026-08-20T00", "secret": "ef" * 32}] + TEST_KEYS[:1]}
|
||||
]
|
||||
fake = _FakeSecretsManager([{"keys": TEST_KEYS}] + rotated)
|
||||
clock = {"t": 1000.0}
|
||||
_wire_cache(monkeypatch, delivery, fake, clock)
|
||||
|
||||
assert delivery._get_hmac_keys()[0]["kid"] == "2026-07-20T00"
|
||||
assert fake.calls == 1
|
||||
|
||||
# Inside the 300s TTL: served from cache, no refetch.
|
||||
clock["t"] = 1000.0 + 299.0
|
||||
assert delivery._get_hmac_keys()[0]["kid"] == "2026-07-20T00"
|
||||
assert fake.calls == 1
|
||||
|
||||
# TTL expired: refetch picks up the rotated keys[0] (cache invalidation
|
||||
# is what makes 30-day rotation propagate within 5 minutes).
|
||||
clock["t"] = 1000.0 + 300.5
|
||||
assert delivery._get_hmac_keys()[0]["kid"] == "2026-08-20T00"
|
||||
assert fake.calls == 2
|
||||
assert fake.secret_ids[0] == delivery.HMAC_SECRET_ARN
|
||||
|
||||
|
||||
def test_empty_or_missing_keys_is_retryable_bootstrap_state(
|
||||
monkeypatch, delivery, cache_reset
|
||||
):
|
||||
clock = {"t": 5000.0}
|
||||
for payload in ({"keys": []}, {"keys": [], "bootstrap_entropy": "seed"}, {}):
|
||||
monkeypatch.setattr(delivery, "_hmac_keys_cache", None)
|
||||
monkeypatch.setattr(delivery, "_hmac_keys_cached_at", 0.0)
|
||||
fake = _FakeSecretsManager([payload])
|
||||
_wire_cache(monkeypatch, delivery, fake, clock)
|
||||
with pytest.raises(delivery.RetryableDeliveryError):
|
||||
delivery._get_hmac_keys()
|
||||
|
||||
|
||||
def test_secret_fetch_failure_is_retryable(monkeypatch, delivery, cache_reset):
|
||||
class _Boom:
|
||||
def get_secret_value(self, SecretId): # noqa: N803 (boto3 kwarg name)
|
||||
raise RuntimeError("throttled")
|
||||
|
||||
clock = {"t": 9000.0}
|
||||
_wire_cache(monkeypatch, delivery, _Boom(), clock)
|
||||
with pytest.raises(delivery.RetryableDeliveryError):
|
||||
delivery._get_hmac_keys()
|
||||
|
|
@ -1,304 +0,0 @@
|
|||
"""Stream-record -> SHOC webhook envelope mapping tests (plan Phase 5).
|
||||
|
||||
Pins the emitter's classification and envelope-building contract
|
||||
(docs/shoc-webhook-contract.md sections 3-4) against realistic DynamoDB
|
||||
stream records: event-type classification incl. the cancelled transition and
|
||||
the already-cancelled no-false-cancel case, the write_origin echo guard,
|
||||
exact eventSourceARN table discrimination, delivery_id/occurred_at mapping,
|
||||
data null-fill + source_email_s3_key exclusion, and Decimal JSON-safety.
|
||||
|
||||
Also the enum golden pin: the emitted event_type set must equal the four
|
||||
contract names AND the published spec's webhooks keys (lambdas/api/
|
||||
openapi.json), and the classification's status/record_type value space must
|
||||
stay anchored to the WO pipeline's template_parser enums.
|
||||
|
||||
Offline and pure -- envelope.py has no boto3 clients, no env reads.
|
||||
"""
|
||||
|
||||
import json
|
||||
from datetime import datetime, timezone
|
||||
from decimal import Decimal
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from tests.support import REPO_ROOT, load_lambda_module
|
||||
|
||||
# Realistic stream ARNs for BOTH tables: "WorkOrders" is a leading substring
|
||||
# of "WorkOrderComments", which is exactly the trap the exact-segment parse
|
||||
# in envelope._table_name exists to avoid.
|
||||
WO_STREAM_ARN = (
|
||||
"arn:aws:dynamodb:us-east-1:011934824531:table/WorkOrders"
|
||||
"/stream/2026-07-23T00:00:00.000"
|
||||
)
|
||||
COMMENTS_STREAM_ARN = (
|
||||
"arn:aws:dynamodb:us-east-1:011934824531:table/WorkOrderComments"
|
||||
"/stream/2026-07-23T00:00:00.000"
|
||||
)
|
||||
|
||||
CONTRACT_EVENT_TYPES = {
|
||||
"work_order.created",
|
||||
"work_order.updated",
|
||||
"work_order.cancelled",
|
||||
"work_order.comment_added",
|
||||
}
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def envelope():
|
||||
return load_lambda_module("wo", "shoc_emitter/envelope")
|
||||
|
||||
|
||||
def _record(arn, event_name, new_image=None, old_image=None, **meta):
|
||||
"""Build a stream record; ``meta`` overrides event_id / creation."""
|
||||
stream = {
|
||||
"ApproximateCreationDateTime": meta.get("creation", 1784642602.0),
|
||||
"SequenceNumber": meta.get("sequence_number", "100"),
|
||||
"StreamViewType": "NEW_AND_OLD_IMAGES",
|
||||
}
|
||||
if new_image is not None:
|
||||
stream["NewImage"] = new_image
|
||||
if old_image is not None:
|
||||
stream["OldImage"] = old_image
|
||||
return {
|
||||
"eventID": meta.get("event_id", "4b7c2f0e-0001"),
|
||||
"eventName": event_name,
|
||||
"eventSource": "aws:dynamodb",
|
||||
"eventSourceARN": arn,
|
||||
"dynamodb": stream,
|
||||
}
|
||||
|
||||
|
||||
def _wo_image(**overrides):
|
||||
image = {
|
||||
"work_order_id": {"S": "11144580730"},
|
||||
"wo_status": {"S": "new"},
|
||||
"description": {"S": "Dock door 14 won't close"},
|
||||
"record_type": {"S": "new_work_order"},
|
||||
"created_at": {"S": "2026-07-16T14:03:22.114208+00:00"},
|
||||
"updated_at": {"S": "2026-07-16T14:03:22.114208+00:00"},
|
||||
}
|
||||
image.update(overrides)
|
||||
return image
|
||||
|
||||
|
||||
def _comment_image(**overrides):
|
||||
image = {
|
||||
"work_order_id": {"S": "11144580730"},
|
||||
"comment_id": {"S": "11144580730#2026-04-27T23:51:48#a1b2c3d4e5f6"},
|
||||
"record_type": {"S": "comment"},
|
||||
"commenter": {"S": "APM Technician"},
|
||||
"text": {"S": "Vendor dispatched, ETA tomorrow AM."},
|
||||
"created_at": {"S": "2026-04-27T23:51:48"},
|
||||
"ingested_at": {"S": "2026-07-16T14:05:00+00:00"},
|
||||
}
|
||||
image.update(overrides)
|
||||
return image
|
||||
|
||||
|
||||
# --- Classification ----------------------------------------------------------
|
||||
|
||||
|
||||
def test_workorders_insert_is_created(envelope):
|
||||
event = envelope.build_event(_record(WO_STREAM_ARN, "INSERT", _wo_image()))
|
||||
assert event["event_type"] == "work_order.created"
|
||||
|
||||
|
||||
def test_workorders_modify_is_updated(envelope):
|
||||
event = envelope.build_event(
|
||||
_record(
|
||||
WO_STREAM_ARN,
|
||||
"MODIFY",
|
||||
_wo_image(wo_status={"S": "assigned"}),
|
||||
_wo_image(wo_status={"S": "new"}),
|
||||
)
|
||||
)
|
||||
assert event["event_type"] == "work_order.updated"
|
||||
|
||||
|
||||
def test_cancelled_transition_is_cancelled(envelope):
|
||||
event = envelope.build_event(
|
||||
_record(
|
||||
WO_STREAM_ARN,
|
||||
"MODIFY",
|
||||
_wo_image(wo_status={"S": "cancelled"}),
|
||||
_wo_image(wo_status={"S": "assigned"}),
|
||||
)
|
||||
)
|
||||
assert event["event_type"] == "work_order.cancelled"
|
||||
|
||||
|
||||
def test_already_cancelled_modify_is_updated_not_cancelled(envelope):
|
||||
# OLD already cancelled + NEW cancelled: any later touch to a cancelled
|
||||
# WO must NOT re-emit a cancelled event (no false cancels on retries).
|
||||
event = envelope.build_event(
|
||||
_record(
|
||||
WO_STREAM_ARN,
|
||||
"MODIFY",
|
||||
_wo_image(wo_status={"S": "cancelled"}),
|
||||
_wo_image(wo_status={"S": "cancelled"}),
|
||||
)
|
||||
)
|
||||
assert event["event_type"] == "work_order.updated"
|
||||
|
||||
|
||||
def test_comment_insert_is_comment_added(envelope):
|
||||
event = envelope.build_event(
|
||||
_record(COMMENTS_STREAM_ARN, "INSERT", _comment_image())
|
||||
)
|
||||
assert event["event_type"] == "work_order.comment_added"
|
||||
assert set(event["data"]) == set(envelope.COMMENT_DATA_FIELDS)
|
||||
|
||||
|
||||
def test_remove_is_skipped(envelope):
|
||||
assert envelope.build_event(_record(WO_STREAM_ARN, "REMOVE")) is None
|
||||
assert envelope.build_event(_record(COMMENTS_STREAM_ARN, "REMOVE")) is None
|
||||
|
||||
|
||||
def test_comment_modify_is_skipped(envelope):
|
||||
record = _record(COMMENTS_STREAM_ARN, "MODIFY", _comment_image(), _comment_image())
|
||||
assert envelope.build_event(record) is None
|
||||
|
||||
|
||||
def test_shoc_write_origin_echo_guard_skips(envelope):
|
||||
record = _record(
|
||||
WO_STREAM_ARN,
|
||||
"INSERT",
|
||||
_wo_image(write_origin={"S": "shoc-write-api"}),
|
||||
)
|
||||
assert envelope.build_event(record) is None
|
||||
# Any OTHER write_origin value is not an echo and must still deliver.
|
||||
record = _record(
|
||||
WO_STREAM_ARN,
|
||||
"INSERT",
|
||||
_wo_image(write_origin={"S": "email-processor"}),
|
||||
)
|
||||
assert envelope.build_event(record) is not None
|
||||
|
||||
|
||||
# --- eventSourceARN table discrimination -------------------------------------
|
||||
|
||||
|
||||
def test_table_discrimination_is_exact_segment_not_substring(envelope):
|
||||
# The WorkOrders stream must classify as a WO state event and the
|
||||
# WorkOrderComments stream as a comment event -- never cross-parsed even
|
||||
# though "WorkOrders" is a leading substring of "WorkOrderComments".
|
||||
wo_event = envelope.build_event(_record(WO_STREAM_ARN, "INSERT", _wo_image()))
|
||||
assert wo_event["event_type"] == "work_order.created"
|
||||
comment_event = envelope.build_event(
|
||||
_record(COMMENTS_STREAM_ARN, "INSERT", _comment_image())
|
||||
)
|
||||
assert comment_event["event_type"] == "work_order.comment_added"
|
||||
|
||||
# Superstring table names must not substring-match either known table.
|
||||
for table in ("WorkOrdersLegacy", "WorkOrderCommentsArchive", "XWorkOrders"):
|
||||
arn = (
|
||||
f"arn:aws:dynamodb:us-east-1:011934824531:table/{table}"
|
||||
"/stream/2026-07-23T00:00:00.000"
|
||||
)
|
||||
assert envelope.build_event(_record(arn, "INSERT", _wo_image())) is None
|
||||
|
||||
# A malformed / missing ARN skips rather than raises (total function).
|
||||
assert envelope.build_event(_record("", "INSERT", _wo_image())) is None
|
||||
|
||||
|
||||
# --- Envelope fields ---------------------------------------------------------
|
||||
|
||||
|
||||
def test_envelope_constant_fields_and_delivery_id(envelope):
|
||||
record = _record(WO_STREAM_ARN, "INSERT", _wo_image(), event_id="evt-abc-123")
|
||||
event = envelope.build_event(record)
|
||||
assert event["schema_version"] == 1
|
||||
assert event["source"] == "procurement-ingest/workorder-shoc-emitter"
|
||||
assert event["replay"] is False
|
||||
# delivery_id is the stream record eventID: unique per source event and
|
||||
# stable across ESM retries (the receiver's idempotency key).
|
||||
assert event["delivery_id"] == "evt-abc-123"
|
||||
|
||||
|
||||
def test_occurred_at_from_decimal_creation_time(envelope):
|
||||
# ApproximateCreationDateTime arrives as epoch seconds; via the low-level
|
||||
# stream API it deserializes as Decimal -- must still map to an aware
|
||||
# ISO-8601 +00:00 instant.
|
||||
creation = Decimal("1784642602")
|
||||
record = _record(WO_STREAM_ARN, "INSERT", _wo_image(), creation=creation)
|
||||
event = envelope.build_event(record)
|
||||
expected = datetime.fromtimestamp(1784642602.0, tz=timezone.utc).isoformat()
|
||||
assert event["occurred_at"] == expected
|
||||
assert event["occurred_at"].endswith("+00:00")
|
||||
|
||||
|
||||
def test_data_null_fills_absent_fields_and_excludes_s3_key(envelope):
|
||||
image = {
|
||||
"work_order_id": {"S": "11144580730"},
|
||||
"wo_status": {"S": "unknown"},
|
||||
# Internal pointer: must never leave the account via the webhook.
|
||||
"source_email_s3_key": {"S": "inbound/2026/07/abc.eml"},
|
||||
}
|
||||
event = envelope.build_event(_record(WO_STREAM_ARN, "INSERT", image))
|
||||
assert set(event["data"]) == set(envelope.WO_DATA_FIELDS)
|
||||
assert "source_email_s3_key" not in event["data"]
|
||||
assert event["data"]["work_order_id"] == "11144580730"
|
||||
for field in set(envelope.WO_DATA_FIELDS) - {"work_order_id", "wo_status"}:
|
||||
assert event["data"][field] is None
|
||||
|
||||
|
||||
def test_decimal_image_values_become_json_safe(envelope):
|
||||
image = _wo_image(
|
||||
severity={"N": "3"},
|
||||
priority={"N": "2.5"},
|
||||
)
|
||||
event = envelope.build_event(_record(WO_STREAM_ARN, "INSERT", image))
|
||||
assert event["data"]["severity"] == 3
|
||||
assert isinstance(event["data"]["severity"], int)
|
||||
assert event["data"]["priority"] == 2.5
|
||||
assert isinstance(event["data"]["priority"], float)
|
||||
# The whole envelope must serialize -- this is the raw body that gets
|
||||
# signed and POSTed.
|
||||
json.dumps(event)
|
||||
|
||||
|
||||
# --- Enum golden pin ---------------------------------------------------------
|
||||
|
||||
|
||||
def test_event_type_set_matches_contract_and_openapi_webhooks(envelope):
|
||||
scenarios = [
|
||||
_record(WO_STREAM_ARN, "INSERT", _wo_image()),
|
||||
_record(
|
||||
WO_STREAM_ARN,
|
||||
"MODIFY",
|
||||
_wo_image(wo_status={"S": "assigned"}),
|
||||
_wo_image(wo_status={"S": "new"}),
|
||||
),
|
||||
_record(
|
||||
WO_STREAM_ARN,
|
||||
"MODIFY",
|
||||
_wo_image(wo_status={"S": "cancelled"}),
|
||||
_wo_image(wo_status={"S": "assigned"}),
|
||||
),
|
||||
_record(COMMENTS_STREAM_ARN, "INSERT", _comment_image()),
|
||||
]
|
||||
emitted = {envelope.build_event(record)["event_type"] for record in scenarios}
|
||||
assert emitted == CONTRACT_EVENT_TYPES
|
||||
|
||||
# The published read-API spec documents the same four outbound webhook
|
||||
# events; emitter and spec must move together.
|
||||
spec_path = Path(REPO_ROOT) / "lambdas" / "api" / "openapi.json"
|
||||
spec = json.loads(spec_path.read_text(encoding="utf-8"))
|
||||
assert set(spec["webhooks"]) == emitted
|
||||
|
||||
|
||||
def test_status_and_record_type_value_space_anchored_to_template_parser(envelope):
|
||||
# The WO pipeline's template_parser enums are the wo_status/record_type
|
||||
# value space the stream carries (prompts.py mirrors them for the AI
|
||||
# path). The cancelled-transition trigger value must be a real status,
|
||||
# and the contract's record_type enum must be exactly the pipeline's
|
||||
# email-type enum -- widen either side only in a PR that updates both.
|
||||
template_parser = load_lambda_module("wo", "email_processor/template_parser")
|
||||
assert envelope.CANCELLED_STATUS in template_parser.VALID_STATUSES
|
||||
assert template_parser.VALID_EMAIL_TYPES == {
|
||||
"new_work_order",
|
||||
"update",
|
||||
"comment",
|
||||
"cancellation",
|
||||
}
|
||||
|
|
@ -1,230 +0,0 @@
|
|||
"""Batch-loop tests for the SHOC webhook emitter handler (plan Phase 5).
|
||||
|
||||
Pins the partial-batch ordering contract: on a retryable failure at record i
|
||||
the loop STOPS -- record i's DynamoDB SequenceNumber is reported via
|
||||
report_batch_item_failures (so the ESM retries from it, in order) and later
|
||||
records are never attempted; earlier in-batch successes are not re-delivered.
|
||||
Non-retryable rejections park the full envelope on the rejected queue and the
|
||||
loop CONTINUES (a contract bug must never block the shard). Skip records
|
||||
(REMOVE / echo guard) produce no delivery attempts at all.
|
||||
|
||||
delivery.deliver is monkeypatched on the handler's own sibling instance (the
|
||||
handler imports it by bare name); envelope.build_event runs for real, so
|
||||
these batches exercise the true record -> envelope -> deliver path. SQS is a
|
||||
plain fake on the handler's public ``sqs`` module global.
|
||||
"""
|
||||
|
||||
import json
|
||||
|
||||
import pytest
|
||||
|
||||
from tests.support import load_lambda_module
|
||||
|
||||
WO_STREAM_ARN = (
|
||||
"arn:aws:dynamodb:us-east-1:011934824531:table/WorkOrders"
|
||||
"/stream/2026-07-23T00:00:00.000"
|
||||
)
|
||||
|
||||
REJECTED_QUEUE_URL = (
|
||||
"https://sqs.us-east-1.amazonaws.com/011934824531/workorder-shoc-emitter-rejected"
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def handler_mod():
|
||||
return load_lambda_module("wo", "shoc_emitter/handler")
|
||||
|
||||
|
||||
class _FakeSQS:
|
||||
def __init__(self):
|
||||
self.sent = []
|
||||
|
||||
def send_message(self, QueueUrl, MessageBody): # noqa: N803 (boto3 kwargs)
|
||||
self.sent.append({"QueueUrl": QueueUrl, "MessageBody": MessageBody})
|
||||
|
||||
|
||||
def _wo_record(event_id, sequence_number, event_name="INSERT", **image_overrides):
|
||||
image = {
|
||||
"work_order_id": {"S": f"wo-{event_id}"},
|
||||
"wo_status": {"S": "new"},
|
||||
"record_type": {"S": "new_work_order"},
|
||||
}
|
||||
image.update(image_overrides)
|
||||
stream = {
|
||||
"ApproximateCreationDateTime": 1784642602.0,
|
||||
"SequenceNumber": sequence_number,
|
||||
"StreamViewType": "NEW_AND_OLD_IMAGES",
|
||||
}
|
||||
if event_name != "REMOVE":
|
||||
stream["NewImage"] = image
|
||||
return {
|
||||
"eventID": event_id,
|
||||
"eventName": event_name,
|
||||
"eventSource": "aws:dynamodb",
|
||||
"eventSourceARN": WO_STREAM_ARN,
|
||||
"dynamodb": stream,
|
||||
}
|
||||
|
||||
|
||||
def _wire(monkeypatch, handler_mod, outcomes):
|
||||
"""Patch delivery.deliver with a per-delivery_id outcome table.
|
||||
|
||||
``outcomes`` maps delivery_id (the record eventID) to either a
|
||||
("delivered"|"rejected", status) tuple or the string "retryable" (raise).
|
||||
Returns the ordered list of attempted delivery_ids and the fake SQS.
|
||||
"""
|
||||
attempted = []
|
||||
|
||||
def _fake_deliver(webhook_event):
|
||||
attempted.append(webhook_event["delivery_id"])
|
||||
outcome = outcomes[webhook_event["delivery_id"]]
|
||||
if outcome == "retryable":
|
||||
raise handler_mod.delivery.RetryableDeliveryError(
|
||||
"receiver returned 503", status_code=503
|
||||
)
|
||||
return outcome
|
||||
|
||||
monkeypatch.setattr(handler_mod.delivery, "deliver", _fake_deliver)
|
||||
fake_sqs = _FakeSQS()
|
||||
monkeypatch.setattr(handler_mod, "sqs", fake_sqs)
|
||||
monkeypatch.setattr(handler_mod, "REJECTED_QUEUE_URL", REJECTED_QUEUE_URL)
|
||||
return attempted, fake_sqs
|
||||
|
||||
|
||||
def test_retryable_middle_record_stops_batch_and_reports_its_sequence(
|
||||
monkeypatch, handler_mod
|
||||
):
|
||||
event = {
|
||||
"Records": [
|
||||
_wo_record("evt-1", "101"),
|
||||
_wo_record("evt-2", "102"),
|
||||
_wo_record("evt-3", "103"),
|
||||
]
|
||||
}
|
||||
attempted, fake_sqs = _wire(
|
||||
monkeypatch,
|
||||
handler_mod,
|
||||
{"evt-1": ("delivered", 200), "evt-2": "retryable"},
|
||||
)
|
||||
|
||||
result = handler_mod.handler(event, None)
|
||||
|
||||
# EXACTLY the failed record's SequenceNumber: the ESM retries from it in
|
||||
# order, and evt-1 (already delivered) is not re-delivered.
|
||||
assert result == {"batchItemFailures": [{"itemIdentifier": "102"}]}
|
||||
# The loop stopped at the failure: record 3 was never attempted.
|
||||
assert attempted == ["evt-1", "evt-2"]
|
||||
assert fake_sqs.sent == []
|
||||
|
||||
|
||||
def test_rejected_record_parks_to_sqs_and_processing_continues(
|
||||
monkeypatch, handler_mod
|
||||
):
|
||||
event = {
|
||||
"Records": [
|
||||
_wo_record("evt-1", "201"),
|
||||
_wo_record("evt-2", "202"),
|
||||
_wo_record("evt-3", "203"),
|
||||
]
|
||||
}
|
||||
attempted, fake_sqs = _wire(
|
||||
monkeypatch,
|
||||
handler_mod,
|
||||
{
|
||||
"evt-1": ("delivered", 200),
|
||||
"evt-2": ("rejected", 422),
|
||||
"evt-3": ("delivered", 204),
|
||||
},
|
||||
)
|
||||
|
||||
result = handler_mod.handler(event, None)
|
||||
|
||||
# A 4xx rejection must NOT block the shard: no batch item failures, and
|
||||
# the records after the rejection were still attempted.
|
||||
assert result == {"batchItemFailures": []}
|
||||
assert attempted == ["evt-1", "evt-2", "evt-3"]
|
||||
|
||||
# The full envelope was parked for operator replay.
|
||||
assert len(fake_sqs.sent) == 1
|
||||
assert fake_sqs.sent[0]["QueueUrl"] == REJECTED_QUEUE_URL
|
||||
parked = json.loads(fake_sqs.sent[0]["MessageBody"])
|
||||
assert set(parked) == {"envelope", "response_status"}
|
||||
assert parked["response_status"] == 422
|
||||
assert parked["envelope"]["delivery_id"] == "evt-2"
|
||||
assert parked["envelope"]["event_type"] == "work_order.created"
|
||||
assert parked["envelope"]["data"]["work_order_id"] == "wo-evt-2"
|
||||
|
||||
|
||||
def test_skip_records_produce_no_delivery_calls(monkeypatch, handler_mod):
|
||||
event = {
|
||||
"Records": [
|
||||
_wo_record("evt-1", "301", event_name="REMOVE"),
|
||||
_wo_record("evt-2", "302", write_origin={"S": "shoc-write-api"}),
|
||||
]
|
||||
}
|
||||
attempted, fake_sqs = _wire(monkeypatch, handler_mod, {})
|
||||
|
||||
result = handler_mod.handler(event, None)
|
||||
|
||||
assert result == {"batchItemFailures": []}
|
||||
assert attempted == []
|
||||
assert fake_sqs.sent == []
|
||||
|
||||
|
||||
def test_unexpected_error_reports_record_not_whole_batch(monkeypatch, handler_mod):
|
||||
# An unexpected exception (here: SQS park failure on a 4xx rejection) must
|
||||
# NOT escape the loop -- that would fail the invocation and make the ESM
|
||||
# re-deliver every earlier success for 24h. The offending record is
|
||||
# reported so the ESM retries from it in order; evt-1 is not re-delivered.
|
||||
event = {
|
||||
"Records": [
|
||||
_wo_record("evt-1", "401"),
|
||||
_wo_record("evt-2", "402"),
|
||||
_wo_record("evt-3", "403"),
|
||||
]
|
||||
}
|
||||
attempted, fake_sqs = _wire(
|
||||
monkeypatch,
|
||||
handler_mod,
|
||||
{
|
||||
"evt-1": ("delivered", 200),
|
||||
"evt-2": ("rejected", 400),
|
||||
"evt-3": ("delivered", 200),
|
||||
},
|
||||
)
|
||||
|
||||
def _boom(QueueUrl, MessageBody): # noqa: N803 (boto3 kwargs)
|
||||
raise RuntimeError("sqs unavailable")
|
||||
|
||||
monkeypatch.setattr(fake_sqs, "send_message", _boom)
|
||||
|
||||
result = handler_mod.handler(event, None)
|
||||
|
||||
assert result == {"batchItemFailures": [{"itemIdentifier": "402"}]}
|
||||
# Stopped at the failing record; evt-3 not attempted.
|
||||
assert attempted == ["evt-1", "evt-2"]
|
||||
|
||||
|
||||
def test_record_missing_sequence_number_is_skipped_not_crashed(
|
||||
monkeypatch, handler_mod
|
||||
):
|
||||
# A record that maps to an envelope but lacks SequenceNumber (unreachable
|
||||
# for real streams) must be skipped, not crash the failure paths (Open SWE
|
||||
# #9/#26). Craft one that build_event accepts but strip SequenceNumber.
|
||||
rec = _wo_record("evt-1", "999")
|
||||
del rec["dynamodb"]["SequenceNumber"]
|
||||
attempted, _ = _wire(
|
||||
monkeypatch, handler_mod, {"evt-1": "retryable", "evt-2": ("delivered", 200)}
|
||||
)
|
||||
good = _wo_record("evt-2", "1000")
|
||||
result = handler_mod.handler({"Records": [rec, good]}, None)
|
||||
# The seq-less record is skipped (never delivered); the good one delivers.
|
||||
assert result == {"batchItemFailures": []}
|
||||
assert "evt-1" not in attempted
|
||||
assert "evt-2" in attempted
|
||||
|
||||
|
||||
def test_empty_batch_returns_no_failures(monkeypatch, handler_mod):
|
||||
attempted, _ = _wire(monkeypatch, handler_mod, {})
|
||||
assert handler_mod.handler({"Records": []}, None) == {"batchItemFailures": []}
|
||||
assert attempted == []
|
||||
|
|
@ -1,351 +0,0 @@
|
|||
"""Rotation-step tests for workorder-shoc-hmac-rotator (plan Phase 5).
|
||||
|
||||
Exercises the Secrets Manager rotation protocol against a dict-backed fake
|
||||
client (no moto, no AWS): createSecret from the CDK bootstrap value and from
|
||||
a populated list (newest-first, truncated to 2), createSecret idempotency on
|
||||
retried tokens, the intra-hour kid-collision fallback, testSecret's
|
||||
shape/hex validation, finishSecret's stage move + idempotency, the setSecret
|
||||
no-op, and the unknown-step failure.
|
||||
|
||||
datetime is monkeypatched to a fixed instant so kid assertions are exact and
|
||||
the collision case is deterministic (no wall-clock hour-boundary flake).
|
||||
"""
|
||||
|
||||
import json
|
||||
import re
|
||||
from datetime import datetime, timezone
|
||||
|
||||
import pytest
|
||||
|
||||
from tests.support import load_lambda_module
|
||||
|
||||
SECRET_ID = "workorder-ingest/shoc-webhook-hmac"
|
||||
HEX_64_RE = re.compile(r"^[0-9a-f]{64}$")
|
||||
|
||||
# Fixed rotation instant: kid "2026-07-24T15" (intra-hour fallback
|
||||
# "2026-07-24T1507").
|
||||
FIXED_NOW = datetime(2026, 7, 24, 15, 7, 42, tzinfo=timezone.utc)
|
||||
|
||||
KEY_A = {"kid": "2026-06-20T00", "secret": "a" * 64}
|
||||
KEY_B = {"kid": "2026-05-21T00", "secret": "b" * 64}
|
||||
|
||||
|
||||
class _ResourceNotFound(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class _Exceptions:
|
||||
ResourceNotFoundException = _ResourceNotFound
|
||||
|
||||
|
||||
class FakeSecretsManager:
|
||||
"""Dict-backed Secrets Manager stub recording every mutating call.
|
||||
|
||||
``versions`` maps version_id -> {"stages": set[str], "value": str}.
|
||||
"""
|
||||
|
||||
exceptions = _Exceptions
|
||||
|
||||
def __init__(self, versions=None):
|
||||
self.versions = {
|
||||
vid: {"stages": set(v["stages"]), "value": v["value"]}
|
||||
for vid, v in (versions or {}).items()
|
||||
}
|
||||
self.put_calls = []
|
||||
self.stage_calls = []
|
||||
|
||||
def describe_secret(self, SecretId): # noqa: N803 (boto3 kwarg names)
|
||||
return {
|
||||
"VersionIdsToStages": {
|
||||
vid: sorted(v["stages"]) for vid, v in self.versions.items()
|
||||
}
|
||||
}
|
||||
|
||||
def get_secret_value( # noqa: N803 (boto3 kwarg names)
|
||||
self, SecretId, VersionId=None, VersionStage=None
|
||||
):
|
||||
if VersionId is not None:
|
||||
version = self.versions.get(VersionId)
|
||||
if version is None or (
|
||||
VersionStage is not None and VersionStage not in version["stages"]
|
||||
):
|
||||
raise _ResourceNotFound(f"{VersionId} / {VersionStage}")
|
||||
return {"SecretString": version["value"]}
|
||||
stage = VersionStage or "AWSCURRENT"
|
||||
for version in self.versions.values():
|
||||
if stage in version["stages"]:
|
||||
return {"SecretString": version["value"]}
|
||||
raise _ResourceNotFound(stage)
|
||||
|
||||
def put_secret_value( # noqa: N803 (boto3 kwarg names)
|
||||
self, SecretId, ClientRequestToken, SecretString, VersionStages
|
||||
):
|
||||
self.put_calls.append(
|
||||
{
|
||||
"SecretId": SecretId,
|
||||
"ClientRequestToken": ClientRequestToken,
|
||||
"SecretString": SecretString,
|
||||
"VersionStages": list(VersionStages),
|
||||
}
|
||||
)
|
||||
self.versions[ClientRequestToken] = {
|
||||
"stages": set(VersionStages),
|
||||
"value": SecretString,
|
||||
}
|
||||
|
||||
def update_secret_version_stage(self, **kwargs):
|
||||
self.stage_calls.append(kwargs)
|
||||
stage = kwargs["VersionStage"]
|
||||
removed = kwargs.get("RemoveFromVersionId")
|
||||
if removed is not None:
|
||||
self.versions[removed]["stages"].discard(stage)
|
||||
self.versions[kwargs["MoveToVersionId"]]["stages"].add(stage)
|
||||
|
||||
|
||||
class _FixedDatetime:
|
||||
@classmethod
|
||||
def now(cls, tz=None):
|
||||
return FIXED_NOW
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def rotator():
|
||||
return load_lambda_module("wo", "shoc_hmac_rotator/handler")
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def fixed_now(monkeypatch, rotator):
|
||||
monkeypatch.setattr(rotator, "datetime", _FixedDatetime)
|
||||
|
||||
|
||||
def _run(monkeypatch, rotator, fake, step, token="tok-1"):
|
||||
monkeypatch.setattr(rotator, "secretsmanager_client", fake)
|
||||
rotator.handler(
|
||||
{"SecretId": SECRET_ID, "ClientRequestToken": token, "Step": step}, None
|
||||
)
|
||||
|
||||
|
||||
def _bootstrap_fake(value=None):
|
||||
current = value if value is not None else {"keys": [], "bootstrap_entropy": "x"}
|
||||
return FakeSecretsManager(
|
||||
{"cur-1": {"stages": {"AWSCURRENT"}, "value": json.dumps(current)}}
|
||||
)
|
||||
|
||||
|
||||
# --- createSecret ------------------------------------------------------------
|
||||
|
||||
|
||||
def test_create_from_bootstrap_stages_one_fresh_key(monkeypatch, rotator, fixed_now):
|
||||
fake = _bootstrap_fake()
|
||||
_run(monkeypatch, rotator, fake, "createSecret")
|
||||
|
||||
assert len(fake.put_calls) == 1
|
||||
put = fake.put_calls[0]
|
||||
assert put["SecretId"] == SECRET_ID
|
||||
assert put["ClientRequestToken"] == "tok-1"
|
||||
assert put["VersionStages"] == ["AWSPENDING"]
|
||||
staged = json.loads(put["SecretString"])
|
||||
assert list(staged) == ["keys"]
|
||||
assert len(staged["keys"]) == 1
|
||||
assert staged["keys"][0]["kid"] == "2026-07-24T15"
|
||||
assert HEX_64_RE.fullmatch(staged["keys"][0]["secret"])
|
||||
|
||||
|
||||
def test_create_prepends_and_truncates_to_two_newest_first(
|
||||
monkeypatch, rotator, fixed_now
|
||||
):
|
||||
fake = _bootstrap_fake({"keys": [KEY_A, KEY_B]})
|
||||
_run(monkeypatch, rotator, fake, "createSecret")
|
||||
|
||||
staged = json.loads(fake.put_calls[0]["SecretString"])
|
||||
assert len(staged["keys"]) == 2
|
||||
# Newest first: the fresh key leads, the previous head is retained for
|
||||
# one overlap cycle, the oldest key falls off.
|
||||
assert staged["keys"][0]["kid"] == "2026-07-24T15"
|
||||
assert staged["keys"][1] == KEY_A
|
||||
assert KEY_B["secret"] not in fake.put_calls[0]["SecretString"]
|
||||
|
||||
|
||||
def test_create_is_idempotent_when_pending_already_staged(
|
||||
monkeypatch, rotator, fixed_now
|
||||
):
|
||||
fake = _bootstrap_fake({"keys": [KEY_A]})
|
||||
fake.versions["tok-1"] = {
|
||||
"stages": {"AWSPENDING"},
|
||||
"value": json.dumps({"keys": [KEY_A]}),
|
||||
}
|
||||
_run(monkeypatch, rotator, fake, "createSecret")
|
||||
assert fake.put_calls == []
|
||||
|
||||
|
||||
def test_create_is_noop_when_token_already_current(monkeypatch, rotator, fixed_now):
|
||||
fake = FakeSecretsManager(
|
||||
{
|
||||
"tok-1": {
|
||||
"stages": {"AWSCURRENT"},
|
||||
"value": json.dumps({"keys": [KEY_A]}),
|
||||
}
|
||||
}
|
||||
)
|
||||
_run(monkeypatch, rotator, fake, "createSecret")
|
||||
assert fake.put_calls == []
|
||||
|
||||
|
||||
def test_create_kid_collision_same_hour_gets_distinct_suffix(
|
||||
monkeypatch, rotator, fixed_now
|
||||
):
|
||||
# Forced re-rotation within the same hour: the hour-format kid already
|
||||
# exists, so the new kid extends to minutes + a random suffix, guaranteed
|
||||
# distinct from every retained kid (never reissue a kid for a new secret).
|
||||
fake = _bootstrap_fake({"keys": [{"kid": "2026-07-24T15", "secret": "c" * 64}]})
|
||||
_run(monkeypatch, rotator, fake, "createSecret")
|
||||
|
||||
staged = json.loads(fake.put_calls[0]["SecretString"])
|
||||
new_kid = staged["keys"][0]["kid"]
|
||||
assert new_kid.startswith("2026-07-24T1507")
|
||||
assert new_kid != "2026-07-24T15"
|
||||
assert staged["keys"][1]["kid"] == "2026-07-24T15"
|
||||
|
||||
|
||||
def test_create_kid_collision_against_second_key_also_avoided(
|
||||
monkeypatch, rotator, fixed_now
|
||||
):
|
||||
# Uniqueness is checked against ALL retained kids, not just keys[0]: if the
|
||||
# hour-format kid matches keys[1] (a third same-hour rotation), it still
|
||||
# gets a distinct suffix rather than being reissued.
|
||||
fake = _bootstrap_fake(
|
||||
{
|
||||
"keys": [
|
||||
{"kid": "2026-07-24T1500ab", "secret": "a" * 64},
|
||||
{"kid": "2026-07-24T15", "secret": "b" * 64},
|
||||
]
|
||||
}
|
||||
)
|
||||
_run(monkeypatch, rotator, fake, "createSecret")
|
||||
new_kid = json.loads(fake.put_calls[0]["SecretString"])["keys"][0]["kid"]
|
||||
assert new_kid not in {"2026-07-24T15", "2026-07-24T1500ab"}
|
||||
|
||||
|
||||
def test_create_malformed_current_value_starts_fresh_list(
|
||||
monkeypatch, rotator, fixed_now
|
||||
):
|
||||
fake = FakeSecretsManager(
|
||||
{"cur-1": {"stages": {"AWSCURRENT"}, "value": "not json {"}}
|
||||
)
|
||||
_run(monkeypatch, rotator, fake, "createSecret")
|
||||
staged = json.loads(fake.put_calls[0]["SecretString"])
|
||||
assert len(staged["keys"]) == 1
|
||||
assert HEX_64_RE.fullmatch(staged["keys"][0]["secret"])
|
||||
|
||||
|
||||
def test_create_transient_current_read_error_propagates_not_swallowed(
|
||||
monkeypatch, rotator, fixed_now
|
||||
):
|
||||
# A transient AWSCURRENT read failure (throttle/KMS blip) must NOT be
|
||||
# swallowed into an empty key list -- that would drop the overlap key and
|
||||
# strand in-flight deliveries. It re-raises so Secrets Manager fails and
|
||||
# retries the rotation with the prior AWSCURRENT intact.
|
||||
class _ThrottlingFake(FakeSecretsManager):
|
||||
def get_secret_value(self, SecretId, VersionId=None, VersionStage=None): # noqa: N803
|
||||
if VersionStage == "AWSCURRENT":
|
||||
raise RuntimeError("ThrottlingException")
|
||||
return super().get_secret_value(
|
||||
SecretId, VersionId=VersionId, VersionStage=VersionStage
|
||||
)
|
||||
|
||||
fake = _ThrottlingFake(
|
||||
{"cur-1": {"stages": {"AWSCURRENT"}, "value": json.dumps({"keys": []})}}
|
||||
)
|
||||
with pytest.raises(RuntimeError, match="ThrottlingException"):
|
||||
_run(monkeypatch, rotator, fake, "createSecret")
|
||||
assert fake.put_calls == []
|
||||
|
||||
|
||||
# --- testSecret --------------------------------------------------------------
|
||||
|
||||
|
||||
def _pending_fake(value):
|
||||
return FakeSecretsManager(
|
||||
{"tok-1": {"stages": {"AWSPENDING"}, "value": json.dumps(value)}}
|
||||
)
|
||||
|
||||
|
||||
def test_test_secret_accepts_valid_pending_value(monkeypatch, rotator):
|
||||
fake = _pending_fake({"keys": [{"kid": "2026-07-24T15", "secret": "d" * 64}]})
|
||||
_run(monkeypatch, rotator, fake, "testSecret") # no raise
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"pending_value",
|
||||
[
|
||||
{"keys": []},
|
||||
{"keys": [{"kid": "2026-07-24T15", "secret": "d" * 63}]},
|
||||
{"keys": [{"kid": "2026-07-24T15", "secret": "z" * 64}]},
|
||||
{"keys": [{"kid": "2026-07-24T15", "secret": "D" * 64}]},
|
||||
{"keys": [{"secret": "d" * 64}]},
|
||||
],
|
||||
ids=["empty-keys", "short-secret", "non-hex", "uppercase-hex", "missing-kid"],
|
||||
)
|
||||
def test_test_secret_rejects_bad_pending_values(monkeypatch, rotator, pending_value):
|
||||
fake = _pending_fake(pending_value)
|
||||
with pytest.raises(ValueError):
|
||||
_run(monkeypatch, rotator, fake, "testSecret")
|
||||
|
||||
|
||||
# --- finishSecret ------------------------------------------------------------
|
||||
|
||||
|
||||
def test_finish_moves_current_stage_from_old_version(monkeypatch, rotator):
|
||||
fake = FakeSecretsManager(
|
||||
{
|
||||
"old-1": {
|
||||
"stages": {"AWSCURRENT"},
|
||||
"value": json.dumps({"keys": [KEY_A]}),
|
||||
},
|
||||
"tok-1": {
|
||||
"stages": {"AWSPENDING"},
|
||||
"value": json.dumps({"keys": [KEY_B, KEY_A]}),
|
||||
},
|
||||
}
|
||||
)
|
||||
_run(monkeypatch, rotator, fake, "finishSecret")
|
||||
|
||||
assert fake.stage_calls == [
|
||||
{
|
||||
"SecretId": SECRET_ID,
|
||||
"VersionStage": "AWSCURRENT",
|
||||
"MoveToVersionId": "tok-1",
|
||||
"RemoveFromVersionId": "old-1",
|
||||
}
|
||||
]
|
||||
assert "AWSCURRENT" in fake.versions["tok-1"]["stages"]
|
||||
assert "AWSCURRENT" not in fake.versions["old-1"]["stages"]
|
||||
|
||||
|
||||
def test_finish_is_idempotent_when_token_already_current(monkeypatch, rotator):
|
||||
fake = FakeSecretsManager(
|
||||
{
|
||||
"tok-1": {
|
||||
"stages": {"AWSCURRENT", "AWSPENDING"},
|
||||
"value": json.dumps({"keys": [KEY_A]}),
|
||||
}
|
||||
}
|
||||
)
|
||||
_run(monkeypatch, rotator, fake, "finishSecret")
|
||||
assert fake.stage_calls == []
|
||||
|
||||
|
||||
# --- Dispatch ----------------------------------------------------------------
|
||||
|
||||
|
||||
def test_set_secret_is_a_noop(monkeypatch, rotator):
|
||||
fake = _bootstrap_fake()
|
||||
_run(monkeypatch, rotator, fake, "setSecret")
|
||||
assert fake.put_calls == []
|
||||
assert fake.stage_calls == []
|
||||
|
||||
|
||||
def test_unknown_step_raises(monkeypatch, rotator):
|
||||
fake = _bootstrap_fake()
|
||||
with pytest.raises(ValueError, match="Unknown rotation step"):
|
||||
_run(monkeypatch, rotator, fake, "rotateHarder")
|
||||
Loading…
Add table
Reference in a new issue