procurement-ingest/docs/shoc-webhook-plan.md

# Implementation Plan — SHOC Realtime Work-Order Webhook

**Branch:** `feat/shoc-wo-webhook` (re-cut on `main` 2026-07-23; all build work targets the post-refactor layout — `cdk/common.py` helpers, explicit LogGroups, decomposed handlers, consolidated test roots — not the pre-refactor style earlier drafts implied).
**Companion docs:** `docs/shoc-webhook-contract.md` (Rev 2026-07-23 — the producer/consumer contract; Luby builds the receiver against it) and the procurement read API OpenAPI spec (`lambdas/api/openapi.json`, built in the sibling `procurement-api` effort — the reconciliation/backfill path).
**Requestor:** Luby (SHOC team). Decisions captured 2026-07-16: all WO event types, seconds-latency, full-payload push, HMAC + automated rotation, in-order per WO, prod-data-in-dev accepted, dual-write retained (DynamoDB retirement deferred for scope discipline — the original blocker, seahaven-slack-bot, was decommissioned 2026-07-23; retirement is now sequenced after SHOC prod + mgmt decommission, not this PR).
**Account (2026-07-23):** everything here builds in **seahaven-prod (011934824531)**. The old management account (328440206208) is frozen/rollback-only.

## Architecture

```
SES → S3 → workorder-email-processor ──writes──▶ WorkOrders ─stream─▶
                                     └─writes──▶ WorkOrderComments ─stream─▶ workorder-shoc-emitter ──HMAC POST──▶ SHOC API
                                                                                    │ (retry-exhausted / non-retryable)
                                                                                    ▼
                                                                        shoc-emitter failure queues → alarms → replay script
```

Rationale (vs. inline call from the processor): DynamoDB Streams gives per-partition-key (per-`work_order_id`) commit-order delivery with shard-blocking retries — the ordering guarantee Luby requires — and keeps the email-ingestion hot path completely decoupled from SHOC availability. No change to PR #99's handler.

## Phase 0 — Prerequisites (blocking)

1. Branch re-cut on `main` (done 2026-07-23).
2. Luby signs off `docs/shoc-webhook-contract.md` Rev 2026-07-23 (endpoint path, PR #10 entity mapping, the `unknown`-status mapping, `SyncVendorReplies` retirement).
3. Confirm the SHOC receiver role ARN for the resource policies (`arn:aws:iam::396287094661:role/shoc-backend-dev`) — the role lives in SHOC's account, outside this repo's control; confirm it still exists before the CDK references it (the first cross-account `get-secret-value` test is the hard proof).
4. **⚠️ ACCOUNT GATE: every deploy in this plan targets seahaven-prod (011934824531) ONLY.** Never deploy this stack to mgmt (328440206208), and never "temporarily" enable streams on mgmt's copies of the WO tables — that is the only path by which the SES dual-delivery bake could ever double-send to SHOC. Verify `aws sts get-caller-identity` resolves to 011934824531 before `cdk deploy`.
5. Confirm with Luby which SHOC environment should receive the feed now — "prod-data-in-dev" was accepted 2026-07-16, pre-migration; re-confirm the target URL.

## Phase 1 — Secret + rotation (CDK, `wo_stack.py`)

This phase is an *incremental* deploy onto the already-migrated `workorder-ingest` stack in seahaven-prod — it is not part of the account-migration work.

- New customer-managed KMS key `workorder-ingest-shoc-webhook-kms` (kebab-case per naming-conventions.md). Required because the default `aws/secretsmanager` key cannot serve cross-account reads. **Deliberately a dedicated key, not `alias/seahaven-dynamodb`:** reusing the DynamoDB CMK would hand the SHOC cross-account grant decrypt reach over the PO table's encryption key — the dedicated key scopes the grant to exactly this secret.
- New Secrets Manager secret **`workorder-ingest/shoc-webhook-hmac`** (naming per secrets-and-config.md `stack-name/value-name`), CDK-managed, encrypted with the CMK. Value shape: `{"keys": [{"kid", "secret"}, ...]}` (dual-key, contract §6.1).
- **RemovalPolicy: `DESTROY`, deliberately.** The value is machine-generated HMAC material with no operator-set content — fully regenerable by one rotation — so `RETAIN` buys nothing and would expose the fixed-name RETAIN orphan deadlock (failed first create orphans an empty shell holding the global name → every later create fails `AlreadyExists`; see `reference_secret_retain_orphan_deadlock`). Add a GOTCHA comment at the removal-policy line so nobody "hardens" it to RETAIN later. If a first deploy does fail, check for and force-delete any **empty** orphaned shell before retrying. Accidental-deletion recovery: recreate + rotate (new keys), notify Luby (receivers refresh within their 5-min TTL), watch the `-failures` queue and replay the gap — documented in the README runbook. Secret gets a `description` and `Purpose`/`ManagedBy` tags for discoverability.
- Rotation Lambda `workorder-shoc-hmac-rotator` (Python 3.12, ARM64, explicit LogGroup via `common.make_function_log_group`, alarms via the house `common.add_standard_lambda_alarms` idiom) on a 30-day `rotation_schedule`: generates a new 32-byte key, prepends as `keys[0]`, truncates to 2 entries. Single-user rotation (no external system holds the value — receivers re-fetch), so the standard 4-step rotation collapses to createSecret/finishSecret. Rotation-outage math: producer signs `keys[0]` with a ≤5-min cache; receiver keeps the previous key and refreshes on unknown `kid` (contract §6.1, receiver-side) — no delivery window where signatures can't verify.
- Cross-account read grants — **both halves required, to the exact role ARN** (`arn:aws:iam::396287094661:role/shoc-backend-dev`), no wildcards: the secret **resource policy** grants `secretsmanager:GetSecretValue` AND the **KMS key policy** grants `kms:Decrypt`; either one alone fails silently at the receiver. Future staging/prod receiver roles are each an explicit policy addition with its own cross-family review — no prefix/wildcard trust. **IAM/resource-policy change → mandatory GPT-4.1 cross-family review (Phase 6).**
- **Post-deploy verification:** assume `shoc-backend-dev` (or have Luby run it) and perform a real cross-account `get-secret-value` before declaring Phase 1 done (synth ≠ authz — the live call is the proof).
- **Handoff:** send Luby the secret ARN, region, and the contract Rev pointer as an explicit step — the grant is useless if she doesn't have the ARN.

## Phase 2 — Streams on the WO tables (CDK)

- Enable `stream=NEW_AND_OLD_IMAGES` on `WorkOrders` and `WorkOrderComments`. In-place CFN update, no replacement, additive. Consumer audit (refreshed 2026-07-23): **neither table has a stream, so no stream consumers can exist, and the tables have NO cross-stack consumers at all** — seahaven-slack-bot (the former sole reader) was decommissioned 2026-07-23 and its grants dropped from CDK. Re-run this audit before deploy anyway; the reasoning pattern (re-audit before assuming safety) is the point. Note the change in the README data-contract section. **Pre-deploy checks:** (a) confirm the prod tables still carry `RemovalPolicy.RETAIN` and their pre-migration logical IDs; (b) `cdk diff` must show the table changes as in-place updates — if either table shows `[Replacement]` (e.g. from an accidental logical-ID change), abort; these are RETAIN production tables and a replacement would orphan them.
- OLD_IMAGE is needed to detect the `wo_status → cancelled` transition (`work_order.cancelled` classification).

## Phase 3 — Emitter Lambda `workorder-shoc-emitter`

- Python 3.12, ARM64, explicit LogGroup via `common.make_function_log_group`, kebab-case name, `memory_size=256`, `timeout=60s` (POST timeout 10s per contract §2). Stdlib HTTP (`urllib.request`) — no new deps. Source at `lambdas/wo/shoc_emitter/`.
- **Ships DARK:** both event-source mappings deploy with `enabled=False` — the full stack (Lambda, queues, alarms, secret, rotation) deploys and is testable with zero deliveries while SHOC has no receiver, so our merge cadence never depends on Luby's. Activation is a deliberate one-line `enabled=True` PR after her receiver passes the shared HMAC test vectors. While disabled, stream records simply age out unprocessed (24 h retention) — that's fine; SHOC loads history via the read API at activation anyway.
- Two event-source mappings (one per stream): `parallelization_factor=1` and `bisect_batch_on_function_error=False` (both required for strict in-order), `batch_size=10`, `maximum_record_age` = 24h (stream retention), `retry_attempts=-1` (retry until expiry — blocking the shard is what preserves order), `on_failure` → SQS destination `workorder-shoc-emitter-failures` (14-day, SSL-enforced, house DLQ style).
- Handler logic:
  1. Deserialize the Dynamo stream image → plain JSON (Decimal-safe).
  2. Classify: `WorkOrders` INSERT → `work_order.created`; MODIFY → `work_order.updated`, or `work_order.cancelled` when OLD.wo_status ≠ `cancelled` ∧ NEW.wo_status = `cancelled`; `WorkOrderComments` INSERT → `work_order.comment_added`. REMOVE events: skip (no deletes in this pipeline). **Echo guard (forward-compat):** if NEW.`write_origin == "shoc-write-api"`, skip — a no-op branch today (nothing writes that attribute yet) that keeps the phase-2 write-back API from echoing SHOC's own writes back at it without re-touching this loop.
  3. Build envelope (contract §3); `delivery_id` = the stream record `eventID` (unique, stable across ESM retries); `occurred_at` = `ApproximateCreationDateTime`.
  4. Sign (contract §6) with `keys[0]` from the secret — cached per handbook Lambda pattern with a **5-minute TTL** so rotation propagates.
  5. POST with a `User-Agent: workorder-shoc-emitter/1` header; classify response per contract §7: `2xx` → done; `429/5xx/timeout/conn-error` → raise (ESM blocks + retries in order); other `4xx` → send record to `workorder-shoc-emitter-rejected` SQS queue, log structured warning, return success (a contract bug must not block the shard for 24h).
  6. Structured JSON logging for every delivery attempt/outcome (delivery_id, event_type, work_order_id, status code, latency) — Logs-Insights-queryable, same style as the processor's `sender_auth_rejected` records.
- Throughput note (`parallelization_factor=1`): ordering serializes per shard, but volume is ~760 events/day with bursts of a few events/second against a ~10s worst-case POST — a shard-blocking bottleneck only matters when SHOC is degraded, which is exactly when we *want* ordered blocking + the iterator-age alarm rather than parallel hammering. Accepted.
- Env config: `SHOC_WEBHOOK_URL` (non-sensitive endpoint URL — env var/CDK context per secrets-and-config.md; HMAC is the auth, the URL is not), `HMAC_SECRET_ARN`.
- IAM: stream read (`grant_stream_read` on both tables), `GetSecretValue` scoped to the one secret, `kms:Decrypt` on the CMK, SQS send to the two failure queues. **→ cross-family review.**
- CI plumbing (post-refactor, easy to miss): the emitter's and rotator's bundling commands need **new AST-pin coverage in `tests/test_bundle_consistency.py`** (the test currently only recognizes email_processor/web_ui command shapes — an unrecognized bundle ships unchecked); add `lambdas/wo/shoc_emitter` (and the rotator dir) to `pytest.ini`'s explicit `--cov=` list (unlisted dirs are invisible to the 80% floor); touch `tests/support/loader.py` only if a real sibling-name collision exists (per-directory resolution usually isolates it).

## Phase 4 — Monitoring (house alarm style: ALARM-only → `site-alerts`, `NOT_BREACHING`)

| Alarm | Metric | Threshold |
|---|---|---|
| `workorder-shoc-emitter-errors` | Lambda Errors Sum 5 min | > 0, eval 1 |
| `workorder-shoc-emitter-throttles` | Lambda Throttles Sum 5 min | > 0, eval 1 |
| `workorder-shoc-emitter-iterator-age` | IteratorAge Max 5 min | ≥ 600,000 ms (10 min lag = SHOC likely down), eval 3/2 |
| `workorder-shoc-emitter-failures-messages` | SQS visible Max 5 min | > 0, eval 1 |
| `workorder-shoc-emitter-rejected-messages` | SQS visible Max 5 min | > 0, eval 1 |

House rules made explicit: every alarm sets **only** `AlarmActions` (no `OKActions`, no `InsufficientDataActions` — `feedback_cloudwatch_alarms`), and every alarm must actually carry the `site-alerts` action (no action = monitoring theater). Errors/throttles/duration come from `common.add_standard_lambda_alarms`; iterator-age and the two SQS-visible alarms don't fit that helper's shape and stay bespoke — don't misread the helper as covering everything. `site-alerts` exists in seahaven-prod (live-verified 2026-07-23) via `Topic.from_topic_arn`, correctly on the `alias/seahaven-alarm-topics` CMK — never `alias/aws/sns`, whose key policy silently blocks CloudWatch publishes (`feedback_sns_alarm_topic_kms`). Post-deploy, verify delivery on one new alarm with `set-alarm-state` + `describe-alarm-history --history-item-type Action`.

Optional EMF `DeliveryOutcome` metric (delivered/rejected, mirroring the ParseMethod pattern) for a delivery-rate dashboard — cheap, same zero-latency EMF approach.

## Phase 5 — Replay tool + tests

- `scripts/replay_shoc_webhooks.py`: rebuild events from `WorkOrders`/`WorkOrderComments` for a `--work-order-id` list or `--since` window and re-POST with `"replay": true` (contract §8). Dry-run default, house script style. Documented with usage examples in the README Scripts section (operator runbook for the `-failures`/`-rejected` alarms).
- Unit tests (offline, no AWS — extend the pytest suite under the consolidated `tests/` conventions): signature generation golden vectors (shareable with Luby so both sides verify against identical vectors); stream-record → envelope mapping incl. Decimal handling and `null` fields; `cancelled` transition classification incl. already-cancelled MODIFYs (no false `cancelled` events); the `write_origin == "shoc-write-api"` skip branch; an enum golden test pinning emitted `wo_status`/`record_type` values to `lambdas/wo/email_processor/prompts.py` (enum drift fails CI here, not in SHOC prod); response-classification matrix (2xx/429/5xx/timeout/4xx); secret-cache TTL + unknown-kid refresh.
- Interaction note: PR #99 advisory A1 (AI-fallback `comment_time` nondeterminism) can duplicate comment rows on async retries → duplicate `comment_added` webhooks. SHOC dedupe on `comment_id`/`delivery_id` absorbs it; fixing A1 stays tracked separately.

## Phase 6 — Gates (before merge, in order)

1. `ruff check` + `ruff format --check` + `pytest` locally (pre-push hook enforced). Pre-flight: deploy credentials resolve to seahaven-prod (Phase 0 gate 4).
2. **GPT-4.1 cross-family review** (`cross_review.py`) of the complete policy surface **as a single diff** — Lambda execution-role policy + KMS key policy + secret resource policy + rotation-Lambda role together, never piecemeal (cross-policy gaps hide between separately-reviewed fragments). Mandatory. All policy changes ship in this one PR so the review sees the complete surface.
3. **`/sh-security-review`** — mandatory (auth surface: HMAC design + rotation; untrusted-input: email-derived data serialized into outbound requests; cross-account exposure). Resolve every confirmed critical/high.
4. PR against `main`, CI green (`gh pr checks`), no admin-merge.

## Phase 7 — Docs + memory (same conversation as ship, per global instructions)

- README: new emitter Lambda, streams, secret + CMK, alarms, data-flow diagram, SHOC consumer contract section, replay-script runbook, the DESTROY-not-RETAIN rationale on the secret — and **remove the stale "Consumer (read-only) — data contract" paragraph naming seahaven-slack-bot** (decommissioned).
- Confluence **AWS Architecture Map** (id 1540098): add emitter, streams, the `workorder-ingest/shoc-webhook-hmac` secret + CMK, the two failure SQS queues, and the cross-account SHOC edge (both the webhook POST and the secret-read trust) to the WorkorderIngestStack subgraph. Read the page first — the 2026-07-23 migration already edited it (v35); don't clobber those edits.
- Memory: update `project_procurement_ingest.md` and `project_shoc.md` (cross-referenced) — webhook shipped, contract Rev, secret/CMK names + exact cross-account grant ARNs, rotation design, DESTROY-policy rationale, dark-ship/activation state, read API replaces SyncController, echo-guard design.

## Deploy/cutover order

1. `cdk deploy` (streams + secret + emitter ship in one deploy, **ESMs disabled** — zero deliveries, zero alarm noise; our merge cadence never depends on SHOC's readiness).
2. Luby builds the receiver against the contract + shared test vectors and loads history via the read API; receiver passes the vectors at the target env.
3. Activation: one-line `enabled=True` PR. The ESM starts at `LATEST` — no historical flood.
4. Verify end-to-end with a live APM email; watch `iterator-age` + `rejected` alarms for 48h.
5. SyncController's Dynamo scan (incl. `SyncVendorReplies`) retires with the mgmt decommission; the read API is the standing reconciliation path.

## Explicit non-goals (this PR)

- No DynamoDB retirement / write-path switch (kept out for scope discipline — the former blocker, seahaven-slack-bot, is decommissioned; revisit after SHOC prod + mgmt decommission).
- No PO-pipeline webhook.
- No change to `workorder-email-processor` or its parser.
- No write-back endpoints (phase 2 of the read API, own PR + own IAM cross-review).