Merge pull request #13 from Sea-Haven-Industries/feature/phase-6-docs

Phase 6: docs and re-enable CD on merge

README.md

@@ -2,8 +2,10 @@
 
 Daily analysis of Amazon **APM work-order** "Last Comment" data for Sea Haven
 facility ops. A curated daily filter-view export (~350 work orders) is classified
-on two axes, pushed to Slack, and surfaced in a self-hosted Grafana dashboard.
-Replaces a legacy Google Apps Script + versioned-Google-Sheet workflow.
+on two axes (comment intent + structured `WO Status`/`Hold Reason`), pushed to
+Slack as a daily summary plus a batched 3rd-escalation alert, and surfaced in a
+self-hosted Grafana dashboard. Replaces a legacy Google Apps Script +
+versioned-Google-Sheet workflow.
 
 This is a **sibling concern** to the `apm@` email pipeline in `procurement-ingest`
 — it consumes a different feed (the manual export) and does **not** read those
@@ -17,79 +19,123 @@ backed by S3 + Athena (Grafana cannot query DynamoDB).
 
 ```
 APM export (xlsx/csv)
-  → S3 raw/  (direct upload OR local launchd drop-folder)
+  → S3 raw/  (direct `aws s3 cp` OR local launchd drop-folder)
       → classifier Lambda (HTML strip + two-axis classify, Haiku fallback)
-          → S3 analytics/dt=YYYY-MM-DD/  (per-WO daily snapshot, Parquet)
-                → Glue table → Athena → Grafana (self-hosted EC2, VPN-only, kiosk)
-          → slack-post Lambda (reads today + yesterday partitions)
+          ├→ S3 analytics/dt=YYYY-MM-DD/  (per-WO snapshot, Parquet)
+          │     → Glue table (partition projection) → Athena → Grafana (EC2, office-IP, kiosk)
+          ├→ S3 meta/dt=YYYY-MM-DD/  (summary.json + details.json — NOT in the table prefix)
+          └→ async-invoke slack-post Lambda
                 → daily summary post  [📊 Open dashboard button]
-                → standalone batched 3rd-escalation alert (suppressed if zero)
+                → standalone batched 3rd-escalation alert (suppressed when zero)
+                → drill-down modals via apm-wo.seahaven.com (signature-verified)
 ```
 
-Two CDK stacks:
+Two CDK stacks (`cdk/app.py` instantiates both):
 
 | Stack | Resources |
 |---|---|
-| `apm-wo-analysis-pipeline` | S3 exports bucket, classifier + slack-post Lambdas, Glue database, Athena workgroup, IAM |
-| `apm-wo-analysis-grafana` | EC2 (Grafana OSS), internal ALB, security group, Route53, Athena datasource IAM role |
+| `apm-wo-analysis-pipeline` | S3 exports bucket, drop-uploader IAM user, classifier + slack-post + slack-interactions Lambdas, classifier DLQ, Glue DB + projection table, Athena workgroup, HTTP API (`apm-wo.seahaven.com`), SSM param, scoped IAM |
+| `apm-wo-analysis-grafana` | EC2 (Grafana OSS), internet-facing **office-IP-restricted** ALB (`grafana.seahaven.com`), security groups, instance IAM role, Route53 alias, daily DLM snapshot, dashboards-as-code S3 deployment |
+
+## AWS Resources
+
+| Resource | Name | Purpose |
+|---|---|---|
+| S3 bucket | `apm-wo-analysis-exports-328440206208` | Single bucket. Prefixes: `raw/` (incoming, 90-day expiry), `analytics/` (Parquet snapshots, kept), `meta/` (summary/details JSON), `athena-results/` (query output, 30-day expiry), `grafana-config/` (dashboards-as-code). SSE-S3, BPA-all, enforce-SSL, `RETAIN`. |
+| IAM user | `apm-wo-drop-uploader` | Drop-folder identity; `s3:PutObject` on `raw/*` only. Access key created out-of-band, stored in local `apm-wo-drop` profile. |
+| Glue database | `apm_wo_analysis` | Analytics catalog. |
+| Glue table | `apm_wo_snapshots` | External Parquet table over `analytics/`, **partition projection** on `dt` (date, `2026-01-01..NOW`) — no crawler, no `MSCK`. 17-column snapshot schema. |
+| Athena workgroup | `apm-wo-analysis` | Enforced result location `athena-results/`, SSE-S3. |
+| SQS queue | `apm-wo-analysis-classifier-dlq` | Dead-letter for failed classifier async invocations (14-day retention). |
+| SSM parameter | `/apm-wo-analysis/grafana-base-url` | Grafana dashboard URL for the 📊 button / modal links (ops-editable, no redeploy). |
+| HTTP API + domain | `apm-wo.seahaven.com` → `POST /slack/interactions` | Slack interactivity endpoint. Stage throttled 10 rps / 20 burst. `*.seahaven.com` ACM cert; Route53 alias. |
+| EC2 instance | Grafana (`t4g.small`, AL2023, ARM64) | Self-hosted Grafana OSS in `seahaven-vpc` private subnets, IMDSv2-only, SSM-managed. gp3 20 GB **encrypted**, `DeleteOnTermination=false`, tagged `apm-grafana-backup`. |
+| ALB | Grafana ALB (`grafana.seahaven.com`) | Internet-facing, HTTPS 443, SG admits **only office CIDRs** (`47.21.61.4/32`, `96.250.164.146/32`); forwards to instance:3000, health `/api/health`. |
+| DLM policy | Grafana volume backup | Daily snapshot (07:00 UTC) of the tagged instance, 7 retained. |
+| Route53 | `apm-wo.seahaven.com`, `grafana.seahaven.com` | Aliases in zone `Z06652411XKH89KTZD3XA` (`seahaven.com`). |
+
+## Lambda Functions
+
+All Python 3.12, ARM64, explicit LogGroup (`/aws/lambda/<name>`, 60-day retention).
+
+| Function | Trigger | Purpose |
+|---|---|---|
+| `apm-wo-analysis-classifier` | S3 `ObjectCreated` on `raw/*.xlsx|.csv` | Parse export, two-axis classify each non-blank-comment row, write per-WO Parquet to `analytics/dt=…/` and `summary.json`/`details.json` to `meta/dt=…/`, then async-invoke slack-post. 512 MB / 120 s. AWS-managed SDK-for-pandas layer (`AWSSDKPandas-Python312-Arm64:27`); DLQ attached. |
+| `apm-wo-analysis-slack-post` | Async-invoked by the classifier (`{"dt": …}`) | Read today's + yesterday's `meta/.../summary.json`, post the daily summary, and (only when `third_escalation_count > 0`) the batched 3rd-escalation alert from `details.json`. 256 MB / 30 s. |
+| `apm-wo-analysis-slack-interactions` | HTTP API `POST /slack/interactions` | Verify the Slack request signature, read `meta/.../details.json`, and `views.open` a filtered WO-list modal within Slack's 3 s `trigger_id` window. 256 MB / 30 s. |
+
+## Configuration
+
+### Secrets Manager (names only — created out-of-band, never in CloudFormation)
+| Secret | Purpose |
+|---|---|
+| `apm-wo-analysis/anthropic-api-key` | Claude Haiku fallback for ambiguous free-text comments. |
+| `apm-wo-analysis/slack-credentials` | JSON `{ botToken, signingSecret, channelId }` for the reused Slack app. |
+
+### SSM Parameters
+| Parameter | Purpose |
+|---|---|
+| `/apm-wo-analysis/grafana-base-url` | Grafana dashboard deep-link base (`https://grafana.seahaven.com/d/apm-wo/...`). |
+
+### Environment Variables (non-secret)
+- **classifier:** `APM_HAIKU_FALLBACK` (`on`/`off`), `SLACK_POST_FUNCTION_NAME`.
+- **slack-post / slack-interactions:** `SLACK_SECRET_NAME`, `DASHBOARD_URL_PARAM`, `ANALYTICS_BUCKET`.
+
+### GitHub repo secret
+- `AWS_DEPLOY_ROLE_ARN` — the OIDC deploy role `githubdeploy-apm-wo-analysis`.
+
+### CDK context (`cdk/cdk.json`)
+`wildcardCertArn`, `hostedZoneId`/`hostedZoneName`, `slackInteractionsDomain`, `grafanaDomain`, `grafanaVpcId`/`grafanaAzs`/`grafana{Public,Private}SubnetIds`, `officeCidrs`, `athenaPluginVersion` (`3.2.0`).
 
 ## The classification model
 
 **Always two-axis, never comment-only.** The legacy script's central flaw was
 reading only the comment while ignoring `WO Status` + `Hold Reason`, which left
 ~17% in "Other". The two-axis model cuts that to ~9% before any AI — and on the
-real 347-row export the current implementation lands "Other" at **5.2%** (18
-rows) with **18 mismatches** flagged.
+real 347-row export the implementation lands "Other" at **5.2%** (18 rows) with
+**18 mismatches** flagged.
 
 - **Axis 1 — comment intent:** regex over the HTML-stripped `Last Comment`,
   most-specific first (escalations → SIM ticket → vendor no-show → scheduling →
   reports → completion → … → other).
-- **Axis 2 — structured state:** `Hold Reason` → category and `WO Status`
+- **Axis 2 — structured state:** `Hold Reason` → category, and `WO Status`
   signals (`RCAN`→Cancelled, `H` corroborates On Hold, `IP`/`R`/`RR` in-flight).
 - **Resolution:** comment intent wins when confident → else structured state →
-  else `Other`. A **Claude Haiku** fallback (Secrets Manager) is reserved for
+  else `Other`. A **Claude Haiku** fallback (Secrets Manager key) is reserved for
   ambiguous free-text with no structured signal.
 - **Mismatch detector (a feature):** flags when comment intent contradicts
-  structured state. Surfaced, never suppressed.
+  structured state (e.g. "completed" while `WO Status` is `IP`). Surfaced, never
+  suppressed.
 
-The authoritative spec lives in [`CLAUDE.md`](./CLAUDE.md); the implementation is
-in `lambdas/classifier/classify.py` (owned by the `classifier-engineer` agent).
-The S3-triggered `lambdas/classifier/handler.py` parses each export, classifies
-every non-blank-comment row, writes a per-WO Parquet snapshot to
-`analytics/dt=YYYY-MM-DD/` (registering the Glue partition via awswrangler), and
-emits a `summary.json` for the Phase 4 slack-post Lambda.
+Authoritative spec: [`CLAUDE.md`](./CLAUDE.md). Implementation:
+`lambdas/classifier/classify.py` (logic) and `lambdas/classifier/handler.py`
+(S3 → Parquet + JSON + invoke).
 
 ## Repository layout
 
 ```
 cdk/
   app.py                 CDK entry point — instantiates both stacks
-  cdk.json
+  cdk.json               context: cert, zone, subnets, office CIDRs, plugin version
   requirements.txt       aws-cdk-lib==2.253.1, constructs>=10.6.0
+  assets/
+    grafana_userdata.sh  EC2 bootstrap: install Grafana + Athena plugin, S3 config sync
   stacks/
-    pipeline_stack.py    S3, Lambdas, Glue, Athena, IAM
-    grafana_stack.py     VPC import, EC2, ALB, SG, Route53, datasource role
+    pipeline_stack.py    S3, Lambdas, DLQ, Glue, Athena, HTTP API, IAM
+    grafana_stack.py     VPC import, EC2, ALB, SG, Route53, instance role, DLM
 lambdas/
-  classifier/            S3-triggered: parse → two-axis classify → Parquet
-  slack_post/            builds + posts the daily summary and alert
+  classifier/            S3-triggered: parse → two-axis classify → Parquet + meta JSON
+  slack_post/            blockkit.py (builders), handler.py (post), interactions.py
+                         (modals), slackio.py (Secrets/SSM/S3/signature)
 grafana/
   provisioning/          Athena datasource + dashboard provider (as code)
-  dashboards/            committed dashboard JSON (source of truth)
+  dashboards/            apm-work-orders.json (uid apm-wo) — source of truth
+slack/manifest.yaml      Slack app manifest (interactivity request URL)
 scripts/                 local drop-folder uploader + launchd plist
-tests/                   classifier smoke test
+tests/                   classifier smoke test + offline synth/blockkit assertions
 docs/BUILD.md            phased, end-to-end build guide
 ```
 
-## Configuration
-
-| Where | What |
-|---|---|
-| **Secrets Manager** | `apm-wo-analysis/anthropic-api-key` (Haiku fallback); `apm-wo-analysis/slack-credentials` = `{ botToken, signingSecret, channelId }` (reused Slack app). |
-| **SSM Parameter Store** | `/apm-wo-analysis/grafana-base-url` (the 📊 dashboard button / modal overflow link; ops-editable). |
-| **GitHub repo secret** | `AWS_DEPLOY_ROLE_ARN` — the OIDC deploy role `githubdeploy-apm-wo-analysis`. |
-
-No secrets in Lambda environment variables.
-
 ## Ingestion (no email)
 
 The export reaches S3 by **direct upload or a local drop-folder**, never SES/email.
@@ -97,72 +143,120 @@ The export reaches S3 by **direct upload or a local drop-folder**, never SES/ema
 - **Direct:** `aws s3 cp ./export.xlsx s3://apm-wo-analysis-exports-328440206208/raw/`
 - **Drop-folder (optional zero-touch):** a launchd agent (`scripts/apm-wo-uploader.sh`
   + `scripts/com.seahaven.apm-wo-uploader.plist`) that watches `~/apm-wo-drop/`,
-  uploads new `.xlsx`/`.csv` files to `raw/`, and archives them to `uploaded/`.
-  It uploads with the scoped `apm-wo-drop` AWS profile (IAM user
-  `apm-wo-drop-uploader` — `s3:PutObject` on `raw/*` only).
+  uploads new `.xlsx`/`.csv` files to `raw/`, and archives them locally. Uploads
+  with the scoped `apm-wo-drop` profile (IAM user `apm-wo-drop-uploader`).
 
-  Install (the runnable copy **must** live outside `~/Documents` — macOS TCC
-  sandbox; a repo-path script fails silently with `LastExitStatus=32256`):
+  Install — the runnable copy and watched folder **must** live outside `~/Documents`
+  (macOS TCC sandbox; a repo-path script fails silently with `LastExitStatus=32256`):
   ```bash
   install -d "$HOME/.local/bin" "$HOME/apm-wo-drop"
   cp scripts/apm-wo-uploader.sh "$HOME/.local/bin/apm-wo-uploader.sh"
   chmod +x "$HOME/.local/bin/apm-wo-uploader.sh"
   cp scripts/com.seahaven.apm-wo-uploader.plist "$HOME/Library/LaunchAgents/"
   launchctl load -w "$HOME/Library/LaunchAgents/com.seahaven.apm-wo-uploader.plist"
+  aws configure --profile apm-wo-drop    # one-time, with the uploader's access key
   ```
-  Re-copy the script to `~/.local/bin` after editing the repo source. Configure
-  the profile once with the uploader's access key:
-  `aws configure --profile apm-wo-drop`.
 
-The classifier Lambda is S3-triggered on the `raw/` prefix regardless of path
-(any `.xlsx`/`.csv` landing under `raw/` invokes it).
-
-## Deployment
-
-CI/CD via the org reusable workflows (no manual prod deploys):
-
-- **CI** (`.github/workflows/ci.yaml`) → `ci-python-sam.yaml@main`: ruff + `cdk synth`.
-- **Deploy** (`.github/workflows/deploy.yaml`) → `cd-cdk.yaml@main`: OIDC assume-role,
-  `cdk deploy --all`, single-flight concurrency.
-
-The OIDC deploy role must exist **before** the first deploy. Deploy order:
-
-```bash
-cd cdk && pip install -r requirements.txt
-cdk deploy apm-wo-analysis-pipeline   # S3, Glue, Athena, Lambdas, IAM
-cdk deploy apm-wo-analysis-grafana    # EC2, ALB, SG, Route53, datasource role
-```
+Any `.xlsx`/`.csv` landing under `raw/` invokes the classifier.
 
 ## Local development
 
-- pyenv Python 3.12; `ruff check` + `ruff format --check` before pushing (hook-enforced).
+- pyenv Python 3.12. `ruff check` + `ruff format --check` before pushing (hook-enforced).
+- Tests: `python -m pytest tests/ -q` (60 tests — classifier smoke test against a real
+  export + offline `cdk.assertions` synth checks + Block Kit builders). No AWS needed.
 - Smoke-test the classifier against a **real export** before declaring any
   classification change done: `~/Downloads/_documents/Sheet1-1.xlsx`.
-- `cdk synth` must pass in CI before merge.
+- `cdk synth` must pass in CI before merge (**Docker required** — Lambda deps are
+  bundled for ARM64).
+
+## Deployment
+
+CI/CD via the org reusable workflows (no manual prod deploys in steady state):
+
+- **CI** (`.github/workflows/ci.yaml`) → `ci-python-sam.yaml@main`: ruff + `cdk synth`. Runs on PRs into `main`.
+- **Deploy** (`.github/workflows/deploy.yaml`) → `cd-cdk.yaml@main`: OIDC assume-role, `cdk deploy --all`, single-flight concurrency. Runs on push to `main`.
+
+Stack name/region/account: `apm-wo-analysis-{pipeline,grafana}` / us-east-1 / 328440206208.
+OIDC deploy role `githubdeploy-apm-wo-analysis` must exist before the first deploy.
+
+> **Note:** CD is **temporarily disabled** (deploy job gated `if: ${{ false }}` on
+> the phase-0 branch) while the Phase 0–5 stack is merged into `main`, to avoid a
+> deploy on every merge. **Re-enable as the first Phase 6 step** by reverting that
+> commit. Both stacks are already deployed manually and validated in prod.
+
+Manual deploy (emergency/reference; pipeline first so the bucket/table exist):
+```bash
+cd cdk && pip install -r requirements.txt
+cdk deploy apm-wo-analysis-pipeline   # S3, Glue, Athena, Lambdas, DLQ, API, IAM
+cdk deploy apm-wo-analysis-grafana    # EC2, ALB, SG, Route53, DLM, dashboards
+```
+
+## Operations
+
+**Verify it's working** — drop a real export into `raw/`, then within ~1 min:
+- `analytics/dt=<today>/*.parquet` and `meta/dt=<today>/{summary,details}.json` appear,
+- the daily summary posts to the WO Slack channel (+ alert if any 3rd escalations),
+- the Grafana dashboard renders over the office network (`grafana.seahaven.com`).
+
+**Logs:** CloudWatch `/aws/lambda/apm-wo-analysis-{classifier,slack-post,slack-interactions}` (60-day retention).
+
+**Failure modes:**
+- Classifier failure (malformed export, transient error) → after Lambda retries, the event lands in **`apm-wo-analysis-classifier-dlq`**. Check the DLQ if a day's data is missing.
+- Slack post failure is best-effort and does **not** fail classification (data still lands in S3).
+- Grafana down → check the instance via **SSM Session Manager** (no SSH); `systemctl status grafana-server`; ALB target health.
+
+**Reprocess a day:** re-upload the same export to `raw/` — the classifier uses
+`overwrite_partitions`, so a same-day re-run replaces that `dt` partition idempotently.
+
+**Grafana admin:** access is office-IP-restricted at the ALB; the instance is
+SSM-only. Dashboards are provisioned from `grafana-config/` in S3 (synced on boot
+and by a 15-min systemd timer); **edit dashboards in-repo, not in the UI**
+(`allowUiUpdates: false`). `grafana.db` lives on the RETAIN'd gp3 volume and is
+snapshotted daily by DLM.
+
+## Documentation
+
+- **Confluence "AWS Architecture Map"** (IT space, page **1540098**): a Mermaid
+  subgraph for this stack — **done** (added under *Serverless Applications*, plus
+  rows in CI/CD Pipelines, EC2 Inventory, and Key Data Stores).
+- **Confluence stack page "APM WO Analysis (apm-wo-analysis)"** (IT space, page
+  **8749057**, under *AWS Cloud Infrastructure*): full resource/Lambda/secrets
+  detail — **done**.
+- **Slack Apps Inventory** (Confluence page 524569): "APM Work Orders" (new dedicated
+  app, App ID `A0B6P28V64B`) added — **done**.
+- This README + `docs/BUILD.md` (phased build guide) + `CLAUDE.md` (domain spec).
+
+## Notes / Gotchas
+
+- **Partition date** comes from the **S3 event time**, not the Lambda wall-clock —
+  stable across retries and the midnight boundary.
+- **`meta/` vs `analytics/`:** summary/details JSON must stay **out** of `analytics/` —
+  Athena reads every object in the table prefix as Parquet and chokes on JSON.
+- **Classifier deps** (awswrangler/pandas/pyarrow/numpy) come from the AWS-managed
+  SDK-for-pandas **layer** — bundling them blows Lambda's 250 MB unzipped limit.
+  Verify the pinned layer ARN/version on region or runtime changes.
+- **Grafana dashboard contract:** dashboard `uid` must stay `apm-wo` (the Slack 📊
+  button deep-links to `d/apm-wo`). Athena plugin query key is `rawSQL` (capital).
+  Datasource `authType: default` (the EC2 instance role; `ec2_iam_role` is rejected
+  by the plugin). Config sync uses `aws s3 sync --exact-timestamps` (plain sync
+  skips same-size edits). Template vars use `refresh: 1` (on load).
+- **No Client VPN exists** — "VPN-only" Grafana is realized as **office-IP SG
+  restriction**. `seahaven-vpc` has a single NAT (one AZ) for instance egress.
+- **Slack interactions endpoint** is unauthenticated at the gateway **by design**;
+  the Lambda verifies the Slack signature (replay window + HMAC). Stage-throttled.
+
+### Known operational debt
+- Re-enable CD (revert the phase-0 disable) once the stack is merged.
+- One **clean instance replacement** is owed to validate the committed user-data
+  from a cold boot and to apply root-volume encryption (can't encrypt in place).
+- Deferred review NITs: `print()`→`logging`, source-IP logging on signature
+  failure, S3 versioning, the `'${site:raw}'` WO-table SQL tidy, and a
+  CIDR-maintenance note in the runbook.
 
 ## Status
 
-**Phase 5 — Grafana (in review); build complete, docs remain.** Build-out per
-[`docs/BUILD.md`](./docs/BUILD.md): ingestion → classifier → Glue/Athena → Slack
-→ Grafana → docs.
-
-- **Phase 0** scaffold — merged-pending (PR #6).
-- **Phase 1** ingestion (S3 bucket, drop-folder uploader, OIDC deploy role) —
-  deployed; PR #7 open.
-- **Phase 2** classifier Lambda + Glue database + S3 `raw/` trigger — implemented
-  and `cdk synth`-green; PR #8 open (stacked on Phase 1, not yet deployed).
-- **Phase 3** `apm_wo_snapshots` projection table + Athena workgroup —
-  implemented and `cdk synth`-green; PR #9 open (stacked on Phase 2). The
-  classifier writes pure Parquet and holds no Glue access (projection handles
-  partitions).
-- **Phase 4** Slack post + interactions Lambdas (daily summary, batched
-  3rd-escalation alert, drill-down modals on `apm-wo.seahaven.com`) — implemented
-  and `cdk synth`-green; PR #10 open (stacked on Phase 3). App manifest in
-  `slack/manifest.yaml`.
-- **Phase 5** self-hosted Grafana (EC2 + ALB on `grafana.seahaven.com`,
-  office-IP-restricted, 7-panel dashboard as code) — implemented and
-  `cdk synth`-green; PR #11 open (stacked on Phase 4).
-- **Phase 6** (final docs / Confluence / runbook) — not started.
-
-All phase PRs are stacked (#6→#7→#8→#9→#10→#11) and **unmerged**; nothing is
-deployed yet. Cross-review and `/security-review` are outstanding across the stack.
+Phases 2–5 implemented, **deployed to prod and validated end-to-end** (classifier,
+Slack post + alert + modal, Grafana dashboard). Stacked PRs **#6→#11** are open and
+unmerged; cross-review (#2/#4/#5) and `/security-review` of the two public endpoints
+are **cleared**. Phase 6 (this docs pass + Confluence + runbook) is in progress on
+`feature/phase-6-docs`.

docs/RUNBOOK.md

@@ -0,0 +1,187 @@
+# apm-wo-analysis — Operational Runbook
+
+Operational procedures and incident response for the daily APM work-order
+analysis pipeline. Account **328440206208** / **us-east-1**. Stacks
+`apm-wo-analysis-pipeline` and `apm-wo-analysis-grafana`. See [`README.md`](../README.md)
+for architecture and resource detail.
+
+**Admin access:** the Grafana box is **SSM Session Manager only** (no SSH/key pair):
+`aws ssm start-session --target <instance-id>`. AWS API via the `office_mac` /
+deploy roles. Grafana UI is office-IP-restricted at the ALB.
+
+---
+
+## 1. Incident Runbook: Missing daily APM WO analysis
+
+**Severity: High** — a day's WO analysis is missing: escalations (incl. 3rd-escalation
+alerts) aren't surfaced and the dashboard has no new snapshot. Recoverable by
+re-processing the export; not permanent data loss.
+
+### Detection
+- No **daily summary** in the WO Slack channel by the usual time (or no 3rd-escalation alert on a day one's expected).
+- **Grafana** shows no `dt = today` in the Snapshot Date dropdown / panels empty for today.
+- **Messages in `apm-wo-analysis-classifier-dlq`** (SQS) — strongest signal the classifier failed.
+- CloudWatch errors in `/aws/lambda/apm-wo-analysis-classifier` or `-slack-post`.
+
+### Context
+| Item | Value |
+|---|---|
+| Stacks | `apm-wo-analysis-pipeline`, `apm-wo-analysis-grafana` |
+| Lambdas | `apm-wo-analysis-classifier`, `-slack-post`, `-slack-interactions` |
+| S3 | `apm-wo-analysis-exports-328440206208` — `raw/`, `analytics/dt=…/`, `meta/dt=…/` |
+| SQS DLQ | `apm-wo-analysis-classifier-dlq` |
+| Glue / Athena | db `apm_wo_analysis`, table `apm_wo_snapshots`, workgroup `apm-wo-analysis` |
+| External | Slack, Anthropic API (Haiku fallback) |
+| Secrets (names) | `apm-wo-analysis/slack-credentials`, `apm-wo-analysis/anthropic-api-key` |
+| SSM | `/apm-wo-analysis/grafana-base-url` |
+
+### Triage
+1. **Export uploaded?** `aws s3 ls s3://apm-wo-analysis-exports-328440206208/raw/` — today's file present? Absent → upstream (§2.1), not the pipeline.
+2. **Classifier ran/failed?** `/aws/lambda/apm-wo-analysis-classifier` logs; peek the DLQ: `aws sqs receive-message --queue-url <dlq-url> --max-number-of-messages 1`.
+3. **Outputs written?** `aws s3 ls .../analytics/dt=<today>/` (Parquet) and `.../meta/dt=<today>/` (`summary.json`, `details.json`).
+4. **slack-post ran/failed?** `/aws/lambda/apm-wo-analysis-slack-post` logs — `SlackApiError` (`invalid_auth`, `not_in_channel`, `invalid_blocks`)?
+5. **Slack creds** valid + bot still in channel? (`apm-wo-analysis/slack-credentials`).
+6. **IAM/throttle:** grep logs for `AccessDenied` / throttling.
+7. **Recent change?** Any merge/deploy to `main` just before the failure.
+
+### Resolution (by root cause)
+1. **Export not uploaded** → `aws s3 cp <export>.xlsx s3://apm-wo-analysis-exports-328440206208/raw/`; then check the drop-folder agent (§2.1).
+2. **Classifier failed (DLQ)** → read the DLQ message; fix; **reprocess by re-uploading the export to `raw/`** (`overwrite_partitions` makes same-`dt` idempotent).
+3. **Outputs present, no Slack post** → re-invoke:
+   ```bash
+   aws lambda invoke --function-name apm-wo-analysis-slack-post \
+     --payload "$(printf '{"dt":"<YYYY-MM-DD>"}' | base64)" /tmp/out.json
+   ```
+   If Slack auth was the cause → rotate `apm-wo-analysis/slack-credentials` (and/or `/invite` the bot), then re-invoke (secret read per-call; no redeploy).
+4. **Code regression** → identify the PR, revert/hotfix, redeploy via CI (push to `main`).
+5. **Data present, Grafana empty** → datasource/dashboard issue (§2.3); check the instance via SSM.
+6. **Anthropic/Haiku down** → non-fatal (deterministic path still classifies ~95%); set classifier env `APM_HAIKU_FALLBACK=off` to bypass.
+
+### Post-Incident
+- Verify re-process → summary posts + Grafana shows today's `dt`.
+- Check for **other missed days** (gaps in `analytics/dt=…`/`meta/`) and reprocess each.
+- Update README/Confluence if knowledge changed; add a memory entry; add a test if code caused it.
+
+---
+
+## 2. Operational Procedures
+
+### 2.1 How the export gets uploaded
+
+The curated daily APM filter-view export (~350 WOs, `.xlsx`/`.csv`) reaches S3 by
+**direct upload or a local drop-folder** — never SES/email. Any object under
+`raw/` with a `.xlsx`/`.csv` suffix triggers the classifier.
+
+- **Direct:** `aws s3 cp ./export.xlsx s3://apm-wo-analysis-exports-328440206208/raw/`
+- **Drop-folder (zero-touch):** a launchd agent (`com.seahaven.apm-wo-uploader`)
+  watches `~/apm-wo-drop/`, uploads new files to `raw/` using the scoped
+  `apm-wo-drop` profile (IAM user **`apm-wo-drop-uploader`** — `s3:PutObject` on
+  `raw/*` only), and archives them locally. The runnable script lives at
+  `~/.local/bin/apm-wo-uploader.sh` (it and the watched folder **must** be outside
+  `~/Documents` — macOS TCC sandbox).
+
+**Verify the agent:**
+```bash
+launchctl list | grep apm-wo-uploader          # present + last exit 0
+tail -f ~/apm-wo-drop/uploader.log             # per-run logging (TBD: confirm log path)
+```
+**Common upstream issues:** agent unloaded (`launchctl load -w …plist`); script
+moved back into `~/Documents` (TCC blocks it — `LastExitStatus=32256`); `apm-wo-drop`
+access key expired/rotated (`aws configure --profile apm-wo-drop`).
+
+### 2.2 Grafana OS / app patching cadence
+
+The Grafana EC2 box (`t4g.small`, **Amazon Linux 2023**, ARM64) is the only
+patch-bearing piece — everything else is serverless. It's **reproducible from
+`cdk/assets/grafana_userdata.sh`**, so the preferred patch path is a **clean
+instance replacement** rather than long-lived in-place drift.
+
+- **OS (recommended monthly + on critical CVEs):** via SSM —
+  `sudo dnf upgrade --security -y && sudo reboot` (Session Manager, or an SSM
+  Run Command / Patch Manager maintenance window — **TBD: not yet automated**).
+- **Grafana OSS:** `sudo dnf upgrade grafana -y && sudo systemctl restart grafana-server`
+  (installed from the pinned `rpm.grafana.com` repo).
+- **Athena datasource plugin:** pinned to **`3.2.0`** in `cdk/cdk.json`
+  (`athenaPluginVersion`). Bump there, then redeploy/replace the instance.
+- **Preferred = clean replacement:** terminate the instance; `cdk deploy
+  apm-wo-analysis-grafana` relaunches it from the latest AL2023 AMI and re-runs
+  user-data (fresh Grafana + plugin + config sync). The root volume is
+  `DeleteOnTermination=false`, so detach/reuse or restore `grafana.db` (§2.4) if
+  local settings must persist. Validates the committed user-data from a cold boot.
+
+> **Outstanding:** one clean instance replacement is owed to validate cold-boot
+> user-data and apply root-volume encryption (encryption can't be added in place).
+
+### 2.3 Dashboard-JSON redeploy
+
+Source of truth is **`grafana/dashboards/apm-work-orders.json`** in this repo
+(uid **`apm-wo`**); the running instance is never the source of truth
+(`allowUiUpdates: false` — UI edits are reverted on the next sync).
+
+**Flow:** edit JSON in repo → `cdk deploy apm-wo-analysis-grafana` (the
+`BucketDeployment` uploads `grafana/` to `s3://…/grafana-config/`) → the instance
+syncs S3 → `/var/lib/grafana/dashboards/` (on boot + a **15-min systemd timer**)
+→ Grafana's file provider polls every **60 s** and reloads.
+
+**Apply immediately** (skip the timer) via SSM:
+```bash
+sudo /usr/local/bin/grafana-config-sync.sh    # pulls grafana-config/ from S3
+# Grafana file provider picks up the dashboard within ~60s
+```
+**Gotchas:**
+- The sync uses `aws s3 sync --exact-timestamps` — required so **same-size edits**
+  (e.g. a one-char query change) actually propagate.
+- **Datasource/provisioning** changes (`grafana/provisioning/*.yaml`) are loaded
+  at **startup** — after syncing, `sudo systemctl restart grafana-server` (a
+  dashboard-only change does **not** need a restart).
+- Athena query key is **`rawSQL`** (capital); datasource `authType: default`;
+  template vars `refresh: 1`. (See README "Notes / Gotchas".)
+
+### 2.4 Backup & restore (config + EBS / grafana.db)
+
+Two distinct layers:
+
+**Config (dashboards, datasources, provisioning)** — fully **reproducible from
+git** (`grafana/` → S3 `grafana-config/`). *Restore:* `cdk deploy
+apm-wo-analysis-grafana` (or `grafana-config-sync.sh` on the box). No snapshot needed.
+
+**Local state (`/var/lib/grafana/grafana.db`)** — Grafana's SQLite (admin user,
+any API keys, org prefs). Lives on the **gp3 root volume** (encrypted,
+`DeleteOnTermination=false`). Backed up by a **daily DLM snapshot** (07:00 UTC,
+7 retained) of the instance (tag `apm-grafana-backup=true`), policy in the
+grafana stack.
+
+*Restore from snapshot:*
+```bash
+# find the latest DLM snapshot
+aws ec2 describe-snapshots --owner-ids self \
+  --filters "Name=tag:aws:dlm:lifecycle-policy-id,Values=*" \
+  --query 'reverse(sort_by(Snapshots,&StartTime))[0].SnapshotId' --output text
+# create a volume from it and attach to a replacement instance, OR mount it and
+# copy /var/lib/grafana/grafana.db onto the new instance, then:
+sudo systemctl restart grafana-server
+```
+Because dashboards + datasource are provisioned from code, the only thing the
+snapshot uniquely protects is `grafana.db` (admin/login state) — low stakes; a
+fresh instance + provisioning recovers everything else.
+
+> **Note:** the Grafana **admin auth model is unsettled (TBD)** — the password was
+> reset ad-hoc during build/testing. Decide the intended model (fixed admin
+> password in Secrets Manager / SSO / anonymous view-only for the kiosk) and
+> document it here.
+
+### 2.5 Common failures (quick index)
+
+| Symptom | Likely cause | Go to |
+|---|---|---|
+| No daily Slack post / no new dashboard day | export not uploaded, classifier failed (DLQ), or slack-post failed | §1 |
+| Dashboard loads but all panels "No data" | datasource/auth, `rawSQL`, template `refresh`, or JSON in `analytics/` prefix | §2.3, README gotchas |
+| Grafana unreachable | instance down / ALB unhealthy / office IP changed (`officeCidrs`) | SSM triage; `aws elbv2 describe-target-health` |
+| Slack modal click does nothing / error | `apm-wo-analysis-slack-interactions`, API Gateway, or signing-secret mismatch | `/aws/lambda/apm-wo-analysis-slack-interactions` logs |
+| Exports never arrive in `raw/` | drop-folder agent unloaded / TCC / expired key | §2.1 |
+| Deploy not applying | OIDC role, CloudFormation rollback, Docker bundling | CloudFormation events; CI logs |
+
+---
+
+*Maintained in-repo (`docs/RUNBOOK.md`) and mirrored to Confluence. Update both
+when operational knowledge changes.*