adam/apm-wo-analysis
1
0
Fork 0
mirror of https://github.com/Sea-Haven-Industries/apm-wo-analysis.git synced 2026-10-05 15:22:00 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Merge pull request #13 from Sea-Haven-Industries/feature/phase-6-docs
Some checks are pending
Deploy / deploy (push) Waiting to run
Details

Browse source 
Phase 6: docs and re-enable CD on merge
This commit is contained in:
Adam Moussa 2026-05-29 14:03:59 -04:00 • committed by GitHub
commit 54ef5400b1
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
2 changed files with 371 additions and 90 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

274
README.md
View file

@ -2,8 +2,10 @@
Daily analysis of Amazon **APM work-order** "Last Comment" data for Sea Haven Daily analysis of Amazon **APM work-order** "Last Comment" data for Sea Haven
facility ops. A curated daily filter-view export (~350 work orders) is classified facility ops. A curated daily filter-view export (~350 work orders) is classified
on two axes, pushed to Slack, and surfaced in a self-hosted Grafana dashboard. on two axes (comment intent + structured `WO Status`/`Hold Reason`), pushed to
Replaces a legacy Google Apps Script + versioned-Google-Sheet workflow. Slack as a daily summary plus a batched 3rd-escalation alert, and surfaced in a
self-hosted Grafana dashboard. Replaces a legacy Google Apps Script +
versioned-Google-Sheet workflow.
This is a **sibling concern** to the `apm@` email pipeline in `procurement-ingest` This is a **sibling concern** to the `apm@` email pipeline in `procurement-ingest`
— it consumes a different feed (the manual export) and does **not** read those — it consumes a different feed (the manual export) and does **not** read those
@ -17,79 +19,123 @@ backed by S3 + Athena (Grafana cannot query DynamoDB).
``` ```
APM export (xlsx/csv) APM export (xlsx/csv)
→ S3 raw/ (direct upload OR local launchd drop-folder) → S3 raw/ (direct `aws s3 cp` OR local launchd drop-folder)
→ classifier Lambda (HTML strip + two-axis classify, Haiku fallback) → classifier Lambda (HTML strip + two-axis classify, Haiku fallback)
→ S3 analytics/dt=YYYY-MM-DD/ (per-WO daily snapshot, Parquet) ├→ S3 analytics/dt=YYYY-MM-DD/ (per-WO snapshot, Parquet)
→ Glue table → Athena → Grafana (self-hosted EC2, VPN-only, kiosk) │ → Glue table (partition projection) → Athena → Grafana (EC2, office-IP, kiosk)
→ slack-post Lambda (reads today + yesterday partitions) ├→ S3 meta/dt=YYYY-MM-DD/ (summary.json + details.json — NOT in the table prefix)
└→ async-invoke slack-post Lambda
→ daily summary post [📊 Open dashboard button] → daily summary post [📊 Open dashboard button]
→ standalone batched 3rd-escalation alert (suppressed if zero) → standalone batched 3rd-escalation alert (suppressed when zero)
→ drill-down modals via apm-wo.seahaven.com (signature-verified)
``` ```
Two CDK stacks: Two CDK stacks (`cdk/app.py` instantiates both):
| Stack | Resources | | Stack | Resources |
|---|---| |---|---|
| `apm-wo-analysis-pipeline` | S3 exports bucket, classifier + slack-post Lambdas, Glue database, Athena workgroup, IAM | | `apm-wo-analysis-pipeline` | S3 exports bucket, drop-uploader IAM user, classifier + slack-post + slack-interactions Lambdas, classifier DLQ, Glue DB + projection table, Athena workgroup, HTTP API (`apm-wo.seahaven.com`), SSM param, scoped IAM |
| `apm-wo-analysis-grafana` | EC2 (Grafana OSS), internal ALB, security group, Route53, Athena datasource IAM role | | `apm-wo-analysis-grafana` | EC2 (Grafana OSS), internet-facing **office-IP-restricted** ALB (`grafana.seahaven.com`), security groups, instance IAM role, Route53 alias, daily DLM snapshot, dashboards-as-code S3 deployment |
## AWS Resources
| Resource | Name | Purpose |
|---|---|---|
| S3 bucket | `apm-wo-analysis-exports-328440206208` | Single bucket. Prefixes: `raw/` (incoming, 90-day expiry), `analytics/` (Parquet snapshots, kept), `meta/` (summary/details JSON), `athena-results/` (query output, 30-day expiry), `grafana-config/` (dashboards-as-code). SSE-S3, BPA-all, enforce-SSL, `RETAIN`. |
| IAM user | `apm-wo-drop-uploader` | Drop-folder identity; `s3:PutObject` on `raw/*` only. Access key created out-of-band, stored in local `apm-wo-drop` profile. |
| Glue database | `apm_wo_analysis` | Analytics catalog. |
| Glue table | `apm_wo_snapshots` | External Parquet table over `analytics/`, **partition projection** on `dt` (date, `2026-01-01..NOW`) — no crawler, no `MSCK`. 17-column snapshot schema. |
| Athena workgroup | `apm-wo-analysis` | Enforced result location `athena-results/`, SSE-S3. |
| SQS queue | `apm-wo-analysis-classifier-dlq` | Dead-letter for failed classifier async invocations (14-day retention). |
| SSM parameter | `/apm-wo-analysis/grafana-base-url` | Grafana dashboard URL for the 📊 button / modal links (ops-editable, no redeploy). |
| HTTP API + domain | `apm-wo.seahaven.com` → `POST /slack/interactions` | Slack interactivity endpoint. Stage throttled 10 rps / 20 burst. `*.seahaven.com` ACM cert; Route53 alias. |
| EC2 instance | Grafana (`t4g.small`, AL2023, ARM64) | Self-hosted Grafana OSS in `seahaven-vpc` private subnets, IMDSv2-only, SSM-managed. gp3 20 GB **encrypted**, `DeleteOnTermination=false`, tagged `apm-grafana-backup`. |
| ALB | Grafana ALB (`grafana.seahaven.com`) | Internet-facing, HTTPS 443, SG admits **only office CIDRs** (`47.21.61.4/32`, `96.250.164.146/32`); forwards to instance:3000, health `/api/health`. |
| DLM policy | Grafana volume backup | Daily snapshot (07:00 UTC) of the tagged instance, 7 retained. |
| Route53 | `apm-wo.seahaven.com`, `grafana.seahaven.com` | Aliases in zone `Z06652411XKH89KTZD3XA` (`seahaven.com`). |
## Lambda Functions
All Python 3.12, ARM64, explicit LogGroup (`/aws/lambda/<name>`, 60-day retention).
| Function | Trigger | Purpose |
|---|---|---|
| `apm-wo-analysis-classifier` | S3 `ObjectCreated` on `raw/*.xlsx|.csv` | Parse export, two-axis classify each non-blank-comment row, write per-WO Parquet to `analytics/dt=…/` and `summary.json`/`details.json` to `meta/dt=…/`, then async-invoke slack-post. 512 MB / 120 s. AWS-managed SDK-for-pandas layer (`AWSSDKPandas-Python312-Arm64:27`); DLQ attached. |
| `apm-wo-analysis-slack-post` | Async-invoked by the classifier (`{"dt": …}`) | Read today's + yesterday's `meta/.../summary.json`, post the daily summary, and (only when `third_escalation_count > 0`) the batched 3rd-escalation alert from `details.json`. 256 MB / 30 s. |
| `apm-wo-analysis-slack-interactions` | HTTP API `POST /slack/interactions` | Verify the Slack request signature, read `meta/.../details.json`, and `views.open` a filtered WO-list modal within Slack's 3 s `trigger_id` window. 256 MB / 30 s. |
## Configuration
### Secrets Manager (names only — created out-of-band, never in CloudFormation)
| Secret | Purpose |
|---|---|
| `apm-wo-analysis/anthropic-api-key` | Claude Haiku fallback for ambiguous free-text comments. |
| `apm-wo-analysis/slack-credentials` | JSON `{ botToken, signingSecret, channelId }` for the reused Slack app. |
### SSM Parameters
| Parameter | Purpose |
|---|---|
| `/apm-wo-analysis/grafana-base-url` | Grafana dashboard deep-link base (`https://grafana.seahaven.com/d/apm-wo/...`). |
### Environment Variables (non-secret)
- **classifier:** `APM_HAIKU_FALLBACK` (`on`/`off`), `SLACK_POST_FUNCTION_NAME`.
- **slack-post / slack-interactions:** `SLACK_SECRET_NAME`, `DASHBOARD_URL_PARAM`, `ANALYTICS_BUCKET`.
### GitHub repo secret
- `AWS_DEPLOY_ROLE_ARN` — the OIDC deploy role `githubdeploy-apm-wo-analysis`.
### CDK context (`cdk/cdk.json`)
`wildcardCertArn`, `hostedZoneId`/`hostedZoneName`, `slackInteractionsDomain`, `grafanaDomain`, `grafanaVpcId`/`grafanaAzs`/`grafana{Public,Private}SubnetIds`, `officeCidrs`, `athenaPluginVersion` (`3.2.0`).
## The classification model ## The classification model
**Always two-axis, never comment-only.** The legacy script's central flaw was **Always two-axis, never comment-only.** The legacy script's central flaw was
reading only the comment while ignoring `WO Status` + `Hold Reason`, which left reading only the comment while ignoring `WO Status` + `Hold Reason`, which left
~17% in "Other". The two-axis model cuts that to ~9% before any AI — and on the ~17% in "Other". The two-axis model cuts that to ~9% before any AI — and on the
real 347-row export the current implementation lands "Other" at **5.2%** (18 real 347-row export the implementation lands "Other" at **5.2%** (18 rows) with
rows) with **18 mismatches** flagged. **18 mismatches** flagged.
- **Axis 1 — comment intent:** regex over the HTML-stripped `Last Comment`, - **Axis 1 — comment intent:** regex over the HTML-stripped `Last Comment`,
most-specific first (escalations → SIM ticket → vendor no-show → scheduling → most-specific first (escalations → SIM ticket → vendor no-show → scheduling →
reports → completion → … → other). reports → completion → … → other).
- **Axis 2 — structured state:** `Hold Reason` → category and `WO Status` - **Axis 2 — structured state:** `Hold Reason` → category, and `WO Status`
signals (`RCAN`→Cancelled, `H` corroborates On Hold, `IP`/`R`/`RR` in-flight). signals (`RCAN`→Cancelled, `H` corroborates On Hold, `IP`/`R`/`RR` in-flight).
- **Resolution:** comment intent wins when confident → else structured state → - **Resolution:** comment intent wins when confident → else structured state →
else `Other`. A **Claude Haiku** fallback (Secrets Manager) is reserved for else `Other`. A **Claude Haiku** fallback (Secrets Manager key) is reserved for
ambiguous free-text with no structured signal. ambiguous free-text with no structured signal.
- **Mismatch detector (a feature):** flags when comment intent contradicts - **Mismatch detector (a feature):** flags when comment intent contradicts
structured state. Surfaced, never suppressed. structured state (e.g. "completed" while `WO Status` is `IP`). Surfaced, never
suppressed.
The authoritative spec lives in [`CLAUDE.md`](./CLAUDE.md); the implementation is Authoritative spec: [`CLAUDE.md`](./CLAUDE.md). Implementation:
in `lambdas/classifier/classify.py` (owned by the `classifier-engineer` agent). `lambdas/classifier/classify.py` (logic) and `lambdas/classifier/handler.py`
The S3-triggered `lambdas/classifier/handler.py` parses each export, classifies (S3 → Parquet + JSON + invoke).
every non-blank-comment row, writes a per-WO Parquet snapshot to
`analytics/dt=YYYY-MM-DD/` (registering the Glue partition via awswrangler), and
emits a `summary.json` for the Phase 4 slack-post Lambda.
## Repository layout ## Repository layout
``` ```
cdk/ cdk/
app.py CDK entry point — instantiates both stacks app.py CDK entry point — instantiates both stacks
cdk.json cdk.json context: cert, zone, subnets, office CIDRs, plugin version
requirements.txt aws-cdk-lib==2.253.1, constructs>=10.6.0 requirements.txt aws-cdk-lib==2.253.1, constructs>=10.6.0
assets/
grafana_userdata.sh EC2 bootstrap: install Grafana + Athena plugin, S3 config sync
stacks/ stacks/
pipeline_stack.py S3, Lambdas, Glue, Athena, IAM pipeline_stack.py S3, Lambdas, DLQ, Glue, Athena, HTTP API, IAM
grafana_stack.py VPC import, EC2, ALB, SG, Route53, datasource role grafana_stack.py VPC import, EC2, ALB, SG, Route53, instance role, DLM
lambdas/ lambdas/
classifier/ S3-triggered: parse → two-axis classify → Parquet classifier/ S3-triggered: parse → two-axis classify → Parquet + meta JSON
slack_post/ builds + posts the daily summary and alert slack_post/ blockkit.py (builders), handler.py (post), interactions.py
(modals), slackio.py (Secrets/SSM/S3/signature)
grafana/ grafana/
provisioning/ Athena datasource + dashboard provider (as code) provisioning/ Athena datasource + dashboard provider (as code)
dashboards/ committed dashboard JSON (source of truth) dashboards/ apm-work-orders.json (uid apm-wo) — source of truth
slack/manifest.yaml Slack app manifest (interactivity request URL)
scripts/ local drop-folder uploader + launchd plist scripts/ local drop-folder uploader + launchd plist
tests/ classifier smoke test tests/ classifier smoke test + offline synth/blockkit assertions
docs/BUILD.md phased, end-to-end build guide docs/BUILD.md phased, end-to-end build guide
``` ```
## Configuration
| Where | What |
|---|---|
| **Secrets Manager** | `apm-wo-analysis/anthropic-api-key` (Haiku fallback); `apm-wo-analysis/slack-credentials` = `{ botToken, signingSecret, channelId }` (reused Slack app). |
| **SSM Parameter Store** | `/apm-wo-analysis/grafana-base-url` (the 📊 dashboard button / modal overflow link; ops-editable). |
| **GitHub repo secret** | `AWS_DEPLOY_ROLE_ARN` — the OIDC deploy role `githubdeploy-apm-wo-analysis`. |
No secrets in Lambda environment variables.
## Ingestion (no email) ## Ingestion (no email)
The export reaches S3 by **direct upload or a local drop-folder**, never SES/email. The export reaches S3 by **direct upload or a local drop-folder**, never SES/email.
@ -97,72 +143,120 @@ The export reaches S3 by **direct upload or a local drop-folder**, never SES/ema
- **Direct:** `aws s3 cp ./export.xlsx s3://apm-wo-analysis-exports-328440206208/raw/` - **Direct:** `aws s3 cp ./export.xlsx s3://apm-wo-analysis-exports-328440206208/raw/`
- **Drop-folder (optional zero-touch):** a launchd agent (`scripts/apm-wo-uploader.sh` - **Drop-folder (optional zero-touch):** a launchd agent (`scripts/apm-wo-uploader.sh`
+ `scripts/com.seahaven.apm-wo-uploader.plist`) that watches `~/apm-wo-drop/`, + `scripts/com.seahaven.apm-wo-uploader.plist`) that watches `~/apm-wo-drop/`,
uploads new `.xlsx`/`.csv` files to `raw/`, and archives them to `uploaded/`. uploads new `.xlsx`/`.csv` files to `raw/`, and archives them locally. Uploads
It uploads with the scoped `apm-wo-drop` AWS profile (IAM user with the scoped `apm-wo-drop` profile (IAM user `apm-wo-drop-uploader`).
`apm-wo-drop-uploader` — `s3:PutObject` on `raw/*` only).
Install (the runnable copy **must** live outside `~/Documents` — macOS TCC Install — the runnable copy and watched folder **must** live outside `~/Documents`
sandbox; a repo-path script fails silently with `LastExitStatus=32256`): (macOS TCC sandbox; a repo-path script fails silently with `LastExitStatus=32256`):
```bash ```bash
install -d "$HOME/.local/bin" "$HOME/apm-wo-drop" install -d "$HOME/.local/bin" "$HOME/apm-wo-drop"
cp scripts/apm-wo-uploader.sh "$HOME/.local/bin/apm-wo-uploader.sh" cp scripts/apm-wo-uploader.sh "$HOME/.local/bin/apm-wo-uploader.sh"
chmod +x "$HOME/.local/bin/apm-wo-uploader.sh" chmod +x "$HOME/.local/bin/apm-wo-uploader.sh"
cp scripts/com.seahaven.apm-wo-uploader.plist "$HOME/Library/LaunchAgents/" cp scripts/com.seahaven.apm-wo-uploader.plist "$HOME/Library/LaunchAgents/"
launchctl load -w "$HOME/Library/LaunchAgents/com.seahaven.apm-wo-uploader.plist" launchctl load -w "$HOME/Library/LaunchAgents/com.seahaven.apm-wo-uploader.plist"
aws configure --profile apm-wo-drop # one-time, with the uploader's access key
``` ```
Re-copy the script to `~/.local/bin` after editing the repo source. Configure
the profile once with the uploader's access key:
`aws configure --profile apm-wo-drop`.
The classifier Lambda is S3-triggered on the `raw/` prefix regardless of path Any `.xlsx`/`.csv` landing under `raw/` invokes the classifier.
(any `.xlsx`/`.csv` landing under `raw/` invokes it).
## Deployment
CI/CD via the org reusable workflows (no manual prod deploys):
- **CI** (`.github/workflows/ci.yaml`) → `ci-python-sam.yaml@main`: ruff + `cdk synth`.
- **Deploy** (`.github/workflows/deploy.yaml`) → `cd-cdk.yaml@main`: OIDC assume-role,
`cdk deploy --all`, single-flight concurrency.
The OIDC deploy role must exist **before** the first deploy. Deploy order:
```bash
cd cdk && pip install -r requirements.txt
cdk deploy apm-wo-analysis-pipeline # S3, Glue, Athena, Lambdas, IAM
cdk deploy apm-wo-analysis-grafana # EC2, ALB, SG, Route53, datasource role
```
## Local development ## Local development
- pyenv Python 3.12; `ruff check` + `ruff format --check` before pushing (hook-enforced). - pyenv Python 3.12. `ruff check` + `ruff format --check` before pushing (hook-enforced).
- Tests: `python -m pytest tests/ -q` (60 tests — classifier smoke test against a real
export + offline `cdk.assertions` synth checks + Block Kit builders). No AWS needed.
- Smoke-test the classifier against a **real export** before declaring any - Smoke-test the classifier against a **real export** before declaring any
classification change done: `~/Downloads/_documents/Sheet1-1.xlsx`. classification change done: `~/Downloads/_documents/Sheet1-1.xlsx`.
- `cdk synth` must pass in CI before merge. - `cdk synth` must pass in CI before merge (**Docker required** — Lambda deps are
bundled for ARM64).
## Deployment
CI/CD via the org reusable workflows (no manual prod deploys in steady state):
- **CI** (`.github/workflows/ci.yaml`) → `ci-python-sam.yaml@main`: ruff + `cdk synth`. Runs on PRs into `main`.
- **Deploy** (`.github/workflows/deploy.yaml`) → `cd-cdk.yaml@main`: OIDC assume-role, `cdk deploy --all`, single-flight concurrency. Runs on push to `main`.
Stack name/region/account: `apm-wo-analysis-{pipeline,grafana}` / us-east-1 / 328440206208.
OIDC deploy role `githubdeploy-apm-wo-analysis` must exist before the first deploy.
> **Note:** CD is **temporarily disabled** (deploy job gated `if: ${{ false }}` on
> the phase-0 branch) while the Phase 0–5 stack is merged into `main`, to avoid a
> deploy on every merge. **Re-enable as the first Phase 6 step** by reverting that
> commit. Both stacks are already deployed manually and validated in prod.
Manual deploy (emergency/reference; pipeline first so the bucket/table exist):
```bash
cd cdk && pip install -r requirements.txt
cdk deploy apm-wo-analysis-pipeline # S3, Glue, Athena, Lambdas, DLQ, API, IAM
cdk deploy apm-wo-analysis-grafana # EC2, ALB, SG, Route53, DLM, dashboards
```
## Operations
**Verify it's working** — drop a real export into `raw/`, then within ~1 min:
- `analytics/dt=<today>/*.parquet` and `meta/dt=<today>/{summary,details}.json` appear,
- the daily summary posts to the WO Slack channel (+ alert if any 3rd escalations),
- the Grafana dashboard renders over the office network (`grafana.seahaven.com`).
**Logs:** CloudWatch `/aws/lambda/apm-wo-analysis-{classifier,slack-post,slack-interactions}` (60-day retention).
**Failure modes:**
- Classifier failure (malformed export, transient error) → after Lambda retries, the event lands in **`apm-wo-analysis-classifier-dlq`**. Check the DLQ if a day's data is missing.
- Slack post failure is best-effort and does **not** fail classification (data still lands in S3).
- Grafana down → check the instance via **SSM Session Manager** (no SSH); `systemctl status grafana-server`; ALB target health.
**Reprocess a day:** re-upload the same export to `raw/` — the classifier uses
`overwrite_partitions`, so a same-day re-run replaces that `dt` partition idempotently.
**Grafana admin:** access is office-IP-restricted at the ALB; the instance is
SSM-only. Dashboards are provisioned from `grafana-config/` in S3 (synced on boot
and by a 15-min systemd timer); **edit dashboards in-repo, not in the UI**
(`allowUiUpdates: false`). `grafana.db` lives on the RETAIN'd gp3 volume and is
snapshotted daily by DLM.
## Documentation
- **Confluence "AWS Architecture Map"** (IT space, page **1540098**): a Mermaid
subgraph for this stack — **done** (added under *Serverless Applications*, plus
rows in CI/CD Pipelines, EC2 Inventory, and Key Data Stores).
- **Confluence stack page "APM WO Analysis (apm-wo-analysis)"** (IT space, page
**8749057**, under *AWS Cloud Infrastructure*): full resource/Lambda/secrets
detail — **done**.
- **Slack Apps Inventory** (Confluence page 524569): "APM Work Orders" (new dedicated
app, App ID `A0B6P28V64B`) added — **done**.
- This README + `docs/BUILD.md` (phased build guide) + `CLAUDE.md` (domain spec).
## Notes / Gotchas
- **Partition date** comes from the **S3 event time**, not the Lambda wall-clock —
stable across retries and the midnight boundary.
- **`meta/` vs `analytics/`:** summary/details JSON must stay **out** of `analytics/` —
Athena reads every object in the table prefix as Parquet and chokes on JSON.
- **Classifier deps** (awswrangler/pandas/pyarrow/numpy) come from the AWS-managed
SDK-for-pandas **layer** — bundling them blows Lambda's 250 MB unzipped limit.
Verify the pinned layer ARN/version on region or runtime changes.
- **Grafana dashboard contract:** dashboard `uid` must stay `apm-wo` (the Slack 📊
button deep-links to `d/apm-wo`). Athena plugin query key is `rawSQL` (capital).
Datasource `authType: default` (the EC2 instance role; `ec2_iam_role` is rejected
by the plugin). Config sync uses `aws s3 sync --exact-timestamps` (plain sync
skips same-size edits). Template vars use `refresh: 1` (on load).
- **No Client VPN exists** — "VPN-only" Grafana is realized as **office-IP SG
restriction**. `seahaven-vpc` has a single NAT (one AZ) for instance egress.
- **Slack interactions endpoint** is unauthenticated at the gateway **by design**;
the Lambda verifies the Slack signature (replay window + HMAC). Stage-throttled.
### Known operational debt
- Re-enable CD (revert the phase-0 disable) once the stack is merged.
- One **clean instance replacement** is owed to validate the committed user-data
from a cold boot and to apply root-volume encryption (can't encrypt in place).
- Deferred review NITs: `print()`→`logging`, source-IP logging on signature
failure, S3 versioning, the `'${site:raw}'` WO-table SQL tidy, and a
CIDR-maintenance note in the runbook.
## Status ## Status
**Phase 5 — Grafana (in review); build complete, docs remain.** Build-out per Phases 2–5 implemented, **deployed to prod and validated end-to-end** (classifier,
[`docs/BUILD.md`](./docs/BUILD.md): ingestion → classifier → Glue/Athena → Slack Slack post + alert + modal, Grafana dashboard). Stacked PRs **#6→#11** are open and
→ Grafana → docs. unmerged; cross-review (#2/#4/#5) and `/security-review` of the two public endpoints
are **cleared**. Phase 6 (this docs pass + Confluence + runbook) is in progress on
- **Phase 0** scaffold — merged-pending (PR #6). `feature/phase-6-docs`.
- **Phase 1** ingestion (S3 bucket, drop-folder uploader, OIDC deploy role) —
deployed; PR #7 open.
- **Phase 2** classifier Lambda + Glue database + S3 `raw/` trigger — implemented
and `cdk synth`-green; PR #8 open (stacked on Phase 1, not yet deployed).
- **Phase 3** `apm_wo_snapshots` projection table + Athena workgroup —
implemented and `cdk synth`-green; PR #9 open (stacked on Phase 2). The
classifier writes pure Parquet and holds no Glue access (projection handles
partitions).
- **Phase 4** Slack post + interactions Lambdas (daily summary, batched
3rd-escalation alert, drill-down modals on `apm-wo.seahaven.com`) — implemented
and `cdk synth`-green; PR #10 open (stacked on Phase 3). App manifest in
`slack/manifest.yaml`.
- **Phase 5** self-hosted Grafana (EC2 + ALB on `grafana.seahaven.com`,
office-IP-restricted, 7-panel dashboard as code) — implemented and
`cdk synth`-green; PR #11 open (stacked on Phase 4).
- **Phase 6** (final docs / Confluence / runbook) — not started.
All phase PRs are stacked (#6→#7→#8→#9→#10→#11) and **unmerged**; nothing is
deployed yet. Cross-review and `/security-review` are outstanding across the stack.

187
docs/RUNBOOK.md Normal file
View file

@ -0,0 +1,187 @@
# apm-wo-analysis — Operational Runbook
Operational procedures and incident response for the daily APM work-order
analysis pipeline. Account **328440206208** / **us-east-1**. Stacks
`apm-wo-analysis-pipeline` and `apm-wo-analysis-grafana`. See [`README.md`](../README.md)
for architecture and resource detail.
**Admin access:** the Grafana box is **SSM Session Manager only** (no SSH/key pair):
`aws ssm start-session --target <instance-id>`. AWS API via the `office_mac` /
deploy roles. Grafana UI is office-IP-restricted at the ALB.
---
## 1. Incident Runbook: Missing daily APM WO analysis
**Severity: High** — a day's WO analysis is missing: escalations (incl. 3rd-escalation
alerts) aren't surfaced and the dashboard has no new snapshot. Recoverable by
re-processing the export; not permanent data loss.
### Detection
- No **daily summary** in the WO Slack channel by the usual time (or no 3rd-escalation alert on a day one's expected).
- **Grafana** shows no `dt = today` in the Snapshot Date dropdown / panels empty for today.
- **Messages in `apm-wo-analysis-classifier-dlq`** (SQS) — strongest signal the classifier failed.
- CloudWatch errors in `/aws/lambda/apm-wo-analysis-classifier` or `-slack-post`.
### Context
| Item | Value |
|---|---|
| Stacks | `apm-wo-analysis-pipeline`, `apm-wo-analysis-grafana` |
| Lambdas | `apm-wo-analysis-classifier`, `-slack-post`, `-slack-interactions` |
| S3 | `apm-wo-analysis-exports-328440206208` — `raw/`, `analytics/dt=…/`, `meta/dt=…/` |
| SQS DLQ | `apm-wo-analysis-classifier-dlq` |
| Glue / Athena | db `apm_wo_analysis`, table `apm_wo_snapshots`, workgroup `apm-wo-analysis` |
| External | Slack, Anthropic API (Haiku fallback) |
| Secrets (names) | `apm-wo-analysis/slack-credentials`, `apm-wo-analysis/anthropic-api-key` |
| SSM | `/apm-wo-analysis/grafana-base-url` |
### Triage
1. **Export uploaded?** `aws s3 ls s3://apm-wo-analysis-exports-328440206208/raw/` — today's file present? Absent → upstream (§2.1), not the pipeline.
2. **Classifier ran/failed?** `/aws/lambda/apm-wo-analysis-classifier` logs; peek the DLQ: `aws sqs receive-message --queue-url <dlq-url> --max-number-of-messages 1`.
3. **Outputs written?** `aws s3 ls .../analytics/dt=<today>/` (Parquet) and `.../meta/dt=<today>/` (`summary.json`, `details.json`).
4. **slack-post ran/failed?** `/aws/lambda/apm-wo-analysis-slack-post` logs — `SlackApiError` (`invalid_auth`, `not_in_channel`, `invalid_blocks`)?
5. **Slack creds** valid + bot still in channel? (`apm-wo-analysis/slack-credentials`).
6. **IAM/throttle:** grep logs for `AccessDenied` / throttling.
7. **Recent change?** Any merge/deploy to `main` just before the failure.
### Resolution (by root cause)
1. **Export not uploaded** → `aws s3 cp <export>.xlsx s3://apm-wo-analysis-exports-328440206208/raw/`; then check the drop-folder agent (§2.1).
2. **Classifier failed (DLQ)** → read the DLQ message; fix; **reprocess by re-uploading the export to `raw/`** (`overwrite_partitions` makes same-`dt` idempotent).
3. **Outputs present, no Slack post** → re-invoke:
```bash
aws lambda invoke --function-name apm-wo-analysis-slack-post \
--payload "$(printf '{"dt":"<YYYY-MM-DD>"}' | base64)" /tmp/out.json
```
If Slack auth was the cause → rotate `apm-wo-analysis/slack-credentials` (and/or `/invite` the bot), then re-invoke (secret read per-call; no redeploy).
4. **Code regression** → identify the PR, revert/hotfix, redeploy via CI (push to `main`).
5. **Data present, Grafana empty** → datasource/dashboard issue (§2.3); check the instance via SSM.
6. **Anthropic/Haiku down** → non-fatal (deterministic path still classifies ~95%); set classifier env `APM_HAIKU_FALLBACK=off` to bypass.
### Post-Incident
- Verify re-process → summary posts + Grafana shows today's `dt`.
- Check for **other missed days** (gaps in `analytics/dt=…`/`meta/`) and reprocess each.
- Update README/Confluence if knowledge changed; add a memory entry; add a test if code caused it.
---
## 2. Operational Procedures
### 2.1 How the export gets uploaded
The curated daily APM filter-view export (~350 WOs, `.xlsx`/`.csv`) reaches S3 by
**direct upload or a local drop-folder** — never SES/email. Any object under
`raw/` with a `.xlsx`/`.csv` suffix triggers the classifier.
- **Direct:** `aws s3 cp ./export.xlsx s3://apm-wo-analysis-exports-328440206208/raw/`
- **Drop-folder (zero-touch):** a launchd agent (`com.seahaven.apm-wo-uploader`)
watches `~/apm-wo-drop/`, uploads new files to `raw/` using the scoped
`apm-wo-drop` profile (IAM user **`apm-wo-drop-uploader`** — `s3:PutObject` on
`raw/*` only), and archives them locally. The runnable script lives at
`~/.local/bin/apm-wo-uploader.sh` (it and the watched folder **must** be outside
`~/Documents` — macOS TCC sandbox).
**Verify the agent:**
```bash
launchctl list | grep apm-wo-uploader # present + last exit 0
tail -f ~/apm-wo-drop/uploader.log # per-run logging (TBD: confirm log path)
```
**Common upstream issues:** agent unloaded (`launchctl load -w …plist`); script
moved back into `~/Documents` (TCC blocks it — `LastExitStatus=32256`); `apm-wo-drop`
access key expired/rotated (`aws configure --profile apm-wo-drop`).
### 2.2 Grafana OS / app patching cadence
The Grafana EC2 box (`t4g.small`, **Amazon Linux 2023**, ARM64) is the only
patch-bearing piece — everything else is serverless. It's **reproducible from
`cdk/assets/grafana_userdata.sh`**, so the preferred patch path is a **clean
instance replacement** rather than long-lived in-place drift.
- **OS (recommended monthly + on critical CVEs):** via SSM —
`sudo dnf upgrade --security -y && sudo reboot` (Session Manager, or an SSM
Run Command / Patch Manager maintenance window — **TBD: not yet automated**).
- **Grafana OSS:** `sudo dnf upgrade grafana -y && sudo systemctl restart grafana-server`
(installed from the pinned `rpm.grafana.com` repo).
- **Athena datasource plugin:** pinned to **`3.2.0`** in `cdk/cdk.json`
(`athenaPluginVersion`). Bump there, then redeploy/replace the instance.
- **Preferred = clean replacement:** terminate the instance; `cdk deploy
apm-wo-analysis-grafana` relaunches it from the latest AL2023 AMI and re-runs
user-data (fresh Grafana + plugin + config sync). The root volume is
`DeleteOnTermination=false`, so detach/reuse or restore `grafana.db` (§2.4) if
local settings must persist. Validates the committed user-data from a cold boot.
> **Outstanding:** one clean instance replacement is owed to validate cold-boot
> user-data and apply root-volume encryption (encryption can't be added in place).
### 2.3 Dashboard-JSON redeploy
Source of truth is **`grafana/dashboards/apm-work-orders.json`** in this repo
(uid **`apm-wo`**); the running instance is never the source of truth
(`allowUiUpdates: false` — UI edits are reverted on the next sync).
**Flow:** edit JSON in repo → `cdk deploy apm-wo-analysis-grafana` (the
`BucketDeployment` uploads `grafana/` to `s3://…/grafana-config/`) → the instance
syncs S3 → `/var/lib/grafana/dashboards/` (on boot + a **15-min systemd timer**)
→ Grafana's file provider polls every **60 s** and reloads.
**Apply immediately** (skip the timer) via SSM:
```bash
sudo /usr/local/bin/grafana-config-sync.sh # pulls grafana-config/ from S3
# Grafana file provider picks up the dashboard within ~60s
```
**Gotchas:**
- The sync uses `aws s3 sync --exact-timestamps` — required so **same-size edits**
(e.g. a one-char query change) actually propagate.
- **Datasource/provisioning** changes (`grafana/provisioning/*.yaml`) are loaded
at **startup** — after syncing, `sudo systemctl restart grafana-server` (a
dashboard-only change does **not** need a restart).
- Athena query key is **`rawSQL`** (capital); datasource `authType: default`;
template vars `refresh: 1`. (See README "Notes / Gotchas".)
### 2.4 Backup & restore (config + EBS / grafana.db)
Two distinct layers:
**Config (dashboards, datasources, provisioning)** — fully **reproducible from
git** (`grafana/` → S3 `grafana-config/`). *Restore:* `cdk deploy
apm-wo-analysis-grafana` (or `grafana-config-sync.sh` on the box). No snapshot needed.
**Local state (`/var/lib/grafana/grafana.db`)** — Grafana's SQLite (admin user,
any API keys, org prefs). Lives on the **gp3 root volume** (encrypted,
`DeleteOnTermination=false`). Backed up by a **daily DLM snapshot** (07:00 UTC,
7 retained) of the instance (tag `apm-grafana-backup=true`), policy in the
grafana stack.
*Restore from snapshot:*
```bash
# find the latest DLM snapshot
aws ec2 describe-snapshots --owner-ids self \
--filters "Name=tag:aws:dlm:lifecycle-policy-id,Values=*" \
--query 'reverse(sort_by(Snapshots,&StartTime))[0].SnapshotId' --output text
# create a volume from it and attach to a replacement instance, OR mount it and
# copy /var/lib/grafana/grafana.db onto the new instance, then:
sudo systemctl restart grafana-server
```
Because dashboards + datasource are provisioned from code, the only thing the
snapshot uniquely protects is `grafana.db` (admin/login state) — low stakes; a
fresh instance + provisioning recovers everything else.
> **Note:** the Grafana **admin auth model is unsettled (TBD)** — the password was
> reset ad-hoc during build/testing. Decide the intended model (fixed admin
> password in Secrets Manager / SSO / anonymous view-only for the kiosk) and
> document it here.
### 2.5 Common failures (quick index)
| Symptom | Likely cause | Go to |
|---|---|---|
| No daily Slack post / no new dashboard day | export not uploaded, classifier failed (DLQ), or slack-post failed | §1 |
| Dashboard loads but all panels "No data" | datasource/auth, `rawSQL`, template `refresh`, or JSON in `analytics/` prefix | §2.3, README gotchas |
| Grafana unreachable | instance down / ALB unhealthy / office IP changed (`officeCidrs`) | SSM triage; `aws elbv2 describe-target-health` |
| Slack modal click does nothing / error | `apm-wo-analysis-slack-interactions`, API Gateway, or signing-secret mismatch | `/aws/lambda/apm-wo-analysis-slack-interactions` logs |
| Exports never arrive in `raw/` | drop-folder agent unloaded / TCC / expired key | §2.1 |
| Deploy not applying | OIDC role, CloudFormation rollback, Docker bundling | CloudFormation events; CI logs |
---
*Maintained in-repo (`docs/RUNBOOK.md`) and mirrored to Confluence. Update both
when operational knowledge changes.*