adam/apm-wo-analysis
1
0
Fork 0
mirror of https://github.com/Sea-Haven-Industries/apm-wo-analysis.git synced 2026-09-30 07:43:15 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Refresh README to full operational doc (Phase 6)

Browse source 
Rewrite to the Sea Haven operational template with real resource names from both
stacks: AWS Resources + Lambda Functions tables, Configuration (Secrets/SSM/env/
context), Operations (verify, logs, classifier DLQ, reprocess, Grafana admin),
Documentation, and Notes/Gotchas incl. the deploy-time lessons + a known-debt
list. Corrects stale bits: ALB is internet-facing + office-IP-restricted (not
internal); classifier uses partition projection (no runtime Glue registration);
summary/details JSON live under meta/ not analytics/; grafana uses an instance
role. Adds the resources missing from the old README (classifier DLQ, slack-
interactions Lambda, HTTP API, meta/ + grafana-config/ prefixes, DLM backup,
encrypted volume).
This commit is contained in:
Adam Moussa 2026-05-29 12:27:05 -04:00
parent eae4d67e00
commit 60e0b878e3
1 changed files with 180 additions and 90 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

270
README.md
View file

@ -2,8 +2,10 @@
Daily analysis of Amazon **APM work-order** "Last Comment" data for Sea Haven Daily analysis of Amazon **APM work-order** "Last Comment" data for Sea Haven
facility ops. A curated daily filter-view export (~350 work orders) is classified facility ops. A curated daily filter-view export (~350 work orders) is classified
on two axes, pushed to Slack, and surfaced in a self-hosted Grafana dashboard. on two axes (comment intent + structured `WO Status`/`Hold Reason`), pushed to
Replaces a legacy Google Apps Script + versioned-Google-Sheet workflow. Slack as a daily summary plus a batched 3rd-escalation alert, and surfaced in a
self-hosted Grafana dashboard. Replaces a legacy Google Apps Script +
versioned-Google-Sheet workflow.
This is a **sibling concern** to the `apm@` email pipeline in `procurement-ingest` This is a **sibling concern** to the `apm@` email pipeline in `procurement-ingest`
— it consumes a different feed (the manual export) and does **not** read those — it consumes a different feed (the manual export) and does **not** read those
@ -17,79 +19,123 @@ backed by S3 + Athena (Grafana cannot query DynamoDB).
``` ```
APM export (xlsx/csv) APM export (xlsx/csv)
→ S3 raw/ (direct upload OR local launchd drop-folder) → S3 raw/ (direct `aws s3 cp` OR local launchd drop-folder)
→ classifier Lambda (HTML strip + two-axis classify, Haiku fallback) → classifier Lambda (HTML strip + two-axis classify, Haiku fallback)
→ S3 analytics/dt=YYYY-MM-DD/ (per-WO daily snapshot, Parquet) ├→ S3 analytics/dt=YYYY-MM-DD/ (per-WO snapshot, Parquet)
→ Glue table → Athena → Grafana (self-hosted EC2, VPN-only, kiosk) │ → Glue table (partition projection) → Athena → Grafana (EC2, office-IP, kiosk)
→ slack-post Lambda (reads today + yesterday partitions) ├→ S3 meta/dt=YYYY-MM-DD/ (summary.json + details.json — NOT in the table prefix)
└→ async-invoke slack-post Lambda
→ daily summary post [📊 Open dashboard button] → daily summary post [📊 Open dashboard button]
→ standalone batched 3rd-escalation alert (suppressed if zero) → standalone batched 3rd-escalation alert (suppressed when zero)
→ drill-down modals via apm-wo.seahaven.com (signature-verified)
``` ```
Two CDK stacks: Two CDK stacks (`cdk/app.py` instantiates both):
| Stack | Resources | | Stack | Resources |
|---|---| |---|---|
| `apm-wo-analysis-pipeline` | S3 exports bucket, classifier + slack-post Lambdas, Glue database, Athena workgroup, IAM | | `apm-wo-analysis-pipeline` | S3 exports bucket, drop-uploader IAM user, classifier + slack-post + slack-interactions Lambdas, classifier DLQ, Glue DB + projection table, Athena workgroup, HTTP API (`apm-wo.seahaven.com`), SSM param, scoped IAM |
| `apm-wo-analysis-grafana` | EC2 (Grafana OSS), internal ALB, security group, Route53, Athena datasource IAM role | | `apm-wo-analysis-grafana` | EC2 (Grafana OSS), internet-facing **office-IP-restricted** ALB (`grafana.seahaven.com`), security groups, instance IAM role, Route53 alias, daily DLM snapshot, dashboards-as-code S3 deployment |
## AWS Resources
| Resource | Name | Purpose |
|---|---|---|
| S3 bucket | `apm-wo-analysis-exports-328440206208` | Single bucket. Prefixes: `raw/` (incoming, 90-day expiry), `analytics/` (Parquet snapshots, kept), `meta/` (summary/details JSON), `athena-results/` (query output, 30-day expiry), `grafana-config/` (dashboards-as-code). SSE-S3, BPA-all, enforce-SSL, `RETAIN`. |
| IAM user | `apm-wo-drop-uploader` | Drop-folder identity; `s3:PutObject` on `raw/*` only. Access key created out-of-band, stored in local `apm-wo-drop` profile. |
| Glue database | `apm_wo_analysis` | Analytics catalog. |
| Glue table | `apm_wo_snapshots` | External Parquet table over `analytics/`, **partition projection** on `dt` (date, `2026-01-01..NOW`) — no crawler, no `MSCK`. 17-column snapshot schema. |
| Athena workgroup | `apm-wo-analysis` | Enforced result location `athena-results/`, SSE-S3. |
| SQS queue | `apm-wo-analysis-classifier-dlq` | Dead-letter for failed classifier async invocations (14-day retention). |
| SSM parameter | `/apm-wo-analysis/grafana-base-url` | Grafana dashboard URL for the 📊 button / modal links (ops-editable, no redeploy). |
| HTTP API + domain | `apm-wo.seahaven.com` → `POST /slack/interactions` | Slack interactivity endpoint. Stage throttled 10 rps / 20 burst. `*.seahaven.com` ACM cert; Route53 alias. |
| EC2 instance | Grafana (`t4g.small`, AL2023, ARM64) | Self-hosted Grafana OSS in `seahaven-vpc` private subnets, IMDSv2-only, SSM-managed. gp3 20 GB **encrypted**, `DeleteOnTermination=false`, tagged `apm-grafana-backup`. |
| ALB | Grafana ALB (`grafana.seahaven.com`) | Internet-facing, HTTPS 443, SG admits **only office CIDRs** (`47.21.61.4/32`, `96.250.164.146/32`); forwards to instance:3000, health `/api/health`. |
| DLM policy | Grafana volume backup | Daily snapshot (07:00 UTC) of the tagged instance, 7 retained. |
| Route53 | `apm-wo.seahaven.com`, `grafana.seahaven.com` | Aliases in zone `Z06652411XKH89KTZD3XA` (`seahaven.com`). |
## Lambda Functions
All Python 3.12, ARM64, explicit LogGroup (`/aws/lambda/<name>`, 60-day retention).
| Function | Trigger | Purpose |
|---|---|---|
| `apm-wo-analysis-classifier` | S3 `ObjectCreated` on `raw/*.xlsx|.csv` | Parse export, two-axis classify each non-blank-comment row, write per-WO Parquet to `analytics/dt=…/` and `summary.json`/`details.json` to `meta/dt=…/`, then async-invoke slack-post. 512 MB / 120 s. AWS-managed SDK-for-pandas layer (`AWSSDKPandas-Python312-Arm64:27`); DLQ attached. |
| `apm-wo-analysis-slack-post` | Async-invoked by the classifier (`{"dt": …}`) | Read today's + yesterday's `meta/.../summary.json`, post the daily summary, and (only when `third_escalation_count > 0`) the batched 3rd-escalation alert from `details.json`. 256 MB / 30 s. |
| `apm-wo-analysis-slack-interactions` | HTTP API `POST /slack/interactions` | Verify the Slack request signature, read `meta/.../details.json`, and `views.open` a filtered WO-list modal within Slack's 3 s `trigger_id` window. 256 MB / 30 s. |
## Configuration
### Secrets Manager (names only — created out-of-band, never in CloudFormation)
| Secret | Purpose |
|---|---|
| `apm-wo-analysis/anthropic-api-key` | Claude Haiku fallback for ambiguous free-text comments. |
| `apm-wo-analysis/slack-credentials` | JSON `{ botToken, signingSecret, channelId }` for the reused Slack app. |
### SSM Parameters
| Parameter | Purpose |
|---|---|
| `/apm-wo-analysis/grafana-base-url` | Grafana dashboard deep-link base (`https://grafana.seahaven.com/d/apm-wo/...`). |
### Environment Variables (non-secret)
- **classifier:** `APM_HAIKU_FALLBACK` (`on`/`off`), `SLACK_POST_FUNCTION_NAME`.
- **slack-post / slack-interactions:** `SLACK_SECRET_NAME`, `DASHBOARD_URL_PARAM`, `ANALYTICS_BUCKET`.
### GitHub repo secret
- `AWS_DEPLOY_ROLE_ARN` — the OIDC deploy role `githubdeploy-apm-wo-analysis`.
### CDK context (`cdk/cdk.json`)
`wildcardCertArn`, `hostedZoneId`/`hostedZoneName`, `slackInteractionsDomain`, `grafanaDomain`, `grafanaVpcId`/`grafanaAzs`/`grafana{Public,Private}SubnetIds`, `officeCidrs`, `athenaPluginVersion` (`3.2.0`).
## The classification model ## The classification model
**Always two-axis, never comment-only.** The legacy script's central flaw was **Always two-axis, never comment-only.** The legacy script's central flaw was
reading only the comment while ignoring `WO Status` + `Hold Reason`, which left reading only the comment while ignoring `WO Status` + `Hold Reason`, which left
~17% in "Other". The two-axis model cuts that to ~9% before any AI — and on the ~17% in "Other". The two-axis model cuts that to ~9% before any AI — and on the
real 347-row export the current implementation lands "Other" at **5.2%** (18 real 347-row export the implementation lands "Other" at **5.2%** (18 rows) with
rows) with **18 mismatches** flagged. **18 mismatches** flagged.
- **Axis 1 — comment intent:** regex over the HTML-stripped `Last Comment`, - **Axis 1 — comment intent:** regex over the HTML-stripped `Last Comment`,
most-specific first (escalations → SIM ticket → vendor no-show → scheduling → most-specific first (escalations → SIM ticket → vendor no-show → scheduling →
reports → completion → … → other). reports → completion → … → other).
- **Axis 2 — structured state:** `Hold Reason` → category and `WO Status` - **Axis 2 — structured state:** `Hold Reason` → category, and `WO Status`
signals (`RCAN`→Cancelled, `H` corroborates On Hold, `IP`/`R`/`RR` in-flight). signals (`RCAN`→Cancelled, `H` corroborates On Hold, `IP`/`R`/`RR` in-flight).
- **Resolution:** comment intent wins when confident → else structured state → - **Resolution:** comment intent wins when confident → else structured state →
else `Other`. A **Claude Haiku** fallback (Secrets Manager) is reserved for else `Other`. A **Claude Haiku** fallback (Secrets Manager key) is reserved for
ambiguous free-text with no structured signal. ambiguous free-text with no structured signal.
- **Mismatch detector (a feature):** flags when comment intent contradicts - **Mismatch detector (a feature):** flags when comment intent contradicts
structured state. Surfaced, never suppressed. structured state (e.g. "completed" while `WO Status` is `IP`). Surfaced, never
suppressed.
The authoritative spec lives in [`CLAUDE.md`](./CLAUDE.md); the implementation is Authoritative spec: [`CLAUDE.md`](./CLAUDE.md). Implementation:
in `lambdas/classifier/classify.py` (owned by the `classifier-engineer` agent). `lambdas/classifier/classify.py` (logic) and `lambdas/classifier/handler.py`
The S3-triggered `lambdas/classifier/handler.py` parses each export, classifies (S3 → Parquet + JSON + invoke).
every non-blank-comment row, writes a per-WO Parquet snapshot to
`analytics/dt=YYYY-MM-DD/` (registering the Glue partition via awswrangler), and
emits a `summary.json` for the Phase 4 slack-post Lambda.
## Repository layout ## Repository layout
``` ```
cdk/ cdk/
app.py CDK entry point — instantiates both stacks app.py CDK entry point — instantiates both stacks
cdk.json cdk.json context: cert, zone, subnets, office CIDRs, plugin version
requirements.txt aws-cdk-lib==2.253.1, constructs>=10.6.0 requirements.txt aws-cdk-lib==2.253.1, constructs>=10.6.0
assets/
grafana_userdata.sh EC2 bootstrap: install Grafana + Athena plugin, S3 config sync
stacks/ stacks/
pipeline_stack.py S3, Lambdas, Glue, Athena, IAM pipeline_stack.py S3, Lambdas, DLQ, Glue, Athena, HTTP API, IAM
grafana_stack.py VPC import, EC2, ALB, SG, Route53, datasource role grafana_stack.py VPC import, EC2, ALB, SG, Route53, instance role, DLM
lambdas/ lambdas/
classifier/ S3-triggered: parse → two-axis classify → Parquet classifier/ S3-triggered: parse → two-axis classify → Parquet + meta JSON
slack_post/ builds + posts the daily summary and alert slack_post/ blockkit.py (builders), handler.py (post), interactions.py
(modals), slackio.py (Secrets/SSM/S3/signature)
grafana/ grafana/
provisioning/ Athena datasource + dashboard provider (as code) provisioning/ Athena datasource + dashboard provider (as code)
dashboards/ committed dashboard JSON (source of truth) dashboards/ apm-work-orders.json (uid apm-wo) — source of truth
slack/manifest.yaml Slack app manifest (interactivity request URL)
scripts/ local drop-folder uploader + launchd plist scripts/ local drop-folder uploader + launchd plist
tests/ classifier smoke test tests/ classifier smoke test + offline synth/blockkit assertions
docs/BUILD.md phased, end-to-end build guide docs/BUILD.md phased, end-to-end build guide
``` ```
## Configuration
| Where | What |
|---|---|
| **Secrets Manager** | `apm-wo-analysis/anthropic-api-key` (Haiku fallback); `apm-wo-analysis/slack-credentials` = `{ botToken, signingSecret, channelId }` (reused Slack app). |
| **SSM Parameter Store** | `/apm-wo-analysis/grafana-base-url` (the 📊 dashboard button / modal overflow link; ops-editable). |
| **GitHub repo secret** | `AWS_DEPLOY_ROLE_ARN` — the OIDC deploy role `githubdeploy-apm-wo-analysis`. |
No secrets in Lambda environment variables.
## Ingestion (no email) ## Ingestion (no email)
The export reaches S3 by **direct upload or a local drop-folder**, never SES/email. The export reaches S3 by **direct upload or a local drop-folder**, never SES/email.
@ -97,72 +143,116 @@ The export reaches S3 by **direct upload or a local drop-folder**, never SES/ema
- **Direct:** `aws s3 cp ./export.xlsx s3://apm-wo-analysis-exports-328440206208/raw/` - **Direct:** `aws s3 cp ./export.xlsx s3://apm-wo-analysis-exports-328440206208/raw/`
- **Drop-folder (optional zero-touch):** a launchd agent (`scripts/apm-wo-uploader.sh` - **Drop-folder (optional zero-touch):** a launchd agent (`scripts/apm-wo-uploader.sh`
+ `scripts/com.seahaven.apm-wo-uploader.plist`) that watches `~/apm-wo-drop/`, + `scripts/com.seahaven.apm-wo-uploader.plist`) that watches `~/apm-wo-drop/`,
uploads new `.xlsx`/`.csv` files to `raw/`, and archives them to `uploaded/`. uploads new `.xlsx`/`.csv` files to `raw/`, and archives them locally. Uploads
It uploads with the scoped `apm-wo-drop` AWS profile (IAM user with the scoped `apm-wo-drop` profile (IAM user `apm-wo-drop-uploader`).
`apm-wo-drop-uploader` — `s3:PutObject` on `raw/*` only).
Install (the runnable copy **must** live outside `~/Documents` — macOS TCC Install — the runnable copy and watched folder **must** live outside `~/Documents`
sandbox; a repo-path script fails silently with `LastExitStatus=32256`): (macOS TCC sandbox; a repo-path script fails silently with `LastExitStatus=32256`):
```bash ```bash
install -d "$HOME/.local/bin" "$HOME/apm-wo-drop" install -d "$HOME/.local/bin" "$HOME/apm-wo-drop"
cp scripts/apm-wo-uploader.sh "$HOME/.local/bin/apm-wo-uploader.sh" cp scripts/apm-wo-uploader.sh "$HOME/.local/bin/apm-wo-uploader.sh"
chmod +x "$HOME/.local/bin/apm-wo-uploader.sh" chmod +x "$HOME/.local/bin/apm-wo-uploader.sh"
cp scripts/com.seahaven.apm-wo-uploader.plist "$HOME/Library/LaunchAgents/" cp scripts/com.seahaven.apm-wo-uploader.plist "$HOME/Library/LaunchAgents/"
launchctl load -w "$HOME/Library/LaunchAgents/com.seahaven.apm-wo-uploader.plist" launchctl load -w "$HOME/Library/LaunchAgents/com.seahaven.apm-wo-uploader.plist"
aws configure --profile apm-wo-drop # one-time, with the uploader's access key
``` ```
Re-copy the script to `~/.local/bin` after editing the repo source. Configure
the profile once with the uploader's access key:
`aws configure --profile apm-wo-drop`.
The classifier Lambda is S3-triggered on the `raw/` prefix regardless of path Any `.xlsx`/`.csv` landing under `raw/` invokes the classifier.
(any `.xlsx`/`.csv` landing under `raw/` invokes it).
## Deployment
CI/CD via the org reusable workflows (no manual prod deploys):
- **CI** (`.github/workflows/ci.yaml`) → `ci-python-sam.yaml@main`: ruff + `cdk synth`.
- **Deploy** (`.github/workflows/deploy.yaml`) → `cd-cdk.yaml@main`: OIDC assume-role,
`cdk deploy --all`, single-flight concurrency.
The OIDC deploy role must exist **before** the first deploy. Deploy order:
```bash
cd cdk && pip install -r requirements.txt
cdk deploy apm-wo-analysis-pipeline # S3, Glue, Athena, Lambdas, IAM
cdk deploy apm-wo-analysis-grafana # EC2, ALB, SG, Route53, datasource role
```
## Local development ## Local development
- pyenv Python 3.12; `ruff check` + `ruff format --check` before pushing (hook-enforced). - pyenv Python 3.12. `ruff check` + `ruff format --check` before pushing (hook-enforced).
- Tests: `python -m pytest tests/ -q` (60 tests — classifier smoke test against a real
export + offline `cdk.assertions` synth checks + Block Kit builders). No AWS needed.
- Smoke-test the classifier against a **real export** before declaring any - Smoke-test the classifier against a **real export** before declaring any
classification change done: `~/Downloads/_documents/Sheet1-1.xlsx`. classification change done: `~/Downloads/_documents/Sheet1-1.xlsx`.
- `cdk synth` must pass in CI before merge. - `cdk synth` must pass in CI before merge (**Docker required** — Lambda deps are
bundled for ARM64).
## Deployment
CI/CD via the org reusable workflows (no manual prod deploys in steady state):
- **CI** (`.github/workflows/ci.yaml`) → `ci-python-sam.yaml@main`: ruff + `cdk synth`. Runs on PRs into `main`.
- **Deploy** (`.github/workflows/deploy.yaml`) → `cd-cdk.yaml@main`: OIDC assume-role, `cdk deploy --all`, single-flight concurrency. Runs on push to `main`.
Stack name/region/account: `apm-wo-analysis-{pipeline,grafana}` / us-east-1 / 328440206208.
OIDC deploy role `githubdeploy-apm-wo-analysis` must exist before the first deploy.
> **Note:** CD is **temporarily disabled** (deploy job gated `if: ${{ false }}` on
> the phase-0 branch) while the Phase 0–5 stack is merged into `main`, to avoid a
> deploy on every merge. **Re-enable as the first Phase 6 step** by reverting that
> commit. Both stacks are already deployed manually and validated in prod.
Manual deploy (emergency/reference; pipeline first so the bucket/table exist):
```bash
cd cdk && pip install -r requirements.txt
cdk deploy apm-wo-analysis-pipeline # S3, Glue, Athena, Lambdas, DLQ, API, IAM
cdk deploy apm-wo-analysis-grafana # EC2, ALB, SG, Route53, DLM, dashboards
```
## Operations
**Verify it's working** — drop a real export into `raw/`, then within ~1 min:
- `analytics/dt=<today>/*.parquet` and `meta/dt=<today>/{summary,details}.json` appear,
- the daily summary posts to the WO Slack channel (+ alert if any 3rd escalations),
- the Grafana dashboard renders over the office network (`grafana.seahaven.com`).
**Logs:** CloudWatch `/aws/lambda/apm-wo-analysis-{classifier,slack-post,slack-interactions}` (60-day retention).
**Failure modes:**
- Classifier failure (malformed export, transient error) → after Lambda retries, the event lands in **`apm-wo-analysis-classifier-dlq`**. Check the DLQ if a day's data is missing.
- Slack post failure is best-effort and does **not** fail classification (data still lands in S3).
- Grafana down → check the instance via **SSM Session Manager** (no SSH); `systemctl status grafana-server`; ALB target health.
**Reprocess a day:** re-upload the same export to `raw/` — the classifier uses
`overwrite_partitions`, so a same-day re-run replaces that `dt` partition idempotently.
**Grafana admin:** access is office-IP-restricted at the ALB; the instance is
SSM-only. Dashboards are provisioned from `grafana-config/` in S3 (synced on boot
and by a 15-min systemd timer); **edit dashboards in-repo, not in the UI**
(`allowUiUpdates: false`). `grafana.db` lives on the RETAIN'd gp3 volume and is
snapshotted daily by DLM.
## Documentation
- **Confluence "AWS Architecture Map"** (IT space, page **1540098**): a Mermaid
subgraph for this stack — **TBD** (Phase 6; previously blocked on the Atlassian
MCP, now unblocked).
- **Slack Apps Inventory** (Confluence page 524569): add the "APM Work Orders" app — **TBD**.
- This README + `docs/BUILD.md` (phased build guide) + `CLAUDE.md` (domain spec).
## Notes / Gotchas
- **Partition date** comes from the **S3 event time**, not the Lambda wall-clock —
stable across retries and the midnight boundary.
- **`meta/` vs `analytics/`:** summary/details JSON must stay **out** of `analytics/` —
Athena reads every object in the table prefix as Parquet and chokes on JSON.
- **Classifier deps** (awswrangler/pandas/pyarrow/numpy) come from the AWS-managed
SDK-for-pandas **layer** — bundling them blows Lambda's 250 MB unzipped limit.
Verify the pinned layer ARN/version on region or runtime changes.
- **Grafana dashboard contract:** dashboard `uid` must stay `apm-wo` (the Slack 📊
button deep-links to `d/apm-wo`). Athena plugin query key is `rawSQL` (capital).
Datasource `authType: default` (the EC2 instance role; `ec2_iam_role` is rejected
by the plugin). Config sync uses `aws s3 sync --exact-timestamps` (plain sync
skips same-size edits). Template vars use `refresh: 1` (on load).
- **No Client VPN exists** — "VPN-only" Grafana is realized as **office-IP SG
restriction**. `seahaven-vpc` has a single NAT (one AZ) for instance egress.
- **Slack interactions endpoint** is unauthenticated at the gateway **by design**;
the Lambda verifies the Slack signature (replay window + HMAC). Stage-throttled.
### Known operational debt
- Re-enable CD (revert the phase-0 disable) once the stack is merged.
- One **clean instance replacement** is owed to validate the committed user-data
from a cold boot and to apply root-volume encryption (can't encrypt in place).
- Deferred review NITs: `print()`→`logging`, source-IP logging on signature
failure, S3 versioning, the `'${site:raw}'` WO-table SQL tidy, and a
CIDR-maintenance note in the runbook.
## Status ## Status
**Phase 5 — Grafana (in review); build complete, docs remain.** Build-out per Phases 2–5 implemented, **deployed to prod and validated end-to-end** (classifier,
[`docs/BUILD.md`](./docs/BUILD.md): ingestion → classifier → Glue/Athena → Slack Slack post + alert + modal, Grafana dashboard). Stacked PRs **#6→#11** are open and
→ Grafana → docs. unmerged; cross-review (#2/#4/#5) and `/security-review` of the two public endpoints
are **cleared**. Phase 6 (this docs pass + Confluence + runbook) is in progress on
- **Phase 0** scaffold — merged-pending (PR #6). `feature/phase-6-docs`.
- **Phase 1** ingestion (S3 bucket, drop-folder uploader, OIDC deploy role) —
deployed; PR #7 open.
- **Phase 2** classifier Lambda + Glue database + S3 `raw/` trigger — implemented
and `cdk synth`-green; PR #8 open (stacked on Phase 1, not yet deployed).
- **Phase 3** `apm_wo_snapshots` projection table + Athena workgroup —
implemented and `cdk synth`-green; PR #9 open (stacked on Phase 2). The
classifier writes pure Parquet and holds no Glue access (projection handles
partitions).
- **Phase 4** Slack post + interactions Lambdas (daily summary, batched
3rd-escalation alert, drill-down modals on `apm-wo.seahaven.com`) — implemented
and `cdk synth`-green; PR #10 open (stacked on Phase 3). App manifest in
`slack/manifest.yaml`.
- **Phase 5** self-hosted Grafana (EC2 + ALB on `grafana.seahaven.com`,
office-IP-restricted, 7-panel dashboard as code) — implemented and
`cdk synth`-green; PR #11 open (stacked on Phase 4).
- **Phase 6** (final docs / Confluence / runbook) — not started.
All phase PRs are stacked (#6→#7→#8→#9→#10→#11) and **unmerged**; nothing is
deployed yet. Cross-review and `/security-review` are outstanding across the stack.