feat(exec-aide): CloudWatch alarm coverage (Lambda + DynamoDB + ECS) #59

Closed

amoussa1229 wants to merge 2 commits from infra-cloudwatch-alarm-coverage into main

amoussa1229 commented

2026-06-17 18:00:19 +00:00

(Migrated from github.com)

Summary

Wave 1 PR H — CloudWatch alarm coverage for the exec-aide CDK (TypeScript) stack. Adds 14 ALARM-only alarms routed to the shared site-alerts SNS topic, mirroring the proposal-system alarm construct and the seahaven-door-unlock-api site-alerts import pattern.

The site-alerts topic is imported once at the stack level (sns.Topic.fromTopicArn) and injected into both constructs via props — the same way the DynamoDB table is wired into socket-mode. Every alarm: SnsAction → site-alerts, no OK / InsufficientData action, treatMissingData: NOT_BREACHING. Names are repo-namespaced kebab-case exec-aide-<resource>-<signal>.

Alarms added

Lambda (fetch-classify, daily-digest, conversation) — 9 alarms:

*-errors (Errors Sum ≥ 1, eval 1)
*-throttles (Throttles Sum ≥ 1, eval 1)
*-duration (p99, eval 3 / datapoints 2, ~80% of timeout)

DynamoDB exec-aide — 2 alarms:

exec-aide-table-throttles (ThrottledRequests)
exec-aide-table-system-errors (SystemErrors)

ECS exec-aide-listener Fargate service — 3 alarms:

exec-aide-listener-cpu-high (CPU Average > 80%, eval 3 / datapoints 2)
exec-aide-listener-memory-high (Memory Average > 80%, eval 3 / datapoints 2)
exec-aide-listener-no-running-tasks (RunningTaskCount Min < 1) — gated, see below

⚠️ NEEDS ADAM SIGN-OFF

Duration thresholds (p99, eval 3 / datapoints 2, ≈80% of timeout)

Function	Timeout	Threshold
`exec-aide-fetch-classify`	120 s	96000 ms
`exec-aide-daily-digest`	120 s	96000 ms
`exec-aide-conversation`	180 s	144000 ms

Container Insights cost/config (separate commit `25277f8`)

exec-aide-listener-no-running-tasks reads RunningTaskCount, which is only published in the ECS/ContainerInsights namespace (AWS/ECS does not emit it). That commit therefore enables Container Insights on the exec-aide cluster (containerInsightsV2: ENABLED) — a recurring CloudWatch ingestion/storage cost for the cluster. It is isolated to its own commit; the CPU/Memory alarms and all other coverage do not depend on it. Revert that single commit to drop both the cost and the RunningTaskCount alarm.

DynamoDB dimension findings

ThrottledRequests and SystemErrors are not published at the bare TableName dimension — AWS keys them by the Operation dimension. CDK confirms this: metricThrottledRequests / metricSystemErrors are deprecated as "invalid metric." Used the *ForOperations math helpers instead. Because an alarm math expression caps at 10 metrics and the default "all operations" set exceeds that (synth failed with TooManyMetricsInMathExpression), the alarms are scoped to the 6 operations this single-table app actually issues (GetItem, PutItem, Query, Scan, UpdateItem, DeleteItem — verified from src/).

Validation

npx tsc --noEmit — clean
npx cdk synth — clean; all 14 alarms render with correct thresholds, eval/datapoints, NOT_BREACHING, no OK/INS actions, and AlarmActions = [site-alerts]
Confirmed the listener FargateService logical id (SocketModeServiceB6F02D46) is unchanged after assigning it to a const
Pre-push security gate: PASS (0 crit/high)

Confluence map (id 1540098) delta — OUTSTANDING

The "AWS Architecture Map" exec-aide subgraph should gain a CloudWatch alarms node fanning the 3 Lambdas, the exec-aide table, and the exec-aide-listener service into the shared site-alerts SNS topic; annotate that exec-aide-listener-no-running-tasks + Container Insights are pending sign-off. Will apply once this PR's approach is confirmed.

Notes

No IAM policy/role changes and no Lambda handler-signature changes — the mandatory cross-family review gate does not apply. Per instructions: one PR, base main, not merged / not deployed.

## Summary Wave 1 PR H — CloudWatch alarm coverage for the `exec-aide` CDK (TypeScript) stack. Adds 14 ALARM-only alarms routed to the shared `site-alerts` SNS topic, mirroring the `proposal-system` alarm construct and the `seahaven-door-unlock-api` site-alerts import pattern. The `site-alerts` topic is imported **once** at the stack level (`sns.Topic.fromTopicArn`) and injected into both constructs via props — the same way the DynamoDB table is wired into `socket-mode`. Every alarm: `SnsAction` → `site-alerts`, **no OK / InsufficientData action**, `treatMissingData: NOT_BREACHING`. Names are repo-namespaced kebab-case `exec-aide-<resource>-<signal>`. ## Alarms added **Lambda** (fetch-classify, daily-digest, conversation) — 9 alarms: - `*-errors` (Errors Sum ≥ 1, eval 1) - `*-throttles` (Throttles Sum ≥ 1, eval 1) - `*-duration` (p99, eval 3 / datapoints 2, ~80% of timeout) **DynamoDB `exec-aide`** — 2 alarms: - `exec-aide-table-throttles` (ThrottledRequests) - `exec-aide-table-system-errors` (SystemErrors) **ECS `exec-aide-listener` Fargate service** — 3 alarms: - `exec-aide-listener-cpu-high` (CPU Average > 80%, eval 3 / datapoints 2) - `exec-aide-listener-memory-high` (Memory Average > 80%, eval 3 / datapoints 2) - `exec-aide-listener-no-running-tasks` (RunningTaskCount Min < 1) — **gated, see below** ## ⚠️ NEEDS ADAM SIGN-OFF ### Duration thresholds (p99, eval 3 / datapoints 2, ≈80% of timeout) | Function | Timeout | Threshold | |---|---|---| | `exec-aide-fetch-classify` | 120 s | **96000 ms** | | `exec-aide-daily-digest` | 120 s | **96000 ms** | | `exec-aide-conversation` | 180 s | **144000 ms** | ### Container Insights cost/config (separate commit `25277f8`) `exec-aide-listener-no-running-tasks` reads `RunningTaskCount`, which is **only** published in the `ECS/ContainerInsights` namespace (`AWS/ECS` does not emit it). That commit therefore enables **Container Insights on the `exec-aide` cluster** (`containerInsightsV2: ENABLED`) — a recurring CloudWatch ingestion/storage cost for the cluster. It is isolated to its own commit; the CPU/Memory alarms and all other coverage do **not** depend on it. Revert that single commit to drop both the cost and the RunningTaskCount alarm. ## DynamoDB dimension findings `ThrottledRequests` and `SystemErrors` are **not published at the bare `TableName` dimension** — AWS keys them by the `Operation` dimension. CDK confirms this: `metricThrottledRequests` / `metricSystemErrors` are deprecated as "invalid metric." Used the `*ForOperations` math helpers instead. Because an alarm math expression caps at **10 metrics** and the default "all operations" set exceeds that (synth failed with `TooManyMetricsInMathExpression`), the alarms are scoped to the **6 operations this single-table app actually issues** (GetItem, PutItem, Query, Scan, UpdateItem, DeleteItem — verified from `src/`). ## Validation - `npx tsc --noEmit` — clean - `npx cdk synth` — clean; all 14 alarms render with correct thresholds, eval/datapoints, `NOT_BREACHING`, no OK/INS actions, and `AlarmActions = [site-alerts]` - Confirmed the listener `FargateService` logical id (`SocketModeServiceB6F02D46`) is **unchanged** after assigning it to a `const` - Pre-push security gate: PASS (0 crit/high) ## Confluence map (id 1540098) delta — OUTSTANDING The "AWS Architecture Map" exec-aide subgraph should gain a CloudWatch alarms node fanning the 3 Lambdas, the `exec-aide` table, and the `exec-aide-listener` service into the shared `site-alerts` SNS topic; annotate that `exec-aide-listener-no-running-tasks` + Container Insights are pending sign-off. Will apply once this PR's approach is confirmed. ## Notes No IAM policy/role changes and no Lambda handler-signature changes — the mandatory cross-family review gate does not apply. Per instructions: one PR, base `main`, **not merged / not deployed**.

amoussa1229 commented

2026-06-17 18:18:36 +00:00

(Migrated from github.com)

Closing without merge: exec-aide is being deprecated soon, so the alarm suite added here (14 alarms + Container Insights) won't be maintained. Kept as reference on the branch if needed. Per the Wave 1 alarm-coverage rollout decision (2026-06-17).

This repo is archived. You cannot comment on pull requests.

No reviewers