Add CloudWatch alarm coverage (meal-order-manager) (#32)
Some checks failed
Deploy / deploy (push) Has been cancelled

* Add CloudWatch alarm coverage for the meal-order-manager stack

Add CloudWatch alarms (all notifying the shared site-alerts SNS topic,
no OKActions, TreatMissingData notBreaching) across the stack:

- Lambda Errors + Throttles alarms for all 7 functions (Sum, 5min,
  threshold 0).
- Lambda Duration p99 alarms at ~80% of each function's timeout;
  API-fronted functions eval 3/3, cron/async functions eval 1/1.
  Thresholds pending sign-off.
- DynamoDB orders-table Read/WriteThrottleEvents alarms (TableName dim).
  ThrottledRequests/SystemErrors are not published at the table-only
  dimension, so they are intentionally omitted.
- API Gateway (OrderApi v2) 5xx, 4xx (threshold 20, 3/2 to absorb
  routine authorizer 401s), and p99 Latency alarms.

Update README with a Monitoring section and correct the Lambda count
to 7 (admin-authorizer was missing).

* Drop pending-sign-off wording from alarm docs

Duration/Latency thresholds are owner-approved; remove PENDING ADAM
SIGN-OFF / pending-sign-off notes from template.yaml comments and README.
This commit is contained in:
Adam Moussa 2026-06-17 14:45:54 -04:00 • committed by GitHub
parent 16dbd3b765
commit c6b16678ce
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
2 changed files with 554 additions and 1 deletions

View file

@ -57,10 +57,30 @@ Stack name: `meal-order-manager` (us-east-1)
- **CloudFront** — HTTPS distribution with custom domain `orders.seahaven.com`
- **DynamoDB** — `meal-order-manager-orders` (orders, menu, roster, config)
- **API Gateway** — HttpApi for order submission and admin operations
- **Lambda** — 6 functions: submit-order, close-form, aggregate-orders, slack-notifier, sync-roster, email-report
- **Lambda** — 7 functions: submit-order, admin-authorizer, close-form, aggregate-orders, slack-notifier, sync-roster, email-report
- **EventBridge** — scheduled rules (dual EST/EDT) for close, reminders, payroll email
- **Secrets Manager** — Slack bot token, form API key
- **SES** — payroll deduction emails
- **CloudWatch Alarms** — coverage across the stack, all notifying the shared `site-alerts` SNS topic (see Monitoring)
## Monitoring
CloudWatch alarms are defined in `template.yaml`. Every alarm sends to the shared
`site-alerts` SNS topic (`arn:aws:sns:us-east-1:328440206208:site-alerts`), has no
OKActions, and treats missing data as not breaching (so idle/cron functions don't
sit in ALARM between runs). Alarm names follow `meal-order-manager-<fn>-<signal>`.
- **Lambda Errors / Throttles** — one alarm each per function (7 functions), Sum
over 5 min, fires on any error/throttle (threshold 0).
- **Lambda Duration** — p99 over 5 min at ~80% of each function's timeout.
API-fronted functions (submit-order, admin-authorizer) evaluate 3/3 datapoints;
cron/async functions evaluate a single datapoint.
- **DynamoDB orders table** — `ReadThrottleEvents` and `WriteThrottleEvents`
(TableName dimension). DynamoDB does not publish `ThrottledRequests`/`SystemErrors`
at the table-only dimension, so those are intentionally not alarmed.
- **API Gateway (OrderApi, HTTP API v2)** — 5xx (threshold 0), 4xx (threshold 20,
3/2 datapoints to absorb routine 401s from the token authorizer), and p99 Latency
(~3000ms).
## Authentication

View file

@ -598,6 +598,539 @@ Resources:
LogGroupName: !Sub '/aws/lambda/${SyncRosterFunction}'
RetentionInDays: 60
# ─── CloudWatch Alarms ─────────────────────────────────────────
# All alarms notify the shared site-alerts SNS topic. No OKActions
# (no recovery spam); TreatMissingData notBreaching so idle / cron
# functions don't sit in ALARM between invocations.
#
# Naming: meal-order-manager-<fn>-<signal> (repo-namespaced kebab-case).
#
# Duration alarms use ExtendedStatistic p99 at ~80% of each function's
# configured timeout. Synchronous (API-fronted) functions evaluate over
# 3 datapoints; cron / async-invoked functions evaluate a single datapoint
# (they fire too rarely for a multi-datapoint window).
# Lambda Errors (Sum, 5min, any error breaches)
SubmitOrderErrorsAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: meal-order-manager-submit-order-errors
AlarmDescription: submit-order Lambda reported one or more errors in 5 minutes.
Namespace: AWS/Lambda
MetricName: Errors
Dimensions:
- Name: FunctionName
Value: !Ref SubmitOrderFunction
Statistic: Sum
Period: 300
EvaluationPeriods: 1
Threshold: 0
ComparisonOperator: GreaterThanThreshold
TreatMissingData: notBreaching
AlarmActions:
- arn:aws:sns:us-east-1:328440206208:site-alerts
AdminAuthorizerErrorsAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: meal-order-manager-admin-authorizer-errors
AlarmDescription: admin-authorizer Lambda reported one or more errors in 5 minutes.
Namespace: AWS/Lambda
MetricName: Errors
Dimensions:
- Name: FunctionName
Value: !Ref AdminAuthorizerFunction
Statistic: Sum
Period: 300
EvaluationPeriods: 1
Threshold: 0
ComparisonOperator: GreaterThanThreshold
TreatMissingData: notBreaching
AlarmActions:
- arn:aws:sns:us-east-1:328440206208:site-alerts
CloseFormErrorsAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: meal-order-manager-close-form-errors
AlarmDescription: close-form Lambda reported one or more errors in 5 minutes.
Namespace: AWS/Lambda
MetricName: Errors
Dimensions:
- Name: FunctionName
Value: !Ref CloseFormFunction
Statistic: Sum
Period: 300
EvaluationPeriods: 1
Threshold: 0
ComparisonOperator: GreaterThanThreshold
TreatMissingData: notBreaching
AlarmActions:
- arn:aws:sns:us-east-1:328440206208:site-alerts
AggregateOrdersErrorsAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: meal-order-manager-aggregate-orders-errors
AlarmDescription: aggregate-orders Lambda reported one or more errors in 5 minutes.
Namespace: AWS/Lambda
MetricName: Errors
Dimensions:
- Name: FunctionName
Value: !Ref AggregateOrdersFunction
Statistic: Sum
Period: 300
EvaluationPeriods: 1
Threshold: 0
ComparisonOperator: GreaterThanThreshold
TreatMissingData: notBreaching
AlarmActions:
- arn:aws:sns:us-east-1:328440206208:site-alerts
SlackNotifierErrorsAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: meal-order-manager-slack-notifier-errors
AlarmDescription: slack-notifier Lambda reported one or more errors in 5 minutes.
Namespace: AWS/Lambda
MetricName: Errors
Dimensions:
- Name: FunctionName
Value: !Ref SlackNotifierFunction
Statistic: Sum
Period: 300
EvaluationPeriods: 1
Threshold: 0
ComparisonOperator: GreaterThanThreshold
TreatMissingData: notBreaching
AlarmActions:
- arn:aws:sns:us-east-1:328440206208:site-alerts
SyncRosterErrorsAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: meal-order-manager-sync-roster-errors
AlarmDescription: sync-roster Lambda reported one or more errors in 5 minutes.
Namespace: AWS/Lambda
MetricName: Errors
Dimensions:
- Name: FunctionName
Value: !Ref SyncRosterFunction
Statistic: Sum
Period: 300
EvaluationPeriods: 1
Threshold: 0
ComparisonOperator: GreaterThanThreshold
TreatMissingData: notBreaching
AlarmActions:
- arn:aws:sns:us-east-1:328440206208:site-alerts
EmailReportErrorsAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: meal-order-manager-email-report-errors
AlarmDescription: email-report Lambda reported one or more errors in 5 minutes.
Namespace: AWS/Lambda
MetricName: Errors
Dimensions:
- Name: FunctionName
Value: !Ref EmailReportFunction
Statistic: Sum
Period: 300
EvaluationPeriods: 1
Threshold: 0
ComparisonOperator: GreaterThanThreshold
TreatMissingData: notBreaching
AlarmActions:
- arn:aws:sns:us-east-1:328440206208:site-alerts
# Lambda Throttles (Sum, 5min, any throttle breaches)
SubmitOrderThrottlesAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: meal-order-manager-submit-order-throttles
AlarmDescription: submit-order Lambda was throttled in the last 5 minutes.
Namespace: AWS/Lambda
MetricName: Throttles
Dimensions:
- Name: FunctionName
Value: !Ref SubmitOrderFunction
Statistic: Sum
Period: 300
EvaluationPeriods: 1
Threshold: 0
ComparisonOperator: GreaterThanThreshold
TreatMissingData: notBreaching
AlarmActions:
- arn:aws:sns:us-east-1:328440206208:site-alerts
AdminAuthorizerThrottlesAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: meal-order-manager-admin-authorizer-throttles
AlarmDescription: admin-authorizer Lambda was throttled in the last 5 minutes.
Namespace: AWS/Lambda
MetricName: Throttles
Dimensions:
- Name: FunctionName
Value: !Ref AdminAuthorizerFunction
Statistic: Sum
Period: 300
EvaluationPeriods: 1
Threshold: 0
ComparisonOperator: GreaterThanThreshold
TreatMissingData: notBreaching
AlarmActions:
- arn:aws:sns:us-east-1:328440206208:site-alerts
CloseFormThrottlesAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: meal-order-manager-close-form-throttles
AlarmDescription: close-form Lambda was throttled in the last 5 minutes.
Namespace: AWS/Lambda
MetricName: Throttles
Dimensions:
- Name: FunctionName
Value: !Ref CloseFormFunction
Statistic: Sum
Period: 300
EvaluationPeriods: 1
Threshold: 0
ComparisonOperator: GreaterThanThreshold
TreatMissingData: notBreaching
AlarmActions:
- arn:aws:sns:us-east-1:328440206208:site-alerts
AggregateOrdersThrottlesAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: meal-order-manager-aggregate-orders-throttles
AlarmDescription: aggregate-orders Lambda was throttled in the last 5 minutes.
Namespace: AWS/Lambda
MetricName: Throttles
Dimensions:
- Name: FunctionName
Value: !Ref AggregateOrdersFunction
Statistic: Sum
Period: 300
EvaluationPeriods: 1
Threshold: 0
ComparisonOperator: GreaterThanThreshold
TreatMissingData: notBreaching
AlarmActions:
- arn:aws:sns:us-east-1:328440206208:site-alerts
SlackNotifierThrottlesAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: meal-order-manager-slack-notifier-throttles
AlarmDescription: slack-notifier Lambda was throttled in the last 5 minutes.
Namespace: AWS/Lambda
MetricName: Throttles
Dimensions:
- Name: FunctionName
Value: !Ref SlackNotifierFunction
Statistic: Sum
Period: 300
EvaluationPeriods: 1
Threshold: 0
ComparisonOperator: GreaterThanThreshold
TreatMissingData: notBreaching
AlarmActions:
- arn:aws:sns:us-east-1:328440206208:site-alerts
SyncRosterThrottlesAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: meal-order-manager-sync-roster-throttles
AlarmDescription: sync-roster Lambda was throttled in the last 5 minutes.
Namespace: AWS/Lambda
MetricName: Throttles
Dimensions:
- Name: FunctionName
Value: !Ref SyncRosterFunction
Statistic: Sum
Period: 300
EvaluationPeriods: 1
Threshold: 0
ComparisonOperator: GreaterThanThreshold
TreatMissingData: notBreaching
AlarmActions:
- arn:aws:sns:us-east-1:328440206208:site-alerts
EmailReportThrottlesAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: meal-order-manager-email-report-throttles
AlarmDescription: email-report Lambda was throttled in the last 5 minutes.
Namespace: AWS/Lambda
MetricName: Throttles
Dimensions:
- Name: FunctionName
Value: !Ref EmailReportFunction
Statistic: Sum
Period: 300
EvaluationPeriods: 1
Threshold: 0
ComparisonOperator: GreaterThanThreshold
TreatMissingData: notBreaching
AlarmActions:
- arn:aws:sns:us-east-1:328440206208:site-alerts
# Lambda Duration p99 (~80% of timeout)
# Synchronous (API-fronted) functions: eval 3 / datapoints 3.
SubmitOrderDurationAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: meal-order-manager-submit-order-duration
AlarmDescription: submit-order p99 duration exceeded 8000ms (80% of 10s timeout).
Namespace: AWS/Lambda
MetricName: Duration
Dimensions:
- Name: FunctionName
Value: !Ref SubmitOrderFunction
ExtendedStatistic: p99
Period: 300
EvaluationPeriods: 3
DatapointsToAlarm: 3
Threshold: 8000
ComparisonOperator: GreaterThanThreshold
TreatMissingData: notBreaching
AlarmActions:
- arn:aws:sns:us-east-1:328440206208:site-alerts
AdminAuthorizerDurationAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: meal-order-manager-admin-authorizer-duration
AlarmDescription: admin-authorizer p99 duration exceeded 8000ms (80% of 10s timeout).
Namespace: AWS/Lambda
MetricName: Duration
Dimensions:
- Name: FunctionName
Value: !Ref AdminAuthorizerFunction
ExtendedStatistic: p99
Period: 300
EvaluationPeriods: 3
DatapointsToAlarm: 3
Threshold: 8000
ComparisonOperator: GreaterThanThreshold
TreatMissingData: notBreaching
AlarmActions:
- arn:aws:sns:us-east-1:328440206208:site-alerts
# Cron / async-invoked functions: single datapoint (eval 1).
CloseFormDurationAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: meal-order-manager-close-form-duration
AlarmDescription: close-form p99 duration exceeded 24000ms (80% of 30s timeout).
Namespace: AWS/Lambda
MetricName: Duration
Dimensions:
- Name: FunctionName
Value: !Ref CloseFormFunction
ExtendedStatistic: p99
Period: 300
EvaluationPeriods: 1
DatapointsToAlarm: 1
Threshold: 24000
ComparisonOperator: GreaterThanThreshold
TreatMissingData: notBreaching
AlarmActions:
- arn:aws:sns:us-east-1:328440206208:site-alerts
AggregateOrdersDurationAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: meal-order-manager-aggregate-orders-duration
AlarmDescription: aggregate-orders p99 duration exceeded 48000ms (80% of 60s timeout).
Namespace: AWS/Lambda
MetricName: Duration
Dimensions:
- Name: FunctionName
Value: !Ref AggregateOrdersFunction
ExtendedStatistic: p99
Period: 300
EvaluationPeriods: 1
DatapointsToAlarm: 1
Threshold: 48000
ComparisonOperator: GreaterThanThreshold
TreatMissingData: notBreaching
AlarmActions:
- arn:aws:sns:us-east-1:328440206208:site-alerts
SlackNotifierDurationAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: meal-order-manager-slack-notifier-duration
AlarmDescription: slack-notifier p99 duration exceeded 24000ms (80% of 30s timeout).
Namespace: AWS/Lambda
MetricName: Duration
Dimensions:
- Name: FunctionName
Value: !Ref SlackNotifierFunction
ExtendedStatistic: p99
Period: 300
EvaluationPeriods: 1
DatapointsToAlarm: 1
Threshold: 24000
ComparisonOperator: GreaterThanThreshold
TreatMissingData: notBreaching
AlarmActions:
- arn:aws:sns:us-east-1:328440206208:site-alerts
SyncRosterDurationAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: meal-order-manager-sync-roster-duration
AlarmDescription: sync-roster p99 duration exceeded 48000ms (80% of 60s timeout).
Namespace: AWS/Lambda
MetricName: Duration
Dimensions:
- Name: FunctionName
Value: !Ref SyncRosterFunction
ExtendedStatistic: p99
Period: 300
EvaluationPeriods: 1
DatapointsToAlarm: 1
Threshold: 48000
ComparisonOperator: GreaterThanThreshold
TreatMissingData: notBreaching
AlarmActions:
- arn:aws:sns:us-east-1:328440206208:site-alerts
EmailReportDurationAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: meal-order-manager-email-report-duration
AlarmDescription: email-report p99 duration exceeded 24000ms (80% of 30s timeout).
Namespace: AWS/Lambda
MetricName: Duration
Dimensions:
- Name: FunctionName
Value: !Ref EmailReportFunction
ExtendedStatistic: p99
Period: 300
EvaluationPeriods: 1
DatapointsToAlarm: 1
Threshold: 24000
ComparisonOperator: GreaterThanThreshold
TreatMissingData: notBreaching
AlarmActions:
- arn:aws:sns:us-east-1:328440206208:site-alerts
# DynamoDB orders table.
# NOTE: DynamoDB does NOT publish ThrottledRequests or SystemErrors at the
# TableName-only dimension (verified via cloudwatch list-metrics on
# 2026-06-17 — those metrics carry a TableName+Operation dimension pair and
# only on-occurrence). A TableName-dim alarm on them would never evaluate.
# The codifiable table-level throttle signals are ReadThrottleEvents and
# WriteThrottleEvents, which DO carry a TableName-only dimension. Those are
# used here for throttle coverage; a TableName-dim SystemErrors alarm is
# omitted (no such metric is emitted). See PR body.
OrdersTableReadThrottleAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: meal-order-manager-orders-read-throttle
AlarmDescription: orders table read requests were throttled in the last 5 minutes.
Namespace: AWS/DynamoDB
MetricName: ReadThrottleEvents
Dimensions:
- Name: TableName
Value: !Ref OrdersTable
Statistic: Sum
Period: 300
EvaluationPeriods: 1
Threshold: 0
ComparisonOperator: GreaterThanThreshold
TreatMissingData: notBreaching
AlarmActions:
- arn:aws:sns:us-east-1:328440206208:site-alerts
OrdersTableWriteThrottleAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: meal-order-manager-orders-write-throttle
AlarmDescription: orders table write requests were throttled in the last 5 minutes.
Namespace: AWS/DynamoDB
MetricName: WriteThrottleEvents
Dimensions:
- Name: TableName
Value: !Ref OrdersTable
Statistic: Sum
Period: 300
EvaluationPeriods: 1
Threshold: 0
ComparisonOperator: GreaterThanThreshold
TreatMissingData: notBreaching
AlarmActions:
- arn:aws:sns:us-east-1:328440206208:site-alerts
# API Gateway (HTTP API v2) — OrderApi. Metrics carry the ApiId dimension.
OrderApi5xxAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: meal-order-manager-order-api-5xx
AlarmDescription: OrderApi returned one or more 5xx responses in 5 minutes.
Namespace: AWS/ApiGateway
MetricName: 5xx
Dimensions:
- Name: ApiId
Value: !Ref OrderApi
Statistic: Sum
Period: 300
EvaluationPeriods: 1
Threshold: 0
ComparisonOperator: GreaterThanThreshold
TreatMissingData: notBreaching
AlarmActions:
- arn:aws:sns:us-east-1:328440206208:site-alerts
# 4xx threshold raised + eval 3 / dp 2 to absorb routine 401s from the
# token-based admin authorizer without paging.
OrderApi4xxAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: meal-order-manager-order-api-4xx
AlarmDescription: OrderApi 4xx responses exceeded 20 in 5 minutes (beyond routine auth noise).
Namespace: AWS/ApiGateway
MetricName: 4xx
Dimensions:
- Name: ApiId
Value: !Ref OrderApi
Statistic: Sum
Period: 300
EvaluationPeriods: 3
DatapointsToAlarm: 2
Threshold: 20
ComparisonOperator: GreaterThanThreshold
TreatMissingData: notBreaching
AlarmActions:
- arn:aws:sns:us-east-1:328440206208:site-alerts
# Latency p99 ~3000ms (see PR body).
OrderApiLatencyAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: meal-order-manager-order-api-latency
AlarmDescription: OrderApi p99 latency exceeded 3000ms.
Namespace: AWS/ApiGateway
MetricName: Latency
Dimensions:
- Name: ApiId
Value: !Ref OrderApi
ExtendedStatistic: p99
Period: 300
EvaluationPeriods: 3
DatapointsToAlarm: 3
Threshold: 3000
ComparisonOperator: GreaterThanThreshold
TreatMissingData: notBreaching
AlarmActions:
- arn:aws:sns:us-east-1:328440206208:site-alerts
# ─── SSM Parameters ────────────────────────────────────────────
SlackChannelParam: