diff --git a/README.md b/README.md index d77857b..03f7a87 100644 --- a/README.md +++ b/README.md @@ -57,10 +57,30 @@ Stack name: `meal-order-manager` (us-east-1) - **CloudFront** — HTTPS distribution with custom domain `orders.seahaven.com` - **DynamoDB** — `meal-order-manager-orders` (orders, menu, roster, config) - **API Gateway** — HttpApi for order submission and admin operations -- **Lambda** — 6 functions: submit-order, close-form, aggregate-orders, slack-notifier, sync-roster, email-report +- **Lambda** — 7 functions: submit-order, admin-authorizer, close-form, aggregate-orders, slack-notifier, sync-roster, email-report - **EventBridge** — scheduled rules (dual EST/EDT) for close, reminders, payroll email - **Secrets Manager** — Slack bot token, form API key - **SES** — payroll deduction emails +- **CloudWatch Alarms** — coverage across the stack, all notifying the shared `site-alerts` SNS topic (see Monitoring) + +## Monitoring + +CloudWatch alarms are defined in `template.yaml`. Every alarm sends to the shared +`site-alerts` SNS topic (`arn:aws:sns:us-east-1:328440206208:site-alerts`), has no +OKActions, and treats missing data as not breaching (so idle/cron functions don't +sit in ALARM between runs). Alarm names follow `meal-order-manager--`. + +- **Lambda Errors / Throttles** — one alarm each per function (7 functions), Sum + over 5 min, fires on any error/throttle (threshold 0). +- **Lambda Duration** — p99 over 5 min at ~80% of each function's timeout. + API-fronted functions (submit-order, admin-authorizer) evaluate 3/3 datapoints; + cron/async functions evaluate a single datapoint. +- **DynamoDB orders table** — `ReadThrottleEvents` and `WriteThrottleEvents` + (TableName dimension). DynamoDB does not publish `ThrottledRequests`/`SystemErrors` + at the table-only dimension, so those are intentionally not alarmed. +- **API Gateway (OrderApi, HTTP API v2)** — 5xx (threshold 0), 4xx (threshold 20, + 3/2 datapoints to absorb routine 401s from the token authorizer), and p99 Latency + (~3000ms). ## Authentication diff --git a/template.yaml b/template.yaml index 2c4ab01..0ee8425 100644 --- a/template.yaml +++ b/template.yaml @@ -598,6 +598,539 @@ Resources: LogGroupName: !Sub '/aws/lambda/${SyncRosterFunction}' RetentionInDays: 60 + # ─── CloudWatch Alarms ───────────────────────────────────────── + # All alarms notify the shared site-alerts SNS topic. No OKActions + # (no recovery spam); TreatMissingData notBreaching so idle / cron + # functions don't sit in ALARM between invocations. + # + # Naming: meal-order-manager-- (repo-namespaced kebab-case). + # + # Duration alarms use ExtendedStatistic p99 at ~80% of each function's + # configured timeout. Synchronous (API-fronted) functions evaluate over + # 3 datapoints; cron / async-invoked functions evaluate a single datapoint + # (they fire too rarely for a multi-datapoint window). + + # Lambda Errors (Sum, 5min, any error breaches) + SubmitOrderErrorsAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: meal-order-manager-submit-order-errors + AlarmDescription: submit-order Lambda reported one or more errors in 5 minutes. + Namespace: AWS/Lambda + MetricName: Errors + Dimensions: + - Name: FunctionName + Value: !Ref SubmitOrderFunction + Statistic: Sum + Period: 300 + EvaluationPeriods: 1 + Threshold: 0 + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + AlarmActions: + - arn:aws:sns:us-east-1:328440206208:site-alerts + + AdminAuthorizerErrorsAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: meal-order-manager-admin-authorizer-errors + AlarmDescription: admin-authorizer Lambda reported one or more errors in 5 minutes. + Namespace: AWS/Lambda + MetricName: Errors + Dimensions: + - Name: FunctionName + Value: !Ref AdminAuthorizerFunction + Statistic: Sum + Period: 300 + EvaluationPeriods: 1 + Threshold: 0 + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + AlarmActions: + - arn:aws:sns:us-east-1:328440206208:site-alerts + + CloseFormErrorsAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: meal-order-manager-close-form-errors + AlarmDescription: close-form Lambda reported one or more errors in 5 minutes. + Namespace: AWS/Lambda + MetricName: Errors + Dimensions: + - Name: FunctionName + Value: !Ref CloseFormFunction + Statistic: Sum + Period: 300 + EvaluationPeriods: 1 + Threshold: 0 + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + AlarmActions: + - arn:aws:sns:us-east-1:328440206208:site-alerts + + AggregateOrdersErrorsAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: meal-order-manager-aggregate-orders-errors + AlarmDescription: aggregate-orders Lambda reported one or more errors in 5 minutes. + Namespace: AWS/Lambda + MetricName: Errors + Dimensions: + - Name: FunctionName + Value: !Ref AggregateOrdersFunction + Statistic: Sum + Period: 300 + EvaluationPeriods: 1 + Threshold: 0 + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + AlarmActions: + - arn:aws:sns:us-east-1:328440206208:site-alerts + + SlackNotifierErrorsAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: meal-order-manager-slack-notifier-errors + AlarmDescription: slack-notifier Lambda reported one or more errors in 5 minutes. + Namespace: AWS/Lambda + MetricName: Errors + Dimensions: + - Name: FunctionName + Value: !Ref SlackNotifierFunction + Statistic: Sum + Period: 300 + EvaluationPeriods: 1 + Threshold: 0 + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + AlarmActions: + - arn:aws:sns:us-east-1:328440206208:site-alerts + + SyncRosterErrorsAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: meal-order-manager-sync-roster-errors + AlarmDescription: sync-roster Lambda reported one or more errors in 5 minutes. + Namespace: AWS/Lambda + MetricName: Errors + Dimensions: + - Name: FunctionName + Value: !Ref SyncRosterFunction + Statistic: Sum + Period: 300 + EvaluationPeriods: 1 + Threshold: 0 + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + AlarmActions: + - arn:aws:sns:us-east-1:328440206208:site-alerts + + EmailReportErrorsAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: meal-order-manager-email-report-errors + AlarmDescription: email-report Lambda reported one or more errors in 5 minutes. + Namespace: AWS/Lambda + MetricName: Errors + Dimensions: + - Name: FunctionName + Value: !Ref EmailReportFunction + Statistic: Sum + Period: 300 + EvaluationPeriods: 1 + Threshold: 0 + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + AlarmActions: + - arn:aws:sns:us-east-1:328440206208:site-alerts + + # Lambda Throttles (Sum, 5min, any throttle breaches) + SubmitOrderThrottlesAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: meal-order-manager-submit-order-throttles + AlarmDescription: submit-order Lambda was throttled in the last 5 minutes. + Namespace: AWS/Lambda + MetricName: Throttles + Dimensions: + - Name: FunctionName + Value: !Ref SubmitOrderFunction + Statistic: Sum + Period: 300 + EvaluationPeriods: 1 + Threshold: 0 + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + AlarmActions: + - arn:aws:sns:us-east-1:328440206208:site-alerts + + AdminAuthorizerThrottlesAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: meal-order-manager-admin-authorizer-throttles + AlarmDescription: admin-authorizer Lambda was throttled in the last 5 minutes. + Namespace: AWS/Lambda + MetricName: Throttles + Dimensions: + - Name: FunctionName + Value: !Ref AdminAuthorizerFunction + Statistic: Sum + Period: 300 + EvaluationPeriods: 1 + Threshold: 0 + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + AlarmActions: + - arn:aws:sns:us-east-1:328440206208:site-alerts + + CloseFormThrottlesAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: meal-order-manager-close-form-throttles + AlarmDescription: close-form Lambda was throttled in the last 5 minutes. + Namespace: AWS/Lambda + MetricName: Throttles + Dimensions: + - Name: FunctionName + Value: !Ref CloseFormFunction + Statistic: Sum + Period: 300 + EvaluationPeriods: 1 + Threshold: 0 + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + AlarmActions: + - arn:aws:sns:us-east-1:328440206208:site-alerts + + AggregateOrdersThrottlesAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: meal-order-manager-aggregate-orders-throttles + AlarmDescription: aggregate-orders Lambda was throttled in the last 5 minutes. + Namespace: AWS/Lambda + MetricName: Throttles + Dimensions: + - Name: FunctionName + Value: !Ref AggregateOrdersFunction + Statistic: Sum + Period: 300 + EvaluationPeriods: 1 + Threshold: 0 + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + AlarmActions: + - arn:aws:sns:us-east-1:328440206208:site-alerts + + SlackNotifierThrottlesAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: meal-order-manager-slack-notifier-throttles + AlarmDescription: slack-notifier Lambda was throttled in the last 5 minutes. + Namespace: AWS/Lambda + MetricName: Throttles + Dimensions: + - Name: FunctionName + Value: !Ref SlackNotifierFunction + Statistic: Sum + Period: 300 + EvaluationPeriods: 1 + Threshold: 0 + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + AlarmActions: + - arn:aws:sns:us-east-1:328440206208:site-alerts + + SyncRosterThrottlesAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: meal-order-manager-sync-roster-throttles + AlarmDescription: sync-roster Lambda was throttled in the last 5 minutes. + Namespace: AWS/Lambda + MetricName: Throttles + Dimensions: + - Name: FunctionName + Value: !Ref SyncRosterFunction + Statistic: Sum + Period: 300 + EvaluationPeriods: 1 + Threshold: 0 + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + AlarmActions: + - arn:aws:sns:us-east-1:328440206208:site-alerts + + EmailReportThrottlesAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: meal-order-manager-email-report-throttles + AlarmDescription: email-report Lambda was throttled in the last 5 minutes. + Namespace: AWS/Lambda + MetricName: Throttles + Dimensions: + - Name: FunctionName + Value: !Ref EmailReportFunction + Statistic: Sum + Period: 300 + EvaluationPeriods: 1 + Threshold: 0 + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + AlarmActions: + - arn:aws:sns:us-east-1:328440206208:site-alerts + + # Lambda Duration p99 (~80% of timeout) + # Synchronous (API-fronted) functions: eval 3 / datapoints 3. + SubmitOrderDurationAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: meal-order-manager-submit-order-duration + AlarmDescription: submit-order p99 duration exceeded 8000ms (80% of 10s timeout). + Namespace: AWS/Lambda + MetricName: Duration + Dimensions: + - Name: FunctionName + Value: !Ref SubmitOrderFunction + ExtendedStatistic: p99 + Period: 300 + EvaluationPeriods: 3 + DatapointsToAlarm: 3 + Threshold: 8000 + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + AlarmActions: + - arn:aws:sns:us-east-1:328440206208:site-alerts + + AdminAuthorizerDurationAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: meal-order-manager-admin-authorizer-duration + AlarmDescription: admin-authorizer p99 duration exceeded 8000ms (80% of 10s timeout). + Namespace: AWS/Lambda + MetricName: Duration + Dimensions: + - Name: FunctionName + Value: !Ref AdminAuthorizerFunction + ExtendedStatistic: p99 + Period: 300 + EvaluationPeriods: 3 + DatapointsToAlarm: 3 + Threshold: 8000 + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + AlarmActions: + - arn:aws:sns:us-east-1:328440206208:site-alerts + + # Cron / async-invoked functions: single datapoint (eval 1). + CloseFormDurationAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: meal-order-manager-close-form-duration + AlarmDescription: close-form p99 duration exceeded 24000ms (80% of 30s timeout). + Namespace: AWS/Lambda + MetricName: Duration + Dimensions: + - Name: FunctionName + Value: !Ref CloseFormFunction + ExtendedStatistic: p99 + Period: 300 + EvaluationPeriods: 1 + DatapointsToAlarm: 1 + Threshold: 24000 + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + AlarmActions: + - arn:aws:sns:us-east-1:328440206208:site-alerts + + AggregateOrdersDurationAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: meal-order-manager-aggregate-orders-duration + AlarmDescription: aggregate-orders p99 duration exceeded 48000ms (80% of 60s timeout). + Namespace: AWS/Lambda + MetricName: Duration + Dimensions: + - Name: FunctionName + Value: !Ref AggregateOrdersFunction + ExtendedStatistic: p99 + Period: 300 + EvaluationPeriods: 1 + DatapointsToAlarm: 1 + Threshold: 48000 + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + AlarmActions: + - arn:aws:sns:us-east-1:328440206208:site-alerts + + SlackNotifierDurationAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: meal-order-manager-slack-notifier-duration + AlarmDescription: slack-notifier p99 duration exceeded 24000ms (80% of 30s timeout). + Namespace: AWS/Lambda + MetricName: Duration + Dimensions: + - Name: FunctionName + Value: !Ref SlackNotifierFunction + ExtendedStatistic: p99 + Period: 300 + EvaluationPeriods: 1 + DatapointsToAlarm: 1 + Threshold: 24000 + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + AlarmActions: + - arn:aws:sns:us-east-1:328440206208:site-alerts + + SyncRosterDurationAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: meal-order-manager-sync-roster-duration + AlarmDescription: sync-roster p99 duration exceeded 48000ms (80% of 60s timeout). + Namespace: AWS/Lambda + MetricName: Duration + Dimensions: + - Name: FunctionName + Value: !Ref SyncRosterFunction + ExtendedStatistic: p99 + Period: 300 + EvaluationPeriods: 1 + DatapointsToAlarm: 1 + Threshold: 48000 + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + AlarmActions: + - arn:aws:sns:us-east-1:328440206208:site-alerts + + EmailReportDurationAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: meal-order-manager-email-report-duration + AlarmDescription: email-report p99 duration exceeded 24000ms (80% of 30s timeout). + Namespace: AWS/Lambda + MetricName: Duration + Dimensions: + - Name: FunctionName + Value: !Ref EmailReportFunction + ExtendedStatistic: p99 + Period: 300 + EvaluationPeriods: 1 + DatapointsToAlarm: 1 + Threshold: 24000 + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + AlarmActions: + - arn:aws:sns:us-east-1:328440206208:site-alerts + + # DynamoDB orders table. + # NOTE: DynamoDB does NOT publish ThrottledRequests or SystemErrors at the + # TableName-only dimension (verified via cloudwatch list-metrics on + # 2026-06-17 — those metrics carry a TableName+Operation dimension pair and + # only on-occurrence). A TableName-dim alarm on them would never evaluate. + # The codifiable table-level throttle signals are ReadThrottleEvents and + # WriteThrottleEvents, which DO carry a TableName-only dimension. Those are + # used here for throttle coverage; a TableName-dim SystemErrors alarm is + # omitted (no such metric is emitted). See PR body. + OrdersTableReadThrottleAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: meal-order-manager-orders-read-throttle + AlarmDescription: orders table read requests were throttled in the last 5 minutes. + Namespace: AWS/DynamoDB + MetricName: ReadThrottleEvents + Dimensions: + - Name: TableName + Value: !Ref OrdersTable + Statistic: Sum + Period: 300 + EvaluationPeriods: 1 + Threshold: 0 + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + AlarmActions: + - arn:aws:sns:us-east-1:328440206208:site-alerts + + OrdersTableWriteThrottleAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: meal-order-manager-orders-write-throttle + AlarmDescription: orders table write requests were throttled in the last 5 minutes. + Namespace: AWS/DynamoDB + MetricName: WriteThrottleEvents + Dimensions: + - Name: TableName + Value: !Ref OrdersTable + Statistic: Sum + Period: 300 + EvaluationPeriods: 1 + Threshold: 0 + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + AlarmActions: + - arn:aws:sns:us-east-1:328440206208:site-alerts + + # API Gateway (HTTP API v2) — OrderApi. Metrics carry the ApiId dimension. + OrderApi5xxAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: meal-order-manager-order-api-5xx + AlarmDescription: OrderApi returned one or more 5xx responses in 5 minutes. + Namespace: AWS/ApiGateway + MetricName: 5xx + Dimensions: + - Name: ApiId + Value: !Ref OrderApi + Statistic: Sum + Period: 300 + EvaluationPeriods: 1 + Threshold: 0 + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + AlarmActions: + - arn:aws:sns:us-east-1:328440206208:site-alerts + + # 4xx threshold raised + eval 3 / dp 2 to absorb routine 401s from the + # token-based admin authorizer without paging. + OrderApi4xxAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: meal-order-manager-order-api-4xx + AlarmDescription: OrderApi 4xx responses exceeded 20 in 5 minutes (beyond routine auth noise). + Namespace: AWS/ApiGateway + MetricName: 4xx + Dimensions: + - Name: ApiId + Value: !Ref OrderApi + Statistic: Sum + Period: 300 + EvaluationPeriods: 3 + DatapointsToAlarm: 2 + Threshold: 20 + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + AlarmActions: + - arn:aws:sns:us-east-1:328440206208:site-alerts + + # Latency p99 ~3000ms (see PR body). + OrderApiLatencyAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: meal-order-manager-order-api-latency + AlarmDescription: OrderApi p99 latency exceeded 3000ms. + Namespace: AWS/ApiGateway + MetricName: Latency + Dimensions: + - Name: ApiId + Value: !Ref OrderApi + ExtendedStatistic: p99 + Period: 300 + EvaluationPeriods: 3 + DatapointsToAlarm: 3 + Threshold: 3000 + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + AlarmActions: + - arn:aws:sns:us-east-1:328440206208:site-alerts + # ─── SSM Parameters ──────────────────────────────────────────── SlackChannelParam: