Add CloudWatch alarm coverage for the meal-order-manager stack

Add CloudWatch alarms (all notifying the shared site-alerts SNS topic,
no OKActions, TreatMissingData notBreaching) across the stack:

- Lambda Errors + Throttles alarms for all 7 functions (Sum, 5min,
  threshold 0).
- Lambda Duration p99 alarms at ~80% of each function's timeout;
  API-fronted functions eval 3/3, cron/async functions eval 1/1.
  Thresholds pending sign-off.
- DynamoDB orders-table Read/WriteThrottleEvents alarms (TableName dim).
  ThrottledRequests/SystemErrors are not published at the table-only
  dimension, so they are intentionally omitted.
- API Gateway (OrderApi v2) 5xx, 4xx (threshold 20, 3/2 to absorb
  routine authorizer 401s), and p99 Latency alarms.

Update README with a Monitoring section and correct the Lambda count
to 7 (admin-authorizer was missing).

README.md

@@ -57,10 +57,30 @@ Stack name: `meal-order-manager` (us-east-1)
 - **CloudFront** — HTTPS distribution with custom domain `orders.seahaven.com`
 - **DynamoDB** — `meal-order-manager-orders` (orders, menu, roster, config)
 - **API Gateway** — HttpApi for order submission and admin operations
-- **Lambda** — 6 functions: submit-order, close-form, aggregate-orders, slack-notifier, sync-roster, email-report
+- **Lambda** — 7 functions: submit-order, admin-authorizer, close-form, aggregate-orders, slack-notifier, sync-roster, email-report
 - **EventBridge** — scheduled rules (dual EST/EDT) for close, reminders, payroll email
 - **Secrets Manager** — Slack bot token, form API key
 - **SES** — payroll deduction emails
+- **CloudWatch Alarms** — coverage across the stack, all notifying the shared `site-alerts` SNS topic (see Monitoring)
+
+## Monitoring
+
+CloudWatch alarms are defined in `template.yaml`. Every alarm sends to the shared
+`site-alerts` SNS topic (`arn:aws:sns:us-east-1:328440206208:site-alerts`), has no
+OKActions, and treats missing data as not breaching (so idle/cron functions don't
+sit in ALARM between runs). Alarm names follow `meal-order-manager-<fn>-<signal>`.
+
+- **Lambda Errors / Throttles** — one alarm each per function (7 functions), Sum
+  over 5 min, fires on any error/throttle (threshold 0).
+- **Lambda Duration** — p99 over 5 min at ~80% of each function's timeout.
+  API-fronted functions (submit-order, admin-authorizer) evaluate 3/3 datapoints;
+  cron/async functions evaluate a single datapoint. Thresholds pending sign-off.
+- **DynamoDB orders table** — `ReadThrottleEvents` and `WriteThrottleEvents`
+  (TableName dimension). DynamoDB does not publish `ThrottledRequests`/`SystemErrors`
+  at the table-only dimension, so those are intentionally not alarmed.
+- **API Gateway (OrderApi, HTTP API v2)** — 5xx (threshold 0), 4xx (threshold 20,
+  3/2 datapoints to absorb routine 401s from the token authorizer), and p99 Latency
+  (~3000ms, pending sign-off).
 
 ## Authentication
 

template.yaml

@@ -598,6 +598,540 @@ Resources:
       LogGroupName: !Sub '/aws/lambda/${SyncRosterFunction}'
       RetentionInDays: 60
 
+  # ─── CloudWatch Alarms ─────────────────────────────────────────
+  # All alarms notify the shared site-alerts SNS topic. No OKActions
+  # (no recovery spam); TreatMissingData notBreaching so idle / cron
+  # functions don't sit in ALARM between invocations.
+  #
+  # Naming: meal-order-manager-<fn>-<signal> (repo-namespaced kebab-case).
+  #
+  # Duration alarms use ExtendedStatistic p99 at ~80% of each function's
+  # configured timeout. Synchronous (API-fronted) functions evaluate over
+  # 3 datapoints; cron / async-invoked functions evaluate a single datapoint
+  # (they fire too rarely for a multi-datapoint window). These thresholds
+  # are pending Adam sign-off — see PR body.
+
+  # Lambda Errors (Sum, 5min, any error breaches)
+  SubmitOrderErrorsAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: meal-order-manager-submit-order-errors
+      AlarmDescription: submit-order Lambda reported one or more errors in 5 minutes.
+      Namespace: AWS/Lambda
+      MetricName: Errors
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref SubmitOrderFunction
+      Statistic: Sum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 0
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - arn:aws:sns:us-east-1:328440206208:site-alerts
+
+  AdminAuthorizerErrorsAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: meal-order-manager-admin-authorizer-errors
+      AlarmDescription: admin-authorizer Lambda reported one or more errors in 5 minutes.
+      Namespace: AWS/Lambda
+      MetricName: Errors
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref AdminAuthorizerFunction
+      Statistic: Sum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 0
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - arn:aws:sns:us-east-1:328440206208:site-alerts
+
+  CloseFormErrorsAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: meal-order-manager-close-form-errors
+      AlarmDescription: close-form Lambda reported one or more errors in 5 minutes.
+      Namespace: AWS/Lambda
+      MetricName: Errors
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref CloseFormFunction
+      Statistic: Sum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 0
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - arn:aws:sns:us-east-1:328440206208:site-alerts
+
+  AggregateOrdersErrorsAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: meal-order-manager-aggregate-orders-errors
+      AlarmDescription: aggregate-orders Lambda reported one or more errors in 5 minutes.
+      Namespace: AWS/Lambda
+      MetricName: Errors
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref AggregateOrdersFunction
+      Statistic: Sum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 0
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - arn:aws:sns:us-east-1:328440206208:site-alerts
+
+  SlackNotifierErrorsAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: meal-order-manager-slack-notifier-errors
+      AlarmDescription: slack-notifier Lambda reported one or more errors in 5 minutes.
+      Namespace: AWS/Lambda
+      MetricName: Errors
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref SlackNotifierFunction
+      Statistic: Sum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 0
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - arn:aws:sns:us-east-1:328440206208:site-alerts
+
+  SyncRosterErrorsAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: meal-order-manager-sync-roster-errors
+      AlarmDescription: sync-roster Lambda reported one or more errors in 5 minutes.
+      Namespace: AWS/Lambda
+      MetricName: Errors
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref SyncRosterFunction
+      Statistic: Sum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 0
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - arn:aws:sns:us-east-1:328440206208:site-alerts
+
+  EmailReportErrorsAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: meal-order-manager-email-report-errors
+      AlarmDescription: email-report Lambda reported one or more errors in 5 minutes.
+      Namespace: AWS/Lambda
+      MetricName: Errors
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref EmailReportFunction
+      Statistic: Sum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 0
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - arn:aws:sns:us-east-1:328440206208:site-alerts
+
+  # Lambda Throttles (Sum, 5min, any throttle breaches)
+  SubmitOrderThrottlesAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: meal-order-manager-submit-order-throttles
+      AlarmDescription: submit-order Lambda was throttled in the last 5 minutes.
+      Namespace: AWS/Lambda
+      MetricName: Throttles
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref SubmitOrderFunction
+      Statistic: Sum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 0
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - arn:aws:sns:us-east-1:328440206208:site-alerts
+
+  AdminAuthorizerThrottlesAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: meal-order-manager-admin-authorizer-throttles
+      AlarmDescription: admin-authorizer Lambda was throttled in the last 5 minutes.
+      Namespace: AWS/Lambda
+      MetricName: Throttles
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref AdminAuthorizerFunction
+      Statistic: Sum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 0
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - arn:aws:sns:us-east-1:328440206208:site-alerts
+
+  CloseFormThrottlesAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: meal-order-manager-close-form-throttles
+      AlarmDescription: close-form Lambda was throttled in the last 5 minutes.
+      Namespace: AWS/Lambda
+      MetricName: Throttles
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref CloseFormFunction
+      Statistic: Sum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 0
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - arn:aws:sns:us-east-1:328440206208:site-alerts
+
+  AggregateOrdersThrottlesAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: meal-order-manager-aggregate-orders-throttles
+      AlarmDescription: aggregate-orders Lambda was throttled in the last 5 minutes.
+      Namespace: AWS/Lambda
+      MetricName: Throttles
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref AggregateOrdersFunction
+      Statistic: Sum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 0
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - arn:aws:sns:us-east-1:328440206208:site-alerts
+
+  SlackNotifierThrottlesAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: meal-order-manager-slack-notifier-throttles
+      AlarmDescription: slack-notifier Lambda was throttled in the last 5 minutes.
+      Namespace: AWS/Lambda
+      MetricName: Throttles
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref SlackNotifierFunction
+      Statistic: Sum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 0
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - arn:aws:sns:us-east-1:328440206208:site-alerts
+
+  SyncRosterThrottlesAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: meal-order-manager-sync-roster-throttles
+      AlarmDescription: sync-roster Lambda was throttled in the last 5 minutes.
+      Namespace: AWS/Lambda
+      MetricName: Throttles
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref SyncRosterFunction
+      Statistic: Sum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 0
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - arn:aws:sns:us-east-1:328440206208:site-alerts
+
+  EmailReportThrottlesAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: meal-order-manager-email-report-throttles
+      AlarmDescription: email-report Lambda was throttled in the last 5 minutes.
+      Namespace: AWS/Lambda
+      MetricName: Throttles
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref EmailReportFunction
+      Statistic: Sum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 0
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - arn:aws:sns:us-east-1:328440206208:site-alerts
+
+  # Lambda Duration p99 (~80% of timeout) — PENDING ADAM SIGN-OFF
+  # Synchronous (API-fronted) functions: eval 3 / datapoints 3.
+  SubmitOrderDurationAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: meal-order-manager-submit-order-duration
+      AlarmDescription: submit-order p99 duration exceeded 8000ms (80% of 10s timeout).
+      Namespace: AWS/Lambda
+      MetricName: Duration
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref SubmitOrderFunction
+      ExtendedStatistic: p99
+      Period: 300
+      EvaluationPeriods: 3
+      DatapointsToAlarm: 3
+      Threshold: 8000
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - arn:aws:sns:us-east-1:328440206208:site-alerts
+
+  AdminAuthorizerDurationAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: meal-order-manager-admin-authorizer-duration
+      AlarmDescription: admin-authorizer p99 duration exceeded 8000ms (80% of 10s timeout).
+      Namespace: AWS/Lambda
+      MetricName: Duration
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref AdminAuthorizerFunction
+      ExtendedStatistic: p99
+      Period: 300
+      EvaluationPeriods: 3
+      DatapointsToAlarm: 3
+      Threshold: 8000
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - arn:aws:sns:us-east-1:328440206208:site-alerts
+
+  # Cron / async-invoked functions: single datapoint (eval 1).
+  CloseFormDurationAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: meal-order-manager-close-form-duration
+      AlarmDescription: close-form p99 duration exceeded 24000ms (80% of 30s timeout).
+      Namespace: AWS/Lambda
+      MetricName: Duration
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref CloseFormFunction
+      ExtendedStatistic: p99
+      Period: 300
+      EvaluationPeriods: 1
+      DatapointsToAlarm: 1
+      Threshold: 24000
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - arn:aws:sns:us-east-1:328440206208:site-alerts
+
+  AggregateOrdersDurationAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: meal-order-manager-aggregate-orders-duration
+      AlarmDescription: aggregate-orders p99 duration exceeded 48000ms (80% of 60s timeout).
+      Namespace: AWS/Lambda
+      MetricName: Duration
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref AggregateOrdersFunction
+      ExtendedStatistic: p99
+      Period: 300
+      EvaluationPeriods: 1
+      DatapointsToAlarm: 1
+      Threshold: 48000
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - arn:aws:sns:us-east-1:328440206208:site-alerts
+
+  SlackNotifierDurationAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: meal-order-manager-slack-notifier-duration
+      AlarmDescription: slack-notifier p99 duration exceeded 24000ms (80% of 30s timeout).
+      Namespace: AWS/Lambda
+      MetricName: Duration
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref SlackNotifierFunction
+      ExtendedStatistic: p99
+      Period: 300
+      EvaluationPeriods: 1
+      DatapointsToAlarm: 1
+      Threshold: 24000
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - arn:aws:sns:us-east-1:328440206208:site-alerts
+
+  SyncRosterDurationAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: meal-order-manager-sync-roster-duration
+      AlarmDescription: sync-roster p99 duration exceeded 48000ms (80% of 60s timeout).
+      Namespace: AWS/Lambda
+      MetricName: Duration
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref SyncRosterFunction
+      ExtendedStatistic: p99
+      Period: 300
+      EvaluationPeriods: 1
+      DatapointsToAlarm: 1
+      Threshold: 48000
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - arn:aws:sns:us-east-1:328440206208:site-alerts
+
+  EmailReportDurationAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: meal-order-manager-email-report-duration
+      AlarmDescription: email-report p99 duration exceeded 24000ms (80% of 30s timeout).
+      Namespace: AWS/Lambda
+      MetricName: Duration
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref EmailReportFunction
+      ExtendedStatistic: p99
+      Period: 300
+      EvaluationPeriods: 1
+      DatapointsToAlarm: 1
+      Threshold: 24000
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - arn:aws:sns:us-east-1:328440206208:site-alerts
+
+  # DynamoDB orders table.
+  # NOTE: DynamoDB does NOT publish ThrottledRequests or SystemErrors at the
+  # TableName-only dimension (verified via cloudwatch list-metrics on
+  # 2026-06-17 — those metrics carry a TableName+Operation dimension pair and
+  # only on-occurrence). A TableName-dim alarm on them would never evaluate.
+  # The codifiable table-level throttle signals are ReadThrottleEvents and
+  # WriteThrottleEvents, which DO carry a TableName-only dimension. Those are
+  # used here for throttle coverage; a TableName-dim SystemErrors alarm is
+  # omitted (no such metric is emitted). See PR body.
+  OrdersTableReadThrottleAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: meal-order-manager-orders-read-throttle
+      AlarmDescription: orders table read requests were throttled in the last 5 minutes.
+      Namespace: AWS/DynamoDB
+      MetricName: ReadThrottleEvents
+      Dimensions:
+        - Name: TableName
+          Value: !Ref OrdersTable
+      Statistic: Sum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 0
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - arn:aws:sns:us-east-1:328440206208:site-alerts
+
+  OrdersTableWriteThrottleAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: meal-order-manager-orders-write-throttle
+      AlarmDescription: orders table write requests were throttled in the last 5 minutes.
+      Namespace: AWS/DynamoDB
+      MetricName: WriteThrottleEvents
+      Dimensions:
+        - Name: TableName
+          Value: !Ref OrdersTable
+      Statistic: Sum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 0
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - arn:aws:sns:us-east-1:328440206208:site-alerts
+
+  # API Gateway (HTTP API v2) — OrderApi. Metrics carry the ApiId dimension.
+  OrderApi5xxAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: meal-order-manager-order-api-5xx
+      AlarmDescription: OrderApi returned one or more 5xx responses in 5 minutes.
+      Namespace: AWS/ApiGateway
+      MetricName: 5xx
+      Dimensions:
+        - Name: ApiId
+          Value: !Ref OrderApi
+      Statistic: Sum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 0
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - arn:aws:sns:us-east-1:328440206208:site-alerts
+
+  # 4xx threshold raised + eval 3 / dp 2 to absorb routine 401s from the
+  # token-based admin authorizer without paging.
+  OrderApi4xxAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: meal-order-manager-order-api-4xx
+      AlarmDescription: OrderApi 4xx responses exceeded 20 in 5 minutes (beyond routine auth noise).
+      Namespace: AWS/ApiGateway
+      MetricName: 4xx
+      Dimensions:
+        - Name: ApiId
+          Value: !Ref OrderApi
+      Statistic: Sum
+      Period: 300
+      EvaluationPeriods: 3
+      DatapointsToAlarm: 2
+      Threshold: 20
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - arn:aws:sns:us-east-1:328440206208:site-alerts
+
+  # Latency p99 ~3000ms — PENDING ADAM SIGN-OFF (see PR body).
+  OrderApiLatencyAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: meal-order-manager-order-api-latency
+      AlarmDescription: OrderApi p99 latency exceeded 3000ms.
+      Namespace: AWS/ApiGateway
+      MetricName: Latency
+      Dimensions:
+        - Name: ApiId
+          Value: !Ref OrderApi
+      ExtendedStatistic: p99
+      Period: 300
+      EvaluationPeriods: 3
+      DatapointsToAlarm: 3
+      Threshold: 3000
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - arn:aws:sns:us-east-1:328440206208:site-alerts
+
   # ─── SSM Parameters ────────────────────────────────────────────
 
   SlackChannelParam: