Add CloudWatch alarm coverage (payments-dashboard) (#51)

* Add CloudWatch alarm coverage for payments-dashboard (Wave 1)

Add 22 CloudWatch alarms to round out observability:
- Lambda Errors alarms for the 4 functions that lacked them
  (slackAppHome, fetchBoaTransactions, expenseReceiver, expenseProcessor)
- Lambda Throttles alarms for all 6 functions
- Lambda Duration alarms for all 6 functions (~80% of timeout, Maximum)
- DynamoDB throttle/system-error alarms for the PaymentsDashboard table
  (ReadThrottleEvents/WriteThrottleEvents/SystemErrors, TableName dim)
- API Gateway v2 alarms on the implicit ServerlessHttpApi
  (5xx, 4xx, Latency p99; ApiId dim)

All alarms publish to the site-alerts SNS topic, ALARM-only, missing data
notBreaching, matching the existing payments-<fn>-errors convention from PR #48.
Documents the full alarm inventory under a new README Monitoring section.

* Drop DynamoDB SystemErrors alarm (never fires at TableName dimension)

AWS/DynamoDB SystemErrors does not emit at the TableName-only dimension,
so payments-dashboard-table-system-errors could never fire. Remove the
alarm resource and its README entry; keep Read/WriteThrottleEvents.

README.md

@@ -79,6 +79,32 @@ All BoA and Slack credentials are stored in AWS Secrets Manager (per `engineerin
 
 `boa-account-number` is duplicated into both BoA secrets. Each Lambda is granted `secretsmanager:GetSecretValue` scoped to only the secret it needs. The Expense Approval Bot uses two additional secrets (`payments-dashboard/expense-slack-token`, `payments-dashboard/expense-slack-signing-secret`).
 
+## Monitoring & Alarms
+
+All CloudWatch alarms publish to the shared `site-alerts` SNS topic (`arn:aws:sns:us-east-1:328440206208:site-alerts`). Alarms are ALARM-only by convention (no OK/recovery action) and treat missing data as `notBreaching`. Each alarm evaluates a single 5-minute period.
+
+**SQS dead-letter queues** (messages-present, Maximum > 0):
+
+| Alarm | Source |
+|-------|--------|
+| `payments-payroll-batch-dlq-messages` | `payments-payroll-batch-dlq` |
+| `payments-processPaymentCsv-async-dlq-messages` | async-invoke OnFailure DLQ |
+| `payments-processPayrollEmail-async-dlq-messages` | async-invoke OnFailure DLQ |
+
+**Lambda** (per function — `payments-<fn>-...`):
+
+| Type | Metric / Statistic | Threshold |
+|------|--------------------|-----------|
+| `-errors` (all 6) | `Errors` / Sum | > 0 |
+| `-throttles` (all 6) | `Throttles` / Sum | > 0 |
+| `-duration` (all 6) | `Duration` / Maximum | ~80% of each function's timeout |
+
+Duration thresholds (ms): processPaymentCsv 96000, processPayrollEmail 48000, fetchBoaTransactions 48000, slackAppHome 24000, expenseProcessor 12000, expenseReceiver 4000.
+
+**DynamoDB** (`PaymentsDashboard` table, `TableName` dimension, Sum > 0): `payments-dashboard-table-read-throttle` (`ReadThrottleEvents`), `payments-dashboard-table-write-throttle` (`WriteThrottleEvents`). The table is PAY_PER_REQUEST; these metrics emit only when a throttle occurs. `SystemErrors` is intentionally not alarmed because it does not emit at the `TableName`-only dimension.
+
+**API Gateway** (implicit HTTP API v2 `ServerlessHttpApi`, `ApiId` dimension): `payments-dashboard-api-5xx` (`5xx` Sum > 0), `payments-dashboard-api-4xx` (`4xx` Sum > 10, client-error noise floor), `payments-dashboard-api-latency-p99` (`Latency` p99 > 3000 ms).
+
 ## Scripts
 
 | Script | Purpose |

template.yaml

@@ -338,6 +338,427 @@ Resources:
       AlarmActions:
         - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts
 
+  # ── Lambda Errors alarms (Wave 1) ──────────────────────────────────────────
+  # Clone of ProcessPaymentCsvErrorsAlarm for the remaining functions. AWS/Lambda
+  # Errors, Sum over 5m, threshold > 0, ALARM-only by convention.
+  SlackAppHomeErrorsAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: payments-slackAppHome-errors
+      AlarmDescription: payments-slackAppHome invocation errors
+      Namespace: AWS/Lambda
+      MetricName: Errors
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref SlackAppHomeFunction
+      Statistic: Sum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 0
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts
+
+  FetchBoaTransactionsErrorsAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: payments-fetchBoaTransactions-errors
+      AlarmDescription: payments-fetchBoaTransactions invocation errors
+      Namespace: AWS/Lambda
+      MetricName: Errors
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref FetchBoaTransactionsFunction
+      Statistic: Sum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 0
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts
+
+  ExpenseReceiverErrorsAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: payments-expenseReceiver-errors
+      AlarmDescription: payments-expenseReceiver invocation errors
+      Namespace: AWS/Lambda
+      MetricName: Errors
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref ExpenseReceiverFunction
+      Statistic: Sum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 0
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts
+
+  ExpenseProcessorErrorsAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: payments-expenseProcessor-errors
+      AlarmDescription: payments-expenseProcessor invocation errors
+      Namespace: AWS/Lambda
+      MetricName: Errors
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref ExpenseProcessorFunction
+      Statistic: Sum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 0
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts
+
+  # ── Lambda Throttles alarms (Wave 1) ───────────────────────────────────────
+  # AWS/Lambda Throttles, Sum over 5m, threshold > 0, ALARM-only. Throttling
+  # signals concurrency exhaustion / reserved-concurrency starvation.
+  ProcessPaymentCsvThrottlesAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: payments-processPaymentCsv-throttles
+      AlarmDescription: payments-processPaymentCsv invocations throttled
+      Namespace: AWS/Lambda
+      MetricName: Throttles
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref ProcessPaymentCsvFunction
+      Statistic: Sum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 0
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts
+
+  ProcessPayrollEmailThrottlesAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: payments-processPayrollEmail-throttles
+      AlarmDescription: payments-processPayrollEmail invocations throttled
+      Namespace: AWS/Lambda
+      MetricName: Throttles
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref ProcessPayrollEmailFunction
+      Statistic: Sum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 0
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts
+
+  FetchBoaTransactionsThrottlesAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: payments-fetchBoaTransactions-throttles
+      AlarmDescription: payments-fetchBoaTransactions invocations throttled
+      Namespace: AWS/Lambda
+      MetricName: Throttles
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref FetchBoaTransactionsFunction
+      Statistic: Sum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 0
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts
+
+  SlackAppHomeThrottlesAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: payments-slackAppHome-throttles
+      AlarmDescription: payments-slackAppHome invocations throttled
+      Namespace: AWS/Lambda
+      MetricName: Throttles
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref SlackAppHomeFunction
+      Statistic: Sum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 0
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts
+
+  ExpenseReceiverThrottlesAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: payments-expenseReceiver-throttles
+      AlarmDescription: payments-expenseReceiver invocations throttled
+      Namespace: AWS/Lambda
+      MetricName: Throttles
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref ExpenseReceiverFunction
+      Statistic: Sum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 0
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts
+
+  ExpenseProcessorThrottlesAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: payments-expenseProcessor-throttles
+      AlarmDescription: payments-expenseProcessor invocations throttled
+      Namespace: AWS/Lambda
+      MetricName: Throttles
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref ExpenseProcessorFunction
+      Statistic: Sum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 0
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts
+
+  # ── Lambda Duration alarms (Wave 1) ────────────────────────────────────────
+  # AWS/Lambda Duration (ms), Statistic Maximum over 5m. Thresholds are ~80% of
+  # each function's configured timeout — early warning before timeout-kills.
+  ProcessPaymentCsvDurationAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: payments-processPaymentCsv-duration
+      AlarmDescription: payments-processPaymentCsv approaching timeout (~80% of 120s)
+      Namespace: AWS/Lambda
+      MetricName: Duration
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref ProcessPaymentCsvFunction
+      Statistic: Maximum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 96000
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts
+
+  ProcessPayrollEmailDurationAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: payments-processPayrollEmail-duration
+      AlarmDescription: payments-processPayrollEmail approaching timeout (~80% of 60s)
+      Namespace: AWS/Lambda
+      MetricName: Duration
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref ProcessPayrollEmailFunction
+      Statistic: Maximum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 48000
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts
+
+  FetchBoaTransactionsDurationAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: payments-fetchBoaTransactions-duration
+      AlarmDescription: payments-fetchBoaTransactions approaching timeout (~80% of 60s)
+      Namespace: AWS/Lambda
+      MetricName: Duration
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref FetchBoaTransactionsFunction
+      Statistic: Maximum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 48000
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts
+
+  ExpenseProcessorDurationAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: payments-expenseProcessor-duration
+      AlarmDescription: payments-expenseProcessor approaching timeout (~80% of 15s)
+      Namespace: AWS/Lambda
+      MetricName: Duration
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref ExpenseProcessorFunction
+      Statistic: Maximum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 12000
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts
+
+  ExpenseReceiverDurationAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: payments-expenseReceiver-duration
+      AlarmDescription: payments-expenseReceiver approaching timeout (~80% of 5s)
+      Namespace: AWS/Lambda
+      MetricName: Duration
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref ExpenseReceiverFunction
+      Statistic: Maximum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 4000
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts
+
+  SlackAppHomeDurationAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: payments-slackAppHome-duration
+      AlarmDescription: payments-slackAppHome approaching timeout (~80% of 30s default)
+      Namespace: AWS/Lambda
+      MetricName: Duration
+      Dimensions:
+        - Name: FunctionName
+          Value: !Ref SlackAppHomeFunction
+      Statistic: Maximum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 24000
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts
+
+  # ── DynamoDB alarms (Wave 1, SCOPE-CONFIRM) ────────────────────────────────
+  # AWS/DynamoDB throttle metrics for the PaymentsDashboard table. These metrics
+  # emit at the TableName dimension and only on the occurrence of a throttle
+  # event — none are currently present in CloudWatch (the table is
+  # PAY_PER_REQUEST, so sustained throttling is unlikely but possible during
+  # burst-capacity ramp). SystemErrors is intentionally not alarmed: AWS/DynamoDB
+  # SystemErrors does not emit at the TableName-only dimension, so it can never
+  # fire. Threshold > 0, Sum over 5m, ALARM-only.
+  DashboardTableReadThrottleAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: payments-dashboard-table-read-throttle
+      AlarmDescription: PaymentsDashboard table read requests throttled
+      Namespace: AWS/DynamoDB
+      MetricName: ReadThrottleEvents
+      Dimensions:
+        - Name: TableName
+          Value: !Ref DashboardTable
+      Statistic: Sum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 0
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts
+
+  DashboardTableWriteThrottleAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: payments-dashboard-table-write-throttle
+      AlarmDescription: PaymentsDashboard table write requests throttled
+      Namespace: AWS/DynamoDB
+      MetricName: WriteThrottleEvents
+      Dimensions:
+        - Name: TableName
+          Value: !Ref DashboardTable
+      Statistic: Sum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 0
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts
+
+  # ── API Gateway (HTTP API v2) alarms (Wave 1, SCOPE-CONFIRM) ────────────────
+  # AWS/ApiGateway v2 metrics on the implicit ServerlessHttpApi (ApiId dim).
+  # v2 metric names are 4xx/5xx/Latency (not 4XXError/5XXError). 5xx and Latency
+  # alarm on the API itself; 4xx is mostly client-driven so its threshold is
+  # set above zero to avoid noise (Slack URL-verification / bad requests).
+  ApiGateway5xxAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: payments-dashboard-api-5xx
+      AlarmDescription: payments-dashboard HTTP API returned 5xx responses
+      Namespace: AWS/ApiGateway
+      MetricName: 5xx
+      Dimensions:
+        - Name: ApiId
+          Value: !Ref ServerlessHttpApi
+      Statistic: Sum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 0
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts
+
+  ApiGateway4xxAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: payments-dashboard-api-4xx
+      AlarmDescription: payments-dashboard HTTP API elevated 4xx responses
+      Namespace: AWS/ApiGateway
+      MetricName: 4xx
+      Dimensions:
+        - Name: ApiId
+          Value: !Ref ServerlessHttpApi
+      Statistic: Sum
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 10
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts
+
+  ApiGatewayLatencyAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: payments-dashboard-api-latency-p99
+      AlarmDescription: payments-dashboard HTTP API p99 latency elevated (>3s)
+      Namespace: AWS/ApiGateway
+      MetricName: Latency
+      Dimensions:
+        - Name: ApiId
+          Value: !Ref ServerlessHttpApi
+      ExtendedStatistic: p99
+      Period: 300
+      EvaluationPeriods: 1
+      Threshold: 3000
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      AlarmActions:
+        - !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:site-alerts
+
   ProcessPayrollEmailLogGroup:
     Type: AWS::Logs::LogGroup
     Properties: